anaconda / anaconda-project

Tool for encapsulating, running, and reproducing data science projects
https://anaconda-project.readthedocs.io/en/latest/
Other
217 stars 88 forks source link

[BUG] add-packages to `dependencies:` section, if present. #337

Open mforbes opened 2 years ago

mforbes commented 2 years ago

When an anaconda-project.yml file has a dependencies: section rather than a packages: section as allowed by the alias implemented in PR #200 and requested in issue #194 so that anaconda-project.yml files can act as environment.yml files, new packages should be added to the existing section.

What happens instead is that anaconda-project add-packages inserts and additional packages: section with the new package rather than reusing the existing dependencies: section. Thus, anaconda-project add-packages pandas migh produce something like this:

#anaconda-project.yaml
...
dependencies:
- numpy
- matplotlib

packages:
- pandas

when it should produce something like:

#anaconda-project.yaml
...
dependencies:
- numpy
- matplotlib
- pandas
AlbertDeFusco commented 2 years ago

ah! thanks for that.

mforbes commented 2 years ago

I will try to take a stab at fixing this.

As per @jbednar's request in #194, can we replace packages with dependencies as the recommended approach going forward once this is resolved? I don't see any good reason for recommending packages except for this issue.

AlbertDeFusco commented 2 years ago

I agree. I've been looking at the code and I see that packages is somewhat hardcoded. I'm trying to come up with searches to help understand

❯ grep "\['packages'\]" -r .
./anaconda_project/env_spec.py:        template_json['something']['packages'] = packages
./anaconda_project/test/test_project_ops.py:            assert spec in env_commented_map['packages'], env_commented_map['packages']
./anaconda_project/project_file.py:                move_list_elements(spec_json['packages'], as_json['packages'])
./anaconda_project/conda_manager.py:        yaml_dict['packages'] = packages_dict
./anaconda_project/project_ops.py:        env_dict['packages'] = old_packages
./anaconda_project/project_ops.py:                        env_dict['packages'].append({'pip': []})
./anaconda_project/project_ops.py:                        env_dict['packages'] = [{'pip': []}]
./anaconda_project/project_ops.py:                    env_dict['packages'].append({'pip': pip_packages})
./anaconda_project/project_ops.py:                env_dict['packages'] = old_packages
./anaconda_project/project_ops.py:            env_dict['packages'] = old_packages
./anaconda_project/requirements_registry/providers/test/test_conda_env.py:        project.project_file.set_value(['packages', deps + ['boguspackage'])
❯ grep -E "get\('packages'.*\)" -r .
./anaconda_project/env_spec.py:        raw_dependencies = yaml.get('packages', [])
./anaconda_project/project_ops.py:        old_packages = env_dict.get('packages', [])
./anaconda_project/project_ops.py:            _pkgs = env_dict.get('packages', [])
./anaconda_project/project_ops.py:            old_packages = env_dict.get('packages', [])
./anaconda_project/project_ops.py:            old_packages = env_dict.get('packages', [])
./anaconda_project/project.py:            packages_by_platform = lock_set.get('packages', {})
./anaconda_project/project.py:                    packages = env_dict.get('packages', [])

This seems like a good opportunity to adopt black (or some other linter/formatter) to swap all singe to double quotes to help my grepping

mforbes commented 2 years ago

@AlbertDeFusco I will push some code later today, but I have made a test fixture that should run all tests with 'dependencies' and 'packages'. I will try to fix some of the resulting errors, but will probably need your guidance tomorrow (did you get my meeting request?).

AlbertDeFusco commented 2 years ago

Great! Looking forward to meeting tomorrow