search

conda-incubator / conda-store

Data science environments, for collaboration. ✨
https://conda.store
BSD 3-Clause "New" or "Revised" License
143 stars 46 forks source link

Pinned `environment.yaml` should not include channel `defaults` #814

Open nkaretnikov opened 5 months ago

nkaretnikov commented 5 months ago

Context

The generated yaml file includes defaults even when the spec doesn't:

Repro ``` channels: - conda-forge dependencies: - python ==3.9 - pip: - flask description: '' name: test prefix: null variables: null ``` ``` channels: - conda-forge - defaults dependencies: - _libgcc_mutex=0.1=conda_forge - _openmp_mutex=4.5=2_gnu - ca-certificates=2024.2.2=hbcca054_0 - ld_impl_linux-64=2.40=h55db66e_0 - libffi=3.3=h58526e2_2 - libgcc-ng=13.2.0=hc881cc4_6 - libgomp=13.2.0=hc881cc4_6 - libsqlite=3.45.3=h2797004_0 - libstdcxx-ng=13.2.0=h95c4c6d_6 - libzlib=1.2.13=hd590300_5 - ncurses=6.4.20240210=h59595ed_0 - openssl=1.1.1w=hd590300_0 - pip=24.0=pyhd8ed1ab_0 - python=3.9.0=hffdb5ce_5_cpython - readline=8.2=h8228510_1 - setuptools=69.5.1=pyhd8ed1ab_0 - sqlite=3.45.3=h2c6b66d_0 - tk=8.6.13=noxft_h4845f30_101 - tzdata=2024a=h0c530f3_0 - wheel=0.43.0=pyhd8ed1ab_1 - xz=5.2.6=h166bdaf_0 - zlib=1.2.13=hd590300_5 - pip: - blinker==1.7.0 - click==8.1.7 - flask==3.0.3 - importlib-metadata==7.1.0 - itsdangerous==2.2.0 - jinja2==3.1.3 - markupsafe==2.1.5 - werkzeug==3.0.2 - zipp==3.18.1 name: /home/test/.local/share/conda-store/state/default/c289b5a3-1714091897-1-test prefix: /home/test/.local/share/conda-store/state/default/c289b5a3-1714091897-1-test ```

This is because conda/mamba env export includes this channel by default. There's an option --override-channels, which should help, I think, but it requires passing at least one -c / --channel flag, which would require extracting channels from the specification.

See action_generate_conda_export and api_get_build_yaml.

Value and/or benefit

Export matches the spec.

Anything else?

No response

jonzeper commented 5 months ago

We just realized that we've been unintentionally installing some packages from Anaconda's pkgs/main for some time, I guess due to this.

I wasn't able to reproduce the issue using mamba on its own. If I create an environment and export it with mamba env export, it does not add defaults. It seems to only happen when conda-store builds the environment. Not sure why - I tried to troubleshoot a bit but didn't get anywhere. Ended up setting the env var CONDA_CHANNELS=conda-forge as a quick fix on our deployment to restrict available channels.

Anyway, just wanted to throw an upvote for fixing this, and maybe should be considered a bug? nbconvert=6.5.4 is a good test case -- it exists on pkgs/main but not conda-forge

trallard commented 2 weeks ago

@jaimergp do you have any insights on the cleanest way to resolve this?

jaimergp commented 1 week ago

The cleanest would be "waiting" til deprecations introduced in https://github.com/conda/conda/pull/14227 are effective by March. It seems that users expect the channels list in their environment.yml file to be the only source of channel information (reasonably so!), so we just need to make sure the input and output YAMLs match this part.

I'd say https://github.com/conda/conda/issues/12356 is the cause behind, but we need to figure it out. The canonical workaround is to run conda config --remove channels defaults if defaults is not in the env.yml.