conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
467 stars 102 forks source link

conda lock doesn't respect lockfile versions #404

Open stefansjs opened 1 year ago

stefansjs commented 1 year ago

Checklist

What happened?

I'm trying to create a reproducible environment from an existing environment. What I'm trying to do is create a lockfile for a docker container that already has conda installed (it's a pytorch image). When I try to lock the environment it ends up updating existing dependencies, including e.g. python and pytorch. Which isn't good for a pytorch image.

What I'm expecting: specifying a lockfile, should freeze some or all of the dependency version to their current version, unless requesting an upgrade

How I'm reproducing it:

docker run -it pytorch/pytorch:1.12.1-cuda11.3-cudnn8-devel conda list --explicit > conda-linux-64.lock
conda lock -k explicit -f environment.yml

Forgive me if the problem is invocation, but the console seems to show that it's parsed the lockfile, but when I look at the values, many of the core packages like python and pytorch have different versions than what I started with.

Conda Info

active environment : None
       user config file : /root/.condarc
 populated config files : 
          conda version : 4.13.0
    conda-build version : 3.21.9
         python version : 3.7.13.final.0
       virtual packages : __linux=5.15.49=0
                          __glibc=2.27=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.13.0 requests/2.27.1 CPython/3.7.13 Linux/5.15.49-linuxkit ubuntu/18.04.6 glibc/2.27
                UID:GID : 0:0
             netrc file : None
           offline mode : False

Conda Config

No response

Conda list

# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             4.5                       1_gnu    defaults
backcall                  0.2.0              pyhd3eb1b0_0    defaults
beautifulsoup4            4.11.1           py37h06a4308_0    defaults
blas                      1.0                         mkl    defaults
brotlipy                  0.7.0           py37h27cfd23_1003    defaults
bzip2                     1.0.8                h7b6447c_0    defaults
ca-certificates           2022.07.19           h06a4308_0    defaults
certifi                   2022.6.15        py37h06a4308_0    defaults
cffi                      1.15.0           py37hd667e15_1    defaults
chardet                   4.0.0           py37h06a4308_1003    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
colorama                  0.4.4              pyhd3eb1b0_0    defaults
conda                     4.13.0           py37h06a4308_0    defaults
conda-build               3.21.9           py37h06a4308_0    defaults
conda-content-trust       0.1.1              pyhd3eb1b0_0    defaults
conda-package-handling    1.8.1            py37h7f8727e_0    defaults
cryptography              37.0.1           py37h9ce1e76_0    defaults
cudatoolkit               11.3.1               ha36c431_9    nvidia
decorator                 5.1.1              pyhd3eb1b0_0    defaults
ffmpeg                    4.3                  hf484d3e_0    pytorch
filelock                  3.6.0              pyhd3eb1b0_0    defaults
freetype                  2.11.0               h70c0345_0    defaults
giflib                    5.2.1                h7b6447c_0    defaults
glob2                     0.7                pyhd3eb1b0_0    defaults
gmp                       6.2.1                h295c915_3    defaults
gnutls                    3.6.15               he1e5248_0    defaults
icu                       58.2                 he6710b0_3    defaults
idna                      3.3                pyhd3eb1b0_0    defaults
intel-openmp              2021.4.0          h06a4308_3561    defaults
ipython                   7.31.1           py37h06a4308_1    defaults
jedi                      0.18.1           py37h06a4308_1    defaults
jinja2                    2.10.1                   py37_0    defaults
jpeg                      9e                   h7f8727e_0    defaults
lame                      3.100                h7b6447c_0    defaults
lcms2                     2.12                 h3be6417_0    defaults
ld_impl_linux-64          2.35.1               h7274673_9    defaults
libarchive                3.5.2                h5de8990_0    defaults
libffi                    3.3                  he6710b0_2    defaults
libgcc-ng                 9.3.0               h5101ec6_17    defaults
libgomp                   9.3.0               h5101ec6_17    defaults
libiconv                  1.16                 h7f8727e_2    defaults
libidn2                   2.3.2                h7f8727e_0    defaults
liblief                   0.11.5               h295c915_1    defaults
libpng                    1.6.37               hbc83047_0    defaults
libstdcxx-ng              9.3.0               hd4cf53a_17    defaults
libtasn1                  4.16.0               h27cfd23_0    defaults
libtiff                   4.2.0                h2818925_1    defaults
libunistring              0.9.10               h27cfd23_0    defaults
libwebp                   1.2.2                h55f646e_0    defaults
libwebp-base              1.2.2                h7f8727e_0    defaults
libxml2                   2.9.14               h74e7548_0    defaults
lz4-c                     1.9.3                h295c915_1    defaults
markupsafe                2.0.1            py37h27cfd23_0    defaults
matplotlib-inline         0.1.2              pyhd3eb1b0_2    defaults
mkl                       2021.4.0           h06a4308_640    defaults
mkl-service               2.4.0            py37h7f8727e_0    defaults
mkl_fft                   1.3.1            py37hd3c417c_0    defaults
mkl_random                1.2.2            py37h51133e4_0    defaults
ncurses                   6.3                  h7f8727e_2    defaults
nettle                    3.7.3                hbbd107a_1    defaults
numpy                     1.21.5           py37he7a7128_2    defaults
numpy-base                1.21.5           py37hf524024_2    defaults
openh264                  2.1.1                h4ff587b_0    defaults
openssl                   1.1.1q               h7f8727e_0    defaults
parso                     0.8.3              pyhd3eb1b0_0    defaults
patchelf                  0.13                 h295c915_0    defaults
pexpect                   4.8.0              pyhd3eb1b0_3    defaults
pickleshare               0.7.5           pyhd3eb1b0_1003    defaults
pillow                    9.0.1            py37h22f2fdc_0    defaults
pip                       22.1.2           py37h06a4308_0    defaults
pkginfo                   1.8.2              pyhd3eb1b0_0    defaults
prompt-toolkit            3.0.20             pyhd3eb1b0_0    defaults
psutil                    5.8.0            py37h27cfd23_1    defaults
ptyprocess                0.7.0              pyhd3eb1b0_2    defaults
py-lief                   0.11.5           py37h295c915_1    defaults
pycosat                   0.6.3            py37h27cfd23_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pygments                  2.11.2             pyhd3eb1b0_0    defaults
pyopenssl                 22.0.0             pyhd3eb1b0_0    defaults
pysocks                   1.7.1                    py37_1    defaults
python                    3.7.13               h12debd9_0    defaults
python-libarchive-c       2.9                pyhd3eb1b0_1    defaults
pytorch                   1.12.1          py3.7_cuda11.3_cudnn8.3.2_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2022.1           py37h06a4308_0    defaults
pyyaml                    6.0              py37h7f8727e_1    defaults
readline                  8.1.2                h7f8727e_1    defaults
requests                  2.27.1             pyhd3eb1b0_0    defaults
ripgrep                   12.1.1                        0    defaults
ruamel_yaml               0.15.100         py37h27cfd23_0    defaults
setuptools                61.2.0           py37h06a4308_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
soupsieve                 2.3.1              pyhd3eb1b0_0    defaults
sqlite                    3.38.2               hc218d9a_0    defaults
tk                        8.6.11               h1ccaba5_0    defaults
torchtext                 0.13.1                     py37    pytorch
torchvision               0.13.1               py37_cu113    pytorch
tqdm                      4.63.0             pyhd3eb1b0_0    defaults
traitlets                 5.1.1              pyhd3eb1b0_0    defaults
typing_extensions         4.3.0            py37h06a4308_0    defaults
tzdata                    2022a                hda174b7_0    defaults
urllib3                   1.26.8             pyhd3eb1b0_0    defaults
wcwidth                   0.2.5              pyhd3eb1b0_0    defaults
wheel                     0.37.1             pyhd3eb1b0_0    defaults
xz                        5.2.5                h7b6447c_0    defaults
yaml                      0.2.5                h7b6447c_0    defaults
zlib                      1.2.12               h7f8727e_1    defaults
zstd                      1.5.2                ha4553b6_0    defaults

Additional Context

I want to install more packages into a docker container in a reproducible way. Ideally I just want a way to lock an environment with specific constraints pre-determined. Possibly there should be a way to pass any of [--freeze-installed | --update-deps | -S | --update-all | --update-specs] to conda-lock