search

conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
495 stars 104 forks source link

conda-lock --update strips auth environment variables and changes extensions #446

Open nicoddemus opened 1 year ago

nicoddemus commented 1 year ago

Checklist

What happened?

Hi folks!

First of all thanks for the amazing tool!

At work we are using private channels with an embeded token via an environment variable.

When we create the lock file the first time, it correctly embeds the token environment variable in each package URL. However, when we call --update, it strips the authentication token from all URLs.

Is this a bug or are we using conda-lock incorrectly?

Reproducer:

name: test
channels:
  - https://${ESSS_CONDA_CHANNEL_TOKEN}@eden.esss.co/conda-channel/mirror-conda-forge  
platforms:
  - linux-64
  - win-64
dependencies:
  - boltons
$ mamba lock

This generates the conda-lock.yml file correctly, with each URL containing the environment variable embeded:

version: 1
metadata:
  content_hash:
    linux-64: 346581838184a77a7ad753d826aa2344e261e15be2579084a81781b2d33255b5
    win-64: e74885d79ac8882146d215d23539fb52c968c24dff91bd8dcee2d4e68b0754d1
  channels:
  - url: https://$ESSS_CONDA_CHANNEL_TOKEN@eden.esss.co/conda-channel/mirror-conda-forge
    used_env_vars:
    - ESSS_CONDA_CHANNEL_TOKEN
  platforms:
  - linux-64
  - win-64
  sources:
  - environment.yml
package:
- name: boltons
  version: 23.0.0
  manager: conda
  platform: linux-64
  dependencies:
    python: =2.7|>=3.7
  url: https://$ESSS_CONDA_CHANNEL_TOKEN@eden.esss.co/conda-channel/mirror-conda-forge/noarch/boltons-23.0.0-pyhd8ed1ab_0.conda
  hash:
    md5: 033eb25fffd222aceeca6d58cd953680
    sha256: 4ff828cceb8f55cb26d23b1a4c174d22c7cd92350221724bcaf2d6632e33fdee
  category: main
  optional: false

Note the url: https://$ESSS_CONDA_CHANNEL_TOKEN@eden.esss.co/conda-channel/mirror-conda-forge/noarch/boltons-23.0.0-pyhd8ed1ab_0.conda line.

However when we execute:

$ mamba lock --update boltons

All the $ESSS_CONDA_CHANNEL_TOKEN strings are stripped from the individual package urls:

version: 1
metadata:
  content_hash:
    linux-64: 346581838184a77a7ad753d826aa2344e261e15be2579084a81781b2d33255b5
    win-64: e74885d79ac8882146d215d23539fb52c968c24dff91bd8dcee2d4e68b0754d1
  channels:
  - url: https://$ESSS_CONDA_CHANNEL_TOKEN@eden.esss.co/conda-channel/mirror-conda-forge
    used_env_vars:
    - ESSS_CONDA_CHANNEL_TOKEN
  platforms:
  - linux-64
  - win-64
  sources:
  - environment.yml
package:
- name: boltons
  version: 23.0.0
  manager: conda
  platform: linux-64
  dependencies:
    python: =2.7|>=3.7
  url: https://eden.esss.co/conda-channel/mirror-conda-forge/noarch/boltons-23.0.0-pyhd8ed1ab_0.tar.bz2
  hash:
    md5: 033eb25fffd222aceeca6d58cd953680
    sha256: 4ff828cceb8f55cb26d23b1a4c174d22c7cd92350221724bcaf2d6632e33fdee
  category: main
  optional: false

Note the url: https://eden.esss.co/conda-channel/mirror-conda-forge/noarch/boltons-23.0.0-pyhd8ed1ab_0.tar.bz2 line: the environment variable has been stripped, and the extension changed from .conda to .tar.bz2.

Conda Info

mamba version : 1.4.4
     active environment : None
            shell level : 0
       user config file : C:\Users\bruno\.condarc
 populated config files : w:\mambaforge\.condarc
                          W:\Mamba\.condarc
          conda version : 23.5.0
    conda-build version : 3.25.0
         python version : 3.10.10.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=11.7=0
                          __win=0=0
       base environment : w:\mambaforge  (writable)
      conda av data dir : w:\mambaforge\etc\conda
  conda av metadata url : None
           channel URLs : https://{REDACTED_URL}/mamba-esss/win-64
                          https://{REDACTED_URL}/mamba-esss/noarch
                          https://{REDACTED_URL}/mirror-conda-forge/win-64
                          https://{REDACTED_URL}/mirror-conda-forge/noarch
                          https://{REDACTED_URL}/mirror-main/win-64
                          https://{REDACTED_URL}/mirror-main/noarch
          package cache : W:\Miniconda\pkgs
                          W:\Mamba\pkgs
       envs directories : W:\Mamba\envs
                          w:\mambaforge\envs
                          C:\Users\bruno\.conda\envs
                          C:\Users\bruno\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.5.0 requests/2.28.2 CPython/3.10.10 Windows/10 Windows/10.0.22000
          administrator : False
             netrc file : None
           offline mode : False

Conda Config

==> w:\mambaforge\.condarc <==
allow_softlinks: False
create_default_packages:
  - pip
  - python
extra_safety_checks: False
path_conflict: warn
channel_alias: {REDACTED_URL}/
channels:
  - https://{REDACTED_URL}/mamba-esss
  - https://{REDACTED_URL}/mirror-main
  - https://{REDACTED_URL}/mirror-conda-forge
show_channel_urls: True
always_yes: True
report_errors: False

==> W:\Mamba\.condarc <==
allow_softlinks: False
default_threads: 16
envs_dirs:
  - W:\Mamba\envs
pkgs_dirs:
  - W:\Miniconda\pkgs
  - W:\Mamba\pkgs
channel_alias: https://{REDACTED_URL}/
channels:
  - https://{REDACTED_URL}/mamba-esss
  - https://{REDACTED_URL}/mirror-conda-forge
  - https://{REDACTED_URL}/mirror-main
show_channel_urls: True
always_yes: True

==> envvars <==
envs_path:
  - W:\Mamba\envs

Conda list

λ conda list --show-channel-urls
# packages in environment at w:\mambaforge:
#
# Name                    Version                   Build  Channel
appdirs                   1.4.4              pyh9f0ad1d_0    mirror-conda-forge
beautifulsoup4            4.12.2             pyha770c72_0    mirror-conda-forge
boltons                   23.0.0             pyhd8ed1ab_0    mirror-conda-forge
brotlipy                  0.7.0           py310h8d17308_1005    https://conda.anaconda.org/conda-forge
bzip2                     1.0.8                h8ffe710_4    https://conda.anaconda.org/conda-forge
ca-certificates           2023.5.7             h56e8100_0    mirror-conda-forge
cachecontrol              0.13.0             pyhd8ed1ab_0    mirror-conda-forge
cachecontrol-with-filecache 0.13.0             pyhd8ed1ab_0    mirror-conda-forge
cachy                     0.3.0              pyhd8ed1ab_1    mirror-conda-forge
certifi                   2023.5.7           pyhd8ed1ab_0    mirror-conda-forge
cffi                      1.15.1          py310h628cb3f_3    https://conda.anaconda.org/conda-forge
chardet                   5.1.0           py310h5588dad_0    mirror-conda-forge
charset-normalizer        3.1.0              pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
click                     8.1.3           win_pyhd8ed1ab_2    mirror-conda-forge
click-default-group       1.2.2              pyhd8ed1ab_1    mirror-conda-forge
clikit                    0.6.2              pyhd8ed1ab_2    mirror-conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
conda                     23.5.0          py310h5588dad_1    mirror-conda-forge
conda-build               3.25.0          py310h5588dad_0    mirror-conda-forge
conda-devenv              3.1.0              pyhd8ed1ab_0    mirror-conda-forge
conda-execute             0.9.0                      py_0    mirror-conda-forge
conda-index               0.2.3              pyhd8ed1ab_0    mirror-conda-forge
conda-lock                2.1.0              pyhd8ed1ab_0    mirror-conda-forge
conda-package-handling    2.0.2              pyh38be061_0    https://conda.anaconda.org/conda-forge
conda-package-streaming   0.7.0              pyhd8ed1ab_1    https://conda.anaconda.org/conda-forge
crashtest                 0.4.1              pyhd8ed1ab_0    mirror-conda-forge
cryptography              40.0.1          py310h6e82f81_0    https://conda.anaconda.org/conda-forge
distlib                   0.3.6              pyhd8ed1ab_0    mirror-conda-forge
ensureconda               1.4.3              pyhd8ed1ab_0    mirror-conda-forge
filelock                  3.12.2             pyhd8ed1ab_0    mirror-conda-forge
fmt                       9.1.0                h181d51b_0    https://conda.anaconda.org/conda-forge
gitdb                     4.0.10             pyhd8ed1ab_0    mirror-conda-forge
gitpython                 3.1.31             pyhd8ed1ab_0    mirror-conda-forge
glob2                     0.7                        py_0    mirror-conda-forge
html5lib                  1.1                pyh9f0ad1d_0    mirror-conda-forge
idna                      3.4                pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
importlib-metadata        6.7.0              pyha770c72_0    mirror-conda-forge
importlib_metadata        6.7.0                hd8ed1ab_0    mirror-conda-forge
jaraco.classes            3.2.3              pyhd8ed1ab_0    mirror-conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    mirror-conda-forge
jsonpatch                 1.32               pyhd8ed1ab_0    mirror-conda-forge
jsonpointer               2.0                        py_0    mirror-conda-forge
keyring                   24.1.0          py310h5588dad_0    mirror-conda-forge
krb5                      1.20.1               heb0366b_0    https://conda.anaconda.org/conda-forge
libarchive                3.6.2                h27c7867_0    https://conda.anaconda.org/conda-forge
libcurl                   8.1.2                h68f0423_0    mirror-conda-forge
libffi                    3.4.2                h8ffe710_5    https://conda.anaconda.org/conda-forge
libiconv                  1.17                 h8ffe710_0    https://conda.anaconda.org/conda-forge
liblief                   0.12.3               h63175ca_0    mirror-conda-forge
libmamba                  1.4.4                ha24f096_1    mirror-conda-forge
libmambapy                1.4.4           py310h3fe4c2e_1    mirror-conda-forge
libsolv                   0.7.24               h12be248_1    mirror-conda-forge
libsqlite                 3.40.0               hcfcfb64_0    https://conda.anaconda.org/conda-forge
libssh2                   1.10.0               h9a1e1f7_3    https://conda.anaconda.org/conda-forge
libxml2                   2.10.3               hc3477c8_6    https://conda.anaconda.org/conda-forge
libzlib                   1.2.13               hcfcfb64_4    https://conda.anaconda.org/conda-forge
lockfile                  0.12.2                     py_1    mirror-conda-forge
lz4-c                     1.9.4                hcfcfb64_0    https://conda.anaconda.org/conda-forge
lzo                       2.10              he774522_1000    https://conda.anaconda.org/conda-forge
m2-msys2-runtime          2.5.0.17080.65c939c               3    mirror-conda-forge
m2-patch                  2.7.5                         2    mirror-conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    mamba-esss
m2w64-gcc-libs            5.3.0                         7    mamba-esss
m2w64-gcc-libs-core       5.3.0                         7    mamba-esss
m2w64-gmp                 6.1.0                         2    mamba-esss
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    mamba-esss
mamba                     1.4.4           py310hd9d798f_1    mirror-conda-forge
markupsafe                2.1.3           py310h8d17308_0    mirror-conda-forge
menuinst                  1.4.19          py310h5588dad_1    https://conda.anaconda.org/conda-forge
miniforge_console_shortcut 1.0                  h57928b3_0    https://conda.anaconda.org/conda-forge
more-itertools            9.1.0              pyhd8ed1ab_0    mirror-conda-forge
msgpack-python            1.0.5           py310h232114e_0    mirror-conda-forge
msys2-conda-epoch         20160418                      1    mamba-esss
mu_repo                   1.8.2              pyhd8ed1ab_0    mirror-conda-forge
openssl                   3.1.1                hcfcfb64_1    mirror-conda-forge
packaging                 23.1               pyhd8ed1ab_0    mirror-conda-forge
pastel                    0.2.1              pyhd8ed1ab_0    mirror-conda-forge
pip                       23.0.1             pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    mirror-conda-forge
platformdirs              3.6.0              pyhd8ed1ab_0    mirror-conda-forge
pluggy                    1.0.0              pyhd8ed1ab_5    https://conda.anaconda.org/conda-forge
psutil                    5.9.5           py310h8d17308_0    mirror-conda-forge
py-lief                   0.12.3          py310h00ffb61_0    mirror-conda-forge
pybind11-abi              4                    hd8ed1ab_3    https://conda.anaconda.org/conda-forge
pycosat                   0.6.4           py310h8d17308_1    https://conda.anaconda.org/conda-forge
pycparser                 2.21               pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
pydantic                  1.10.9          py310h8d17308_0    mirror-conda-forge
pylev                     1.4.0              pyhd8ed1ab_0    mirror-conda-forge
pyopenssl                 23.1.1             pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
pysocks                   1.7.1              pyh0701188_6    https://conda.anaconda.org/conda-forge
python                    3.10.10         h4de0772_0_cpython    https://conda.anaconda.org/conda-forge
python-libarchive-c       4.0             py310h5588dad_2    mirror-conda-forge
python_abi                3.10                    3_cp310    https://conda.anaconda.org/conda-forge
pytz                      2023.3             pyhd8ed1ab_0    mirror-conda-forge
pywin32-ctypes            0.2.1           py310h5588dad_0    mirror-conda-forge
pyyaml                    6.0             py310h8d17308_5    mirror-conda-forge
reproc                    14.2.4               hcfcfb64_0    https://conda.anaconda.org/conda-forge
reproc-cpp                14.2.4               h63175ca_0    https://conda.anaconda.org/conda-forge
requests                  2.28.2             pyhd8ed1ab_1    https://conda.anaconda.org/conda-forge
ripgrep                   13.0.0               h7f3b576_2    mirror-conda-forge
ruamel.yaml               0.17.21         py310h8d17308_3    https://conda.anaconda.org/conda-forge
ruamel.yaml.clib          0.2.7           py310h8d17308_1    https://conda.anaconda.org/conda-forge
setuptools                65.6.3             pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
six                       1.16.0             pyh6c4a22f_0    mirror-conda-forge
smmap                     3.0.5              pyh44b312d_0    mirror-conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    mirror-conda-forge
tk                        8.6.12               h8ffe710_0    https://conda.anaconda.org/conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    mirror-conda-forge
tomlkit                   0.11.8             pyha770c72_0    mirror-conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
tqdm                      4.65.0             pyhd8ed1ab_1    https://conda.anaconda.org/conda-forge
typing-extensions         4.6.3                hd8ed1ab_0    mirror-conda-forge
typing_extensions         4.6.3              pyha770c72_0    mirror-conda-forge
tzdata                    2023c                h71feb2d_0    https://conda.anaconda.org/conda-forge
ucrt                      10.0.22621.0         h57928b3_0    https://conda.anaconda.org/conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
vc                        14.3                hb6edc58_10    https://conda.anaconda.org/conda-forge
vc14_runtime              14.34.31931         h5081d32_13    mirror-conda-forge
virtualenv                20.23.1            pyhd8ed1ab_0    mirror-conda-forge
vs2015_runtime            14.34.31931         h4c5c07a_10    https://conda.anaconda.org/conda-forge
webencodings              0.5.1                      py_1    mirror-conda-forge
wheel                     0.40.0             pyhd8ed1ab_0    https://conda.anaconda.org/conda-forge
win_inet_pton             1.1.0              pyhd8ed1ab_6    https://conda.anaconda.org/conda-forge
xz                        5.2.6                h8d14728_0    https://conda.anaconda.org/conda-forge
yaml                      0.2.5                h8ffe710_2    mirror-conda-forge
yaml-cpp                  0.7.0                h63175ca_2    https://conda.anaconda.org/conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    mirror-conda-forge
zstandard                 0.19.0          py310h0009e47_1    https://conda.anaconda.org/conda-forge
zstd                      1.5.2                h12be248_6    https://conda.anaconda.org/conda-forge

Additional Context

No response

maresb commented 1 year ago

Looks like a bug, thanks for the report!

nicoddemus commented 10 months ago

Noticed that it is also changing the file extensions in the url field from .conda to .tar.bz2, which renders the lock file unusable. I've updated the title and description accordingly. 👍