search

conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
456 stars 101 forks source link

Error creating lockfile from rendered environment #624

Closed sfinkens closed 2 months ago

sfinkens commented 2 months ago

Checklist

What happened?

conda-lock fails to create a lockfile from a rendered environment that includes pip dependencies. To reproduce:

  1. Create environment.yml 
    channels:
  - conda-forge
dependencies:
  - python
  - pip:
    - flit-core
  2. Create lockfile: conda-lock -p linux-64 -f environment.yml
  3. Render environment: conda-lock render -p linux-64 -k env conda-lock.yml
  4. Create lockfile from rendered environment: conda-lock -p linux-64 -f conda-linux-64.lock.yml

This fails with the following traceback.

Traceback ``` The above exception was the direct cause of the following exception: Traceback (most recent call last): File "###/bin/conda-lock", line 10, in sys.exit(main()) ^^^^^^ File "###/lib/python3.12/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/conda_lock/conda_lock.py", line 1402, in lock lock_func( File "###/lib/python3.12/site-packages/conda_lock/conda_lock.py", line 1110, in run_lock make_lock_files( File "###/lib/python3.12/site-packages/conda_lock/conda_lock.py", line 337, in make_lock_files lock_spec = make_lock_spec( ^^^^^^^^^^^^^^^ File "###/python3.12/site-packages/conda_lock/src_parser/__init__.py", line 91, in make_lock_spec lock_specs = _parse_source_files(src_files, platforms) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/conda_lock/src_parser/__init__.py", line 71, in _parse_source_files desired_envs.append(parse_environment_file(src_file, platforms)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/conda_lock/src_parser/environment_yaml.py", line 133, in parse_environment_file platform: _parse_environment_file_for_platform(content, category, platform) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/conda_lock/src_parser/environment_yaml.py", line 73, in _parse_environment_file_for_platform parse_python_requirement( File "###/lib/python3.12/site-packages/conda_lock/src_parser/pyproject_toml.py", line 423, in parse_python_requirement parsed_req = parse_requirement_specifier(requirement) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/conda_lock/src_parser/pyproject_toml.py", line 397, in parse_requirement_specifier return Requirement(requirement) ^^^^^^^^^^^^^^^^^^^^^^^^ File "###/lib/python3.12/site-packages/packaging/requirements.py", line 37, in __init__ raise InvalidRequirement(str(e)) from e packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier) flit-core === 3.9.0 --hash=sha256:7aada352fb0c7f5538c4fafeddf314d3a6a92ee8e2b1de70482329e42de70301 ~~~~~~~~~~^ ```

If I remove the --hash option, conda-lock succeeds.

Apparently conda is able to process the file: conda env create -f conda-linux-64.lock.yml works fine.

Conda Info

Output ```shell active environment : myenv active env location : ###/myenv shell level : 1 user config file : ###/.condarc populated config files : ###/.condarc ###/.condarc conda version : 24.3.0 conda-build version : not installed python version : 3.10.6.final.0 solver : libmamba (default) virtual packages : __archspec=1=broadwell __conda=24.3.0=0 __glibc=2.17=0 __linux=3.10.0=0 __unix=0=0 base environment : ###/conda/latest (read only) conda av data dir : ###/conda/latest/etc/conda conda av metadata url : None channel URLs : https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch package cache : ###/conda/pkgs envs directories : ###/conda/envs platform : linux-64 user-agent : conda/24.3.0 requests/2.31.0 CPython/3.10.6 Linux/3.10.0-1160.92.1.el7.x86_64 centos/7.9.2009 glibc/2.17 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8 UID:GID : 13865:16016 netrc file : ###/.netrc offline mode : False ```

Conda Config

Output ```shell ==> ###/conda/latest/.condarc <== channels: - conda-forge ==> /home/###/.condarc <== envs_dirs: - ###/conda/envs pkgs_dirs: - ###/conda/pkgs ```

Conda list

Output ```shell _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge annotated-types 0.6.0 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge aspell 0.60.8 pl5321hcb278e6_1 conda-forge brotli-python 1.1.0 py312h30efb56_1 conda-forge bzip2 1.0.8 hd590300_5 conda-forge c-ares 1.28.1 hd590300_0 conda-forge ca-certificates 2024.2.2 hbcca054_0 conda-forge cachecontrol 0.14.0 pyhd8ed1ab_0 conda-forge cachecontrol-with-filecache 0.14.0 pyhd8ed1ab_0 conda-forge cachy 0.3.0 pyhd8ed1ab_1 conda-forge certifi 2024.2.2 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py312hf06ca03_0 conda-forge charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge cleo 2.1.0 pyhd8ed1ab_0 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge click-default-group 1.2.4 pyhd8ed1ab_0 conda-forge clikit 0.6.2 pyhd8ed1ab_2 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge conda-lock 2.5.6 pyhd8ed1ab_0 conda-forge crashtest 0.4.1 pyhd8ed1ab_0 conda-forge cryptography 42.0.5 py312h241aef2_0 conda-forge curl 8.7.1 hca28451_0 conda-forge dbus 1.13.6 h5008d03_3 conda-forge distlib 0.3.8 pyhd8ed1ab_0 conda-forge dulwich 0.21.7 py312h98912ed_0 conda-forge ensureconda 1.4.4 pyhd8ed1ab_0 conda-forge expat 2.6.2 h59595ed_0 conda-forge filelock 3.13.4 pyhd8ed1ab_0 conda-forge gettext 0.22.5 h59595ed_2 conda-forge gettext-tools 0.22.5 h59595ed_2 conda-forge git 2.44.0 pl5321h709897a_0 conda-forge gitdb 4.0.11 pyhd8ed1ab_0 conda-forge gitpython 3.1.43 pyhd8ed1ab_0 conda-forge html5lib 1.1 pyh9f0ad1d_0 conda-forge idna 3.7 pyhd8ed1ab_0 conda-forge importlib-metadata 7.1.0 pyha770c72_0 conda-forge importlib_metadata 7.1.0 hd8ed1ab_0 conda-forge importlib_resources 6.4.0 pyhd8ed1ab_0 conda-forge jaraco.classes 3.4.0 pyhd8ed1ab_0 conda-forge jaraco.context 4.3.0 pyhd8ed1ab_0 conda-forge jaraco.functools 4.0.0 pyhd8ed1ab_0 conda-forge jeepney 0.8.0 pyhd8ed1ab_0 conda-forge jinja2 3.1.3 pyhd8ed1ab_0 conda-forge keyring 24.3.1 py312h7900ff3_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge krb5 1.21.2 h659d440_0 conda-forge ld_impl_linux-64 2.40 h41732ed_0 conda-forge libasprintf 0.22.5 h661eb56_2 conda-forge libasprintf-devel 0.22.5 h661eb56_2 conda-forge libblas 3.9.0 22_linux64_openblas conda-forge libcblas 3.9.0 22_linux64_openblas conda-forge libcurl 8.7.1 hca28451_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 hd590300_2 conda-forge libexpat 2.6.2 h59595ed_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 13.2.0 h807b86a_5 conda-forge libgettextpo 0.22.5 h59595ed_2 conda-forge libgettextpo-devel 0.22.5 h59595ed_2 conda-forge libgfortran-ng 13.2.0 h69a702a_5 conda-forge libgfortran5 13.2.0 ha4646dd_5 conda-forge libglib 2.80.0 hf2295e7_5 conda-forge libgomp 13.2.0 h807b86a_5 conda-forge libiconv 1.17 hd590300_2 conda-forge liblapack 3.9.0 22_linux64_openblas conda-forge libnghttp2 1.58.0 h47da74e_1 conda-forge libnsl 2.0.1 hd590300_0 conda-forge libopenblas 0.3.27 pthreads_h413a1c8_0 conda-forge libsqlite 3.45.3 h2797004_0 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-ng 13.2.0 h7e041cc_5 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libzlib 1.2.13 hd590300_5 conda-forge markupsafe 2.1.5 py312h98912ed_0 conda-forge more-itertools 10.2.0 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.7 py312h8572e83_0 conda-forge ncurses 6.4.20240210 h59595ed_0 conda-forge numpy 1.26.4 py312heda63a1_0 conda-forge openssl 3.2.1 hd590300_1 conda-forge packaging 24.0 pyhd8ed1ab_0 conda-forge pastel 0.2.1 pyhd8ed1ab_0 conda-forge pcre2 10.43 hcad00b1_0 conda-forge perl 5.32.1 7_hd590300_perl5 conda-forge pexpect 4.9.0 pyhd8ed1ab_0 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge pkginfo 1.10.0 pyhd8ed1ab_0 conda-forge platformdirs 4.2.0 pyhd8ed1ab_0 conda-forge poetry 1.8.2 linux_pyha804496_0 conda-forge poetry-core 1.9.0 pyhd8ed1ab_0 conda-forge poetry-plugin-export 1.7.1 pyhd8ed1ab_1 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pycparser 2.22 pyhd8ed1ab_0 conda-forge pydantic 2.7.0 pyhd8ed1ab_0 conda-forge pydantic-core 2.18.1 py312h4b3b743_0 conda-forge pylev 1.4.0 pyhd8ed1ab_0 conda-forge pyproject_hooks 1.0.0 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.12.3 hab00c5b_0_cpython conda-forge python-build 1.2.1 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.19.1 pyhd8ed1ab_0 conda-forge python-installer 0.7.0 pyhd8ed1ab_0 conda-forge python_abi 3.12 4_cp312 conda-forge pyyaml 6.0.1 py312h98912ed_1 conda-forge rapidfuzz 3.8.1 py312h30efb56_0 conda-forge readline 8.2 h8228510_1 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge requests-toolbelt 1.0.0 pyhd8ed1ab_0 conda-forge ruamel.yaml 0.18.6 py312h98912ed_0 conda-forge ruamel.yaml.clib 0.2.8 py312h98912ed_0 conda-forge secretstorage 3.3.3 py312h7900ff3_2 conda-forge setuptools 69.5.1 pyhd8ed1ab_0 conda-forge shellingham 1.5.4 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge smmap 5.0.0 pyhd8ed1ab_0 conda-forge tk 8.6.13 noxft_h4845f30_101 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge tomlkit 0.12.4 pyha770c72_0 conda-forge toolz 0.12.1 pyhd8ed1ab_0 conda-forge trove-classifiers 2024.4.10 pyhd8ed1ab_0 conda-forge typing-extensions 4.11.0 hd8ed1ab_0 conda-forge typing_extensions 4.11.0 pyha770c72_0 conda-forge tzdata 2024a h0c530f3_0 conda-forge urllib3 1.26.18 pyhd8ed1ab_0 conda-forge virtualenv 20.25.3 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 pyhd8ed1ab_2 conda-forge wheel 0.43.0 pyhd8ed1ab_1 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zstd 1.5.5 hfc55251_0 conda-forge ```

Additional Context

We use that workflow to add new packages to a lockfile while keeping existing packages as constant as possible.

  1. Render environment from lockfile
  2. Add new package to environment file
  3. Create new lockfile from modified environment file
  4. Commit lockfile changes
maresb commented 2 months ago

Thanks so much @sfinkens for the clear and easily-reproducible report!

In summary, the generated conda-linux-64.lock.yml contains

  - pip:
    - flit-core === 3.9.0 --hash=sha256:7aada352fb0c7f5538c4fafeddf314d3a6a92ee8e2b1de70482329e42de70302

Conda-lock tries to parse this flit-core dependency line here, feeding it into the packaging.Requirements class, which doesn't support the --hash=... representation.

I'm wondering why pip's hash-checking mode is not part of the requirements specification defined in the dependency specifiers grammar. (That seems to be the reason it's not handled by the packaging.Requirements class.)

@sfinkens, would you be willing to look into how to resolve this and create a PR?

sfinkens commented 2 months ago

@maresb Thanks for your quick reply! Sure, I'll give it a try. What strategy would you prefer? Just strip the hash or try to pass it through to where the lockfile is created?

maresb commented 2 months ago

@sfinkens, wonderful, thanks so much!

I'd prefer to pass through the lockfile hashes, and at first glance it doesn't seem like they need to travel very far. But if that proves too difficult we could also emit a clear warning and strip it.

sfinkens commented 2 months ago

@maresb While debugging my example I noticed that our workflow might be too complicated. Is it correct, that conda-lock uses an existing lockfile as constraint for the next solution? For example

conda-lock -f environment.yml
# A couple weeks later: Add a new package to environment.yml
conda-lock -f environment.yml  # does this use conda-lock.yml as constraint?

I tried with a simple case and that seems to work. If that's the case, I think it would be nice to mention it in the documentation.

maresb commented 2 months ago

If there is an existing lockfile, it is indeed used when regenerating.

Personally, I agree that this behavior is really confusing, and I dislike it, but I also am reluctant to change it since it may break existing workflows. I'm in the habit of always deleting conda-lock.yml before relocking.

I'd be very happy to accept a documentation update to make this more clear.

sfinkens commented 2 months ago

Thanks for the feedback! I'll prepare a PR for the documentation.