Document conda-lock and solver (conda/mamba) versions in conda-lock.yml

qmarcou commented 1 month ago

Checklist

[X] I added a descriptive title
[X] I searched open requests and couldn't find a duplicate

What is the idea?

Right now the (awesome) conda-lock.yml file contains all information to get exact packages to be installed, but no information about the conda-lock version and solver (conda or mamba) version used to perform the solve.

Why is this needed?

Due to some breaking changes (e.g between 1.x and 2.x) a lock file might become unusable by a third party user if the conda-lock version and solver version used to build the lock file is not explicitly given. Expecting the user to go over all previous versions of conda-lock and solver to get a working version is not realistic, and nothing in the documentation suggest to trace this information along with the lock file itself. Moreover conda and mamba solvers might end up with different solutions for the environment, for reproducibility's sake (which is the whole point of using conda-lock in a first place) the solver identity alone would already be a precious information.

What should happen?

I believe conda-lock.yml 's header should begin with this information:

# This lock file was generated by conda-lock v2.5.7 (https://github.com/conda/conda-lock). DO NOT EDIT!
#
# Solver used: mamba
# Solver versions:
# - mamba 1.5.8
# - conda 24.4.0
# 
# A "lock file" contains a concrete list of package versions (with checksums) to be installed. Unlike
# e.g. `conda env create`, the resulting environment will not change as new package versions become
# available, unless you explicitly update the lock file.
# ...

This way anybody knows that using these versions to install a lock file shoud just work. A bonus would be to have conda-lock process this information upon failing to read a lockfile to suggest the user to downgrade (or upgrade in the future) conda-lock ton install this particular lockfile.

Additional Context

I actually encountered this exact problem when I needed to reproduce some old results in a scientific project. Luckily I still had the information about the mamba and conda-lock version used somewhere, and I've been able to install the corresponding miniforge and conda-lock version.

maresb commented 1 month ago

Hey, thanks so much for the thoughtful writeup.

Could you explain in more detail the issue with the older lockfile? You should always be able to use the current version to install a lockfile generated by a previous version, and if not that seems like a bug.

qmarcou commented 1 month ago

Hi, Sure here's a bit more context: the lockfile in question is here

Just tried again with the conda-lock's latest version:

> mamba create -n test_conda-lock conda-lock -c conda-forge
> mamba activate test_conda-lock
> conda-lock --version
conda-lock, version 2.5.7
> conda-lock install tf_gpu.conda-lock.yml -n test_install 
Traceback (most recent call last):
  File "/home/quentin/anaconda3/envs/test_conda-lock/bin/conda-lock", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/conda_lock/conda_lock.py", line 1498, in click_install
    install(
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/conda_lock/conda_lock.py", line 1545, in install
    with _render_lockfile_for_install(
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/conda_lock/conda_lock.py", line 988, in _render_lockfile_for_install
    lock_content = parse_conda_lock_file(pathlib.Path(filename))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/conda_lock/lockfile/__init__.py", line 145, in parse_conda_lock_file
    lockfile = lockfile_v1_to_v2(LockfileV1.parse_obj(content))
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/pydantic/main.py", line 1118, in parse_obj
    return cls.model_validate(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/pydantic/main.py", line 551, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for Lockfile
package.276.optional
  Field required [type=missing, input_value={'dependencies': {}, 'has...l', 'version': '14.0.1'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing

Now using the old version of conda-lock that I used to create the lock file in the first place (in combination with an old mamba) still works:

> mamba create -n test_conda-lock conda-lock=1.2.1 -c conda-forge
> conda activate test_conda-lock
> conda-lock --version
/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/pydantic/_internal/_config.py:334: UserWarning: Valid config keys have changed in V2:
* 'allow_mutation' has been removed
  warnings.warn(message, UserWarning)
conda-lock, version 1.2.1
> mamba --version
mamba 0.24.0
conda 4.13.0
> conda-lock install tf_gpu.conda-lock.yml -n test_install 
/home/quentin/anaconda3/envs/test_conda-lock/lib/python3.12/site-packages/pydantic/_internal/_config.py:334: UserWarning: Valid config keys have changed in V2:
* 'allow_mutation' has been removed
  warnings.warn(message, UserWarning)
INFO:root:                  __    __    __    __
INFO:root:                 /  \  /  \  /  \  /  \
INFO:root:                /    \/    \/    \/    \
INFO:root:███████████████/  /██/  /██/  /██/  /████████████████████████
INFO:root:              /  / \   / \   / \   / \  \____
INFO:root:             /  /   \_/   \_/   \_/   \    o \__,
INFO:root:            / _/                       \_____/  `
INFO:root:            |/
INFO:root:        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
INFO:root:        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
INFO:root:        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
INFO:root:        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
INFO:root:        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
INFO:root:        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝
INFO:root:
INFO:root:        mamba (0.24.0) supported by @QuantStack
.......

I dare not replicate the issue with the up to date version of mamba by updating it on this machine just yet, but with the latest mamba version and conda-lock v1.2.3 the installation failed too.

I know you're putting a lot of efforts on keeping it all back compatible, but I would tend to think that documenting the versions of conda-lock and the solver used at solving time is a real cheap safety net in case it fails somewhere and would give users options to proceed before a patch is made to ensure back compatibility.

Thanks for all the great work!

maresb commented 1 month ago

@qmarcou thanks for this, this is really fascinating.

It's complaining because the optional: false line is missing from the libclang package. It seems that the validation got stricter. I'm pretty curious how this line was omitted in the first place.

Here's a script to test for the issue:

import yaml
from pathlib import Path
import sys

f = Path("conda-lock.yml")
y = yaml.load(f.read_text(), Loader=yaml.SafeLoader)

bad_packages = [p for p in y["package"] if "optional" not in p]
print(len(bad_packages))
print(bad_packages)

Running it on your lockfile produces

1
[{'dependencies': {}, 'hash': {'sha256': '02bacd219959601c627872f2c7c7090ce57cf6bd497618388e41813c7ee75a3a'}, 'manager': 'pip', 'name': 'libclang', 'platform': 'linux-64', 'source': None, 'url': 'https://files.pythonhosted.org/packages/ab/2f/c6f380aec0b064bccbd81141fecba9862b5634c838f13fff727adc84ceb9/libclang-14.0.1-py2.py3-none-manylinux1_x86_64.whl', 'version': '14.0.1'}]

Are you able to consistently reproduce such lockfiles that break with the latest version?

I'm slightly reluctant to add the version number since it adds entropy and it shouldn't matter. But I definitely see your point, I'm really curious which version of conda-lock produced this lockfile! :joy:

qmarcou commented 1 month ago

What do you mean by "consistently reproduce such lockfiles that break with the latest version"? Is conda-lock latest running and creating a proper lock-file ? I think yes but cannot test it right now. If you want to try it yourself I have all the recipes and README I have used to create the lock-file in the repo I have linked before.

I'm pretty sure the version used to build the lock-file in the first place is 1.2.1, I had a dedicated lock-env for that project just in case (turns out I've been right to do so)

conda / conda-lock