New lock file format

[wolfv]

It would be great to have a new lockfile format. The current conda lockfile format (explicit env format) has quite a bunch of shortcomings: it's a weird ad-hoc format and only supports MD5 sums (and not even by default, I think & SHA256 is much better). The command to export an explicit environment in conda is conda list --explicit [--md5]

Micromamba already improves on this by changing the command to micromamba env export --explicit [--no-md5] (ie. it uses the env subcommand and defaults to add --md5 hashes).

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-arm64
@EXPLICIT
https://conda.anaconda.org/conda-forge/osx-arm64/argp-standalone-1.3-h3422bc3_0.tar.bz2#b744f29f1ef63fcedcb63b45d7ceed4a
https://conda.anaconda.org/conda-forge/osx-arm64/git-lfs-2.13.3-hce30654_0.tar.bz2#d0a6dda324b5d970c296dca838965193
...

I am thinking it would be nice to replace this with a proper YAML based format.

I am proposing something like:

metadata:
  spec: explicit-1.0
  description: ... # optional

name: myenv
# channels: ?
explicit-packages:
  linux-64:
  - name: xyz
    version: 0.15.0
    resolved: https://conda.anaconda.org/conda-forge/linux-64/xyz-0.15.0-had123.tar.bz2
    sha256: 123123123123123sjadalkjdlkajsk
    signature: ... ? # we need to also have certain metadata to validate signatures, though
  - name: pip
    version: 1.15.0
    resolved: https://conda.anaconda.org/conda-forge/noarch/pip-1.15.0-had123.tar.bz2
    sha256: 123123123123123sjadalkjdlkajsk
  osx-64:
  - name: xyz
    version: 0.15.0
    resolved: https://conda.anaconda.org/conda-forge/osx-64/xyz-0.15.0-had123.tar.bz2
    sha256: 123123123123123sjadalkjdlkajsk
  - name: abc
    version: 0.16.0
    resolved: https://conda.anaconda.org/conda-forge/osx-64/abc-0.16.0-had123.tar.bz2
    sha256: 123jk1lk23j1kl2j3kj12k3jlj1lk2

The explicit packages would contain a list per (supported) subdir. The list would be the full env resolution (including noarch pacakges) and in the correct order for installation (as current lockfiles today).

[jvansanten]

We should probably also include a field for the schema version so that we can recognize when the lockfile's consumer needs to be updated.

Definitely. Luckily there's https://github.com/conda-incubator/conda-lock/blob/7d9bd6d67d59fdd30d92a2ace3ee344aafa6a2b1/conda_lock/src_parser/lockfile.py#L79

[maresb]

Thanks @jvansanten, I just noticed that. (I was looking in the metadata section, not the top-level.)

[wholtz]

I was rather surprised that my lock file needed to end with -lock.yaml to work, and the error message was not helpful to me:

critical libmamba Invalid spec, no package name found: 

A better error message or more intelligent parsing of the yaml file to detect it is a lock file would be useful.

[maresb]

@wholtz, there have also been complaints about this on the conda-lock side (https://github.com/conda-incubator/conda-lock/pull/280). Probably the strategy there will be to first attempt to parse as yaml (for new style) and on failure fall back to parsing as explicit ( = old style).

[mfisher87]

I was rather surprised that my lock file needed to end with -lock.yaml to work

I was surprised recently in the opposite respect: when I try to install from a standard environment YAML formatted "lock file" (without hashes) generated with conda env export | grep -v "^prefix" > environment-lock.yml, micromamba complains that the file is in the wrong format. This file is accepted by mamba and conda with no issues.

critical libmamba YAML parsing error while reading environment lockfile located at '/tmp/environment-lock.yml' : invalid node; first invalid key: "version"

I work around this by renaming the file, but this was still surprising to not be able to install from a file following standard environment YAML format if it has a certain naming pattern.