search

conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
459 stars 102 forks source link

Issue with fake_conda_environment and mamba >= 1.4.6 #452

Closed riccardoporreca closed 1 year ago

riccardoporreca commented 1 year ago

Checklist

What is the idea?

The idea is to fully understand the issue detected in https://github.com/conda/conda-lock/pull/448#issuecomment-1627546017

Why is this needed?

Understand whether there is an interoperability issue between conda-lock and mamba>=1.4.6, and take required action

What should happen?

TBD

Additional Context

No response

riccardoporreca commented 1 year ago

Minimal example showing the behavior of mamba on a fake conda environment created by fake_conda_environment

Manual creation of an minimal fake conda env, based on what conda-lock would create:

rmdir -Rf /tmp/conda-lock-fake-env-minimal
mkdir -p /tmp/conda-lock-fake-env-minimal/conda-meta/
touch /tmp/conda-lock-fake-env-minimal/conda-meta/history
echo '{
  "name": "pydantic",
  "channel": "https://conda.anaconda.org/conda-forge/linux-64",
  "url": "https://conda.anaconda.org/conda-forge/linux-64/pydantic-1.7-py39h07f9747_0.tar.bz2",
  "md5": "f33be4cbe9070ef3671852aaa304e2f6",
  "build": "py39h07f9747_0",
  "build_number": 0,
  "version": "1.7",
  "subdir": "linux-64",
  "depends": [
    "libgcc-ng >=7.5.0",
    "python >=3.9,<3.10.0a0",
    "python_abi 3.9.*"
  ],
  "sha256": "09bbf313f929700e639b7fdc421130d567c958ac625a81e30a8cafbf17c35e8b"
}' > /tmp/conda-lock-fake-env-minimal/conda-meta/pydantic-1.7-py39h07f9747_0.json

Setup environments for various mamba version

mamba create -n mamba-1.4.5 mamba=1.4.5
mamba create -n mamba-1.4.7 mamba=1.4.6
mamba create -n mamba-1.4.7 mamba=1.4.7

Minimal mamba update behavior (actually, the same occurs with mamba install)

~/miniconda3/envs/mamba-1.4.5/bin/mamba -V > mamba-1.4.5-update.out
~/miniconda3/envs/mamba-1.4.5/bin/mamba update --override-channels --channel conda-forge -p /tmp/conda-lock-fake-env-minimal --json --dry-run pydantic==1.9.0 --no-deps -v &>> mamba-1.4.5-update.out

~/miniconda3/envs/mamba-1.4.6/bin/mamba -V > mamba-1.4.6-update.out
~/miniconda3/envs/mamba-1.4.6/bin/mamba update --override-channels --channel conda-forge -p /tmp/conda-lock-fake-env-minimal --json --dry-run pydantic==1.9.0 --no-deps -v &>> mamba-1.4.6-update.out

~/miniconda3/envs/mamba-1.4.7/bin/mamba -V > mamba-1.4.7-update.out
~/miniconda3/envs/mamba-1.4.7/bin/mamba update --override-channels --channel conda-forge -p /tmp/conda-lock-fake-env-minimal --json --dry-run pydantic==1.9.0 --no-deps -v &>> mamba-1.4.7-update.out

Relevant part of the output:

mamba-1.4.5-update.out

INFO conda.core.link:__init__(217): initializing UnlinkLinkTransaction with
  target_prefix: /tmp/conda-lock-fake-env-minimal
  unlink_precs:
    conda-forge/linux-64::pydantic-1.7-py39h07f9747_0
  link_precs:
    conda-forge/linux-64::pydantic-1.9.0-py310h5764c6d_1

mamba-1.4.6-update.out

INFO conda.core.link:__init__(217): initializing UnlinkLinkTransaction with
  target_prefix: /tmp/conda-lock-fake-env-minimal
  unlink_precs:

  link_precs:
    conda-forge/linux-64::pydantic-1.9.0-py310h5764c6d_1

No package record found!

mamba-1.4.7-update.out

INFO conda.core.link:__init__(217): initializing UnlinkLinkTransaction with
  target_prefix: /tmp/conda-lock-fake-env-minimal
  unlink_precs:

  link_precs:
    conda-forge/linux-64::pydantic-1.9.0-py310h5764c6d_1

No package record found!
riccardoporreca commented 1 year ago

Carrying on with the investigations...

riccardoporreca commented 1 year ago

Although I could not really pin down what happened in mamba 1.4.6 causing a difference behavior (it might have to do with the major rework done in https://github.com/mamba-org/mamba/pull/2537), I have had an intuition and tried to include `"fn" in the JSON file in the fake environment, and it does solve the issue!

rmdir -Rf /tmp/conda-lock-fake-env-minimal-with-fn
mkdir -p /tmp/conda-lock-fake-env-minimal-with-fn/conda-meta/
touch /tmp/conda-lock-fake-env-minimal-with-fn/conda-meta/history
echo '{
  "name": "pydantic",
  "channel": "https://conda.anaconda.org/conda-forge/linux-64",
  "url": "https://conda.anaconda.org/conda-forge/linux-64/pydantic-1.7-py39h07f9747_0.tar.bz2",
  "md5": "f33be4cbe9070ef3671852aaa304e2f6",
  "fn": "pydantic-1.7-py39h07f9747_0.tar.bz2",
  "build": "py39h07f9747_0",
  "build_number": 0,
  "version": "1.7",
  "subdir": "linux-64",
  "depends": [
    "libgcc-ng >=7.5.0",
    "python >=3.9,<3.10.0a0",
    "python_abi 3.9.*"
  ],
  "sha256": "09bbf313f929700e639b7fdc421130d567c958ac625a81e30a8cafbf17c35e8b"
}' > /tmp/conda-lock-fake-env-minimal-with-fn/conda-meta/pydantic-1.7-py39h07f9747_0.json

~/miniconda3/envs/mamba-1.4.7/bin/mamba -V > mamba-1.4.7-update-with-fn.out
~/miniconda3/envs/mamba-1.4.7/bin/mamba update --override-channels --channel conda-forge -p /tmp/conda-lock-fake-env-minimal-with-fn --json --dry-run pydantic==1.9.0 --no-deps -v &>> mamba-1.4.7-update-with-fn.out

mamba-1.4.7-update-with-fn.out

INFO conda.core.link:__init__(217): initializing UnlinkLinkTransaction with
  target_prefix: /tmp/conda-lock-fake-env-minimal-with-fn
  unlink_precs:
    conda-forge/linux-64::pydantic-1.7-py39h07f9747_0
  link_precs:
    conda-forge/linux-64::pydantic-1.9.0-py310h5764c6d_1
maresb commented 1 year ago

Thanks a lot for the investigations @riccardoporreca!!! I think this will be of interest to @jonashaag.

riccardoporreca commented 1 year ago

@maresb, @jonashaag, I think I got it!

What caused the issue in mamba 1.4.6 is the change to mamba.py from mamba-org/mamba#2632, which, more or less intentionally, kept only the use_mamba_experimental logic implying load_conda_installed is not called, which from a high-level look would be the one creating the filename ("fn") if not present.

I could verify this in a "hacky" way by replacing the 1.4.5 mamba.py in my mamba-1.4.6 test environment from above (also using a simpler run w/o --json)

Reproduce first the issue with the nominal 1.4.6

~/miniconda3/envs/mamba-1.4.6/bin/mamba update --override-channels --channel conda-forge -p /tmp/conda-lock-fake-env-minimal pydantic==1.9.0 \
   --dry-run --no-deps -v &> mamba-1.4.6-no-json.out

mamba-1.4.6-no-json.out

INFO conda.core.link:__init__(217): initializing UnlinkLinkTransaction with
  target_prefix: /tmp/conda-lock-fake-env-minimal
  unlink_precs:

  link_precs:
    conda-forge/linux-64::pydantic-1.9.0-py310h5764c6d_1

[...]

No package record found!

New run with the hacky replacement of 1.4.5 mamba.py

mv /home/riccardo/miniconda3/envs/mamba-1.4.6/lib/python3.11/site-packages/mamba/mamba.py \
   /home/riccardo/miniconda3/envs/mamba-1.4.6/lib/python3.11/site-packages/mamba/mamba.py.bak
cp /home/riccardo/miniconda3/envs/mamba-1.4.5/lib/python3.11/site-packages/mamba/mamba.py \
   /home/riccardo/miniconda3/envs/mamba-1.4.6/lib/python3.11/site-packages/mamba/mamba.py

~/miniconda3/envs/mamba-1.4.6/bin/mamba update --override-channels --channel conda-forge -p /tmp/conda-lock-fake-env-minimal pydantic==1.9.0 \
   --dry-run --no-deps -v &> mamba-1.4.6-mamba.py-1.4.5-no-json.out

mamba-1.4.6-mamba.py-1.4.5-no-json.out

INFO conda.core.link:__init__(217): initializing UnlinkLinkTransaction with
  target_prefix: /tmp/conda-lock-fake-env-minimal
  unlink_precs:
    conda-forge/linux-64::pydantic-1.7-py39h07f9747_0
  link_precs:
    conda-forge/linux-64::pydantic-1.9.0-py310h5764c6d_1

I guess it should be raised to mamba-org/mamba whether the change in mamba-org/mamba#2632 about use_mamba_experimental was intentional or not.

In any case, I have drafted a PR #453 showing the effectiveness of constructing "fn" in the fake conda environment

jonashaag commented 1 year ago

@AntoinePrv