search

anaconda / anaconda-project

Tool for encapsulating, running, and reproducing data science projects
https://anaconda-project.readthedocs.io/en/latest/
Other
216 stars 88 forks source link

[BUG] Cross-platform locking failing on recent Python versions with `defaults` from arm64 to linux-64 #398

Open maximlt opened 8 months ago

maximlt commented 8 months ago

Running anaconda-project lock with the following project fails solving (using version 0.11.1). I am running anaconda-project from an environment created with conda create -n aproject python=3.10 anaconda-project, I'm on a Mac M2.

# anaconda-project.yml
name: bug?

packages:
  - python=3.10
  - scipy

platforms:
  - linux-64

Note that if I change python=3.10 for python=3.9 in the file below, the solve succeeds.

Output of the failing solve:

Updating locked dependencies for env spec default...
Resolving conda packages for linux-64
Error resolving dependencies for default: Error resolving for linux-64: /Users/mliquet/miniconda3/bin/conda create --override-channels --yes --quiet --json --dry-run --prefix /var/folders/wn/xthz1j85517f9bs0891kn6cr0000gp/T/_anaconda_project_resolve_5qg1wpx9 --channel defaults python=3.10 scipy: Encountered problems while solving:
  - nothing provides __glibc >=2.17 needed by libgcc-ng-11.2.0-h1234567_0

Could not solve for environment specs
The following packages are incompatible
├─ python 3.10**  is installable with the potential options
│  ├─ python [2.7.18|3.10.10|...|3.9.18] would require
│  │  ├─ libgcc-ng >=11.2.0 , which requires
│  │  │  └─ __glibc >=2.17 , which is missing on the system;
│  │  └─ libstdcxx-ng >=11.2.0 , which requires
│  │     └─ __glibc >=2.17 , which is missing on the system;
│  └─ python [3.10.0|3.10.3|3.10.4], which can be installed;
└─ scipy is not installable because there are no viable options
   ├─ scipy [0.19.1|1.0.0|...|1.2.1] would require
   │  └─ python >=2.7,<2.8.0a0  but there are no viable options
   │     ├─ python [2.7.13|2.7.14|...|2.7.18] conflicts with any installable versions previously reported;
   │     └─ python [2.7.18|3.10.10|...|3.9.18], which cannot be installed (as previously explained);
   ├─ scipy [0.19.1|1.0.0|1.0.1|1.1.0] would require
   │  └─ python >=3.5,<3.6.0a0 , which conflicts with any installable versions previously reported;
   ├─ scipy [0.19.1|1.0.0|...|1.5.2] would require
   │  └─ python >=3.6,<3.7.0a0 , which conflicts with any installable versions previously reported;
   ├─ scipy [1.1.0|1.2.0|...|1.7.3] would require
   │  └─ python >=3.7,<3.8.0a0  but there are no viable options
   │     ├─ python [2.7.18|3.10.10|...|3.9.18], which cannot be installed (as previously explained);
   │     └─ python [3.7.0|3.7.1|...|3.7.9] conflicts with any installable versions previously reported;
   ├─ scipy [1.10.0|1.10.1|...|1.9.3] would require
   │  └─ libstdcxx-ng >=11.2.0 , which cannot be installed (as previously explained);
   ├─ scipy [1.3.1|1.3.2|...|1.7.3] would require
   │  └─ python >=3.8,<3.9.0a0  but there are no viable options
   │     ├─ python [2.7.18|3.10.10|...|3.9.18], which cannot be installed (as previously explained);
   │     └─ python [3.8.0|3.8.1|...|3.8.8] conflicts with any installable versions previously reported;
   ├─ scipy [1.5.2|1.6.0|...|1.7.3] would require
   │  └─ python >=3.9,<3.10.0a0  but there are no viable options
   │     ├─ python [2.7.18|3.10.10|...|3.9.18], which cannot be installed (as previously explained);
   │     └─ python [3.9.0|3.9.1|...|3.9.7] conflicts with any installable versions previously reported;
   └─ scipy 1.7.3 would require
      ├─ libgfortran-ng with the potential options
      │  ├─ libgfortran-ng [11.2.0|7.5.0|8.4.0|9.3.0] would require
      │  │  └─ __glibc >=2.17 , which is missing on the system;
      │  └─ libgfortran-ng [5.4.0|7.2.0|7.3.0|8.2.0], which can be installed;
      └─ libgfortran4 >=7.5.0 , which requires
         └─ libgfortran-ng 7.5.0 *_17, which conflicts with any installable versions previously reported..

conda info:

     active environment : examples-gallery-manage
    active env location : /Users/mliquet/miniconda3/envs/examples-gallery-manage
            shell level : 1
       user config file : /Users/mliquet/.condarc
 populated config files : /Users/mliquet/.condarc
          conda version : 23.9.0
    conda-build version : not installed
         python version : 3.10.13.final.0
       virtual packages : __archspec=1=arm64
                          __osx=13.5.2=0
                          __unix=0=0
       base environment : /Users/mliquet/miniconda3  (writable)
      conda av data dir : /Users/mliquet/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/mliquet/miniconda3/pkgs
                          /Users/mliquet/.conda/pkgs
       envs directories : /Users/mliquet/miniconda3/envs
                          /Users/mliquet/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.9.0 requests/2.31.0 CPython/3.10.13 Darwin/22.6.0 OSX/13.5.2 solver/libmamba conda-libmamba-solver/23.9.1 libmambapy/1.5.1
                UID:GID : 502:20
             netrc file : None
           offline mode : False
jlstevens commented 8 months ago

Just to note I have reproduced this issue and here the solve error that I see:

(analytics)  jstevens /tmp/test $ anaconda-project lock
Updating locked dependencies for env spec default...
Resolving conda packages for linux-64
Error resolving dependencies for default: Error resolving for linux-64: /Users/jstevens/minicondaM1/bin/conda create --override-channels --yes --quiet --json --dry-run --prefix /var/folders/xb/2kdrlzn177d4vjfwhm3hnlq80000gp/T/_anaconda_project_resolve_7_i3qbf7 --channel defaults python=3.10 datashader: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
datashader -> python[version='>=2.7,<2.8.0a0|>=2.7|>=3.10,<3.11.0a0|>=3.8,<3.9.0a0|>=3.9,<3.10.0a0|>=3.11,<3.12.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0']
datashader -> dask-core -> python[version='2.7.*|3.5.*|3.6.*|>=3.5|>=3.6|>=3.7|>=3.8|>=3.7.1,<3.8.0a0|>=3.5.3']
python=3.10The following specifications were found to be incompatible with your system:

  - python=3.10 -> libgcc-ng[version='>=11.2.0'] -> __glibc[version='>=2.17']

Your installed version is: not available
maximlt commented 8 months ago

I had a similar issue with another dependency (panel) and asked @jbednar to reproduce it.

name: buggy
packages:
- python=3.9
- panel >=1
platforms:
- linux-64

Traceback:

> anaconda-project lock
Updating locked dependencies for env spec default...
Resolving conda packages for linux-64
Error resolving dependencies for default: Error resolving for linux-64: conda create --yes --quiet --json --dry-run --prefix /var/folders/q0/p4dg3t913ln18l9_6yfj7gv00000gp/T/_anaconda_project_resolve_jf3lxopf python=3.9 panel >=1: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
panel[version='>=1'] -> bleach -> python[version='>=2.7|>=2.7,<2.8.0a0|>=3.6|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0|>=3.7.1,<3.8.0a0|>=3.5']
panel[version='>=1'] -> python[version='>=3.10,<3.11.0a0|>=3.9,<3.10.0a0|>=3.11,<3.12.0a0|>=3.8,<3.9.0a0']
python=3.9The following specifications were found to be incompatible with your system:

  - python=3.9 -> libgcc-ng[version='>=11.2.0'] -> __glibc[version='>=2.17']

Your installed version is: not available
.
26.336u 1.338s 0:29.91 92.4%    0+0k 0+0io 1097pf+0w
maximlt commented 8 months ago

I can reproduce the issue with a pure conda command:

CONDA_SUBDIR=linux-64 conda create --override-channels --yes --dry-run --channel defaults -n testsubdir python=3.10 "scipy"

It works if I do one of the followings:

Using the last option, I can lock the project by running CONDA_OVERRIDE_GLIBC="2.22" anaconda-project lock.

Separately I tried to lock the same set of dependencies on linux-64 with conda-lock, and it worked fine without me having to do anything else.

Copied from someone who helped me looking at the issue internally:

conda-lock allows virtual packages to be specified in file using --virtual-package-spec. If this is not specified a default set is used.

Fixing this is anaconda-project in a manner similar to how conda-lock address virtual packages seems like a logic path forward.

maximlt commented 5 months ago

Cross-ref: https://github.com/conda/conda-libmamba-solver/issues/368