conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
457 stars 102 forks source link

conda-lock pip dependency gets .tar.gz sdist rather than manylinux wheel when available manylinux wheel is more recent than 2_17 #517

Closed lesteve closed 8 months ago

lesteve commented 9 months ago

Checklist

What happened?

.tar.gz sdist are found instead of .whl when manylinux wheel are more recent than 2_17. This can cause long compilation on the machine doing conda-lock install. I believe https://github.com/conda/conda-lock/issues/395 has exactly the same root cause since the .whl in this case is a 2_27 and is not found by conda-lock and there is no sdist available on PyPI for open3d.

To reproduce:

# environment.yml
channels:
  - conda-forge

dependencies:
  - pip
  - pip:
    - lightgbm
conda-lock lock -f environment.yml --platform linux-64

The lockfile has a sdist .tar.gz rather than a .whl.

❯ grep lightgbm conda-lock.yml
- name: lightgbm
  url: https://files.pythonhosted.org/packages/98/a9/01f50aee85949ba713733b69a3f0b42d39719a414a0e29bdf2a9f05ecc53/lightgbm-4.1.0.tar.gz

lightgm has some manylinux_2_28 wheels that I can install fine on my Linux machine with pip: https://pypi.org/project/lightgbm/4.1.0/#files

I think this comes from the fact that the possible manylinux wheels recognised by conda-lock are hard-coded here (up until 2_17): https://github.com/conda/conda-lock/blob/f64b74ec91c57218bd11692b8e5cda3611d64649/conda_lock/pypi_solver.py#L42

It is a bit tricky to know which manylinux wheels should be supported, since the ones supported by the machine running conda-lock lock are not necessarily supported by the machine running conda-lock install. Maybe some heuristics could be used like all the manylinux wheels until the most recent docker image supported by manylinux? Right now it is 2_28 according to https://github.com/pypa/manylinux/blob/main/README.rst#docker-images.

Conda Info

active environment : conda-lock
    active env location : /home/lesteve/micromamba/envs/conda-lock
            shell level : 1
       user config file : /home/lesteve/.condarc
 populated config files : /home/lesteve/dev/dotfiles/.condarc
          conda version : 23.7.2
    conda-build version : not installed
         python version : 3.11.4.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.38=0
                          __linux=6.5.3=0
                          __unix=0=0
       base environment : /home/lesteve/micromamba  (writable)
      conda av data dir : /home/lesteve/micromamba/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/lesteve/micromamba/pkgs
                          /home/lesteve/.conda/pkgs
       envs directories : /home/lesteve/micromamba/envs
                          /home/lesteve/.conda/envs
               platform : linux-64
             user-agent : conda/23.7.2 requests/2.31.0 CPython/3.11.4 Linux/6.5.3-arch1-1 endeavouros/rolling glibc/2.38
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Conda Config

==> /home/lesteve/dev/dotfiles/.condarc <==
changeps1: False
channels:
  - conda-forge

==> cmd_line <==
debug: False
json: False

Conda list

# packages in environment at /home/lesteve/micromamba/envs/conda-lock:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
brotlipy                  0.7.0           py311hd4cff14_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cachecontrol              0.13.0             pyhd8ed1ab_0    conda-forge
cachecontrol-with-filecache 0.13.0             pyhd8ed1ab_0    conda-forge
cachy                     0.3.0              pyhd8ed1ab_1    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py311h409f033_3    conda-forge
charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
click-default-group       1.2.2              pyhd8ed1ab_1    conda-forge
clikit                    0.6.2              pyhd8ed1ab_2    conda-forge
conda-lock                2.3.0              pyhd8ed1ab_0    conda-forge
crashtest                 0.4.1              pyhd8ed1ab_0    conda-forge
cryptography              41.0.1          py311h63ff55d_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
distlib                   0.3.6              pyhd8ed1ab_0    conda-forge
ensureconda               1.4.3              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
filelock                  3.12.2             pyhd8ed1ab_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.31             pyhd8ed1ab_0    conda-forge
html5lib                  1.1                pyh9f0ad1d_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.7.0              pyha770c72_0    conda-forge
importlib_metadata        6.7.0                hd8ed1ab_0    conda-forge
jaraco.classes            3.2.3              pyhd8ed1ab_0    conda-forge
jeepney                   0.8.0              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
keyring                   23.13.1         py311h38be061_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libglib                   2.76.3               hebfc3b9_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lockfile                  0.12.2                     py_1    conda-forge
markupsafe                2.1.3           py311h459d7ec_0    conda-forge
more-itertools            9.1.0              pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.5           py311ha3edf6b_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
openssl                   3.1.3                hd590300_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pastel                    0.2.1              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pip                       23.1.2             pyhd8ed1ab_0    conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    conda-forge
platformdirs              3.6.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.10.9          py311h459d7ec_0    conda-forge
pylev                     1.4.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.4          hab00c5b_0_cpython    conda-forge
python_abi                3.11                    3_cp311    conda-forge
pyyaml                    6.0             py311hd4cff14_5    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.17.32         py311h459d7ec_0    conda-forge
ruamel.yaml.clib          0.2.7           py311h2582759_1    conda-forge
secretstorage             3.3.3           py311h38be061_1    conda-forge
setuptools                67.7.2             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomlkit                   0.11.8             pyha770c72_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
typing-extensions         4.6.3                hd8ed1ab_0    conda-forge
typing_extensions         4.6.3              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
virtualenv                20.23.1            pyhd8ed1ab_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge

Additional Context

No response