search

conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
456 stars 101 forks source link

conda-lock hangs with a combination of python 3.11 and dsub 0.4.9 #619

Open aaymeloglu opened 3 months ago

aaymeloglu commented 3 months ago

Checklist

What happened?

When I run conda-lock -p osx-arm64 -f environment.yml -k env with the following environment.yml file, it hangs indefinitely.

name: myenv
channels:
- conda-forge
dependencies:
- pip=24.0
- python=3.11.8
- pip:
  - dsub==0.4.8

If I change the python version to 3.10.13, it completes. If I change the dsub version to 0.4.7 or earlier, it completes. 0.4.8 or later fails. If I run conda env update -f environment.yml (conda version 24.1.2) it builds the environment successfully.

Conda Info

active environment : base
    active env location : /Users/aaymeloglu/miniconda3
            shell level : 1
       user config file : /Users/aaymeloglu/.condarc
 populated config files : /Users/aaymeloglu/.condarc
          conda version : 24.1.2
    conda-build version : not installed
         python version : 3.11.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=m1
                          __conda=24.1.2=0
                          __osx=14.2.1=0
                          __unix=0=0
       base environment : /Users/aaymeloglu/miniconda3  (writable)
      conda av data dir : /Users/aaymeloglu/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/aaymeloglu/miniconda3/pkgs
                          /Users/aaymeloglu/.conda/pkgs
       envs directories : /Users/aaymeloglu/miniconda3/envs
                          /Users/aaymeloglu/.conda/envs
               platform : osx-arm64
             user-agent : conda/24.1.2 requests/2.31.0 CPython/3.11.8 Darwin/23.2.0 OSX/14.2.1 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.7
                UID:GID : 501:20
             netrc file : None
           offline mode : False

Conda Config

==> /Users/aaymeloglu/.condarc <==
subdirs: []
channel_priority: strict
channels:
  - conda-forge
  - defaults

Conda list

# packages in environment at /Users/aaymeloglu/miniconda3:
#
# Name                    Version                   Build  Channel
archspec                  0.2.1              pyhd3eb1b0_0    defaults
boltons                   23.0.0          py311hca03da5_0    defaults
brotli-python             1.0.9           py311h313beb8_7    defaults
bzip2                     1.0.8                h620ffc9_4    defaults
c-ares                    1.27.0               h93a5062_0    conda-forge
ca-certificates           2024.2.2             hf0a4a13_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py311h80987f9_0    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
conda                     24.1.2          py311h267d04e_0    conda-forge
conda-content-trust       0.2.0           py311hca03da5_0    defaults
conda-libmamba-solver     23.12.0            pyhd3eb1b0_1    defaults
conda-package-handling    2.2.0           py311hca03da5_0    defaults
conda-package-streaming   0.9.0           py311hca03da5_0    defaults
cryptography              41.0.7          py311hd4332d6_0    defaults
distro                    1.8.0           py311hca03da5_0    defaults
fmt                       10.2.1               h2ffa867_0    conda-forge
icu                       73.2                 hc8870d7_0    conda-forge
idna                      3.4             py311hca03da5_0    defaults
jsonpatch                 1.32               pyhd3eb1b0_0    defaults
jsonpointer               2.1                pyhd3eb1b0_0    defaults
krb5                      1.21.2               h92f50d5_0    conda-forge
libarchive                3.7.2                hcacb583_1    conda-forge
libcurl                   8.5.0                h2d989ff_0    conda-forge
libcxx                    16.0.6               h4653b0c_0    conda-forge
libedit                   3.1.20230828         h80987f9_0    defaults
libev                     4.33                 h1a28f6b_1    defaults
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.4                hca03da5_0    defaults
libiconv                  1.17                 h0d3ecfb_2    conda-forge
libmamba                  1.5.7                h90c426b_0    conda-forge
libmambapy                1.5.7           py311h26e1311_0    conda-forge
libnghttp2                1.58.0               ha4dd798_1    conda-forge
libsolv                   0.7.24               h514c7bf_0    defaults
libsqlite                 3.45.1               h091b4b1_0    conda-forge
libssh2                   1.11.0               h7a5bd25_0    conda-forge
libxml2                   2.12.5               h0d0cfa8_0    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
lz4-c                     1.9.4                h313beb8_0    defaults
lzo                       2.10              h642e427_1000    conda-forge
mamba                     1.5.7           py311hb045da1_0    conda-forge
menuinst                  2.0.1           py311hca03da5_1    defaults
ncurses                   6.4                  h313beb8_0    defaults
openssl                   3.2.1                h0d3ecfb_0    conda-forge
packaging                 23.1            py311hca03da5_0    defaults
pcre2                     10.42                hb066dcc_0    defaults
pip                       23.3.1          py311hca03da5_0    defaults
platformdirs              3.10.0          py311hca03da5_0    defaults
pluggy                    1.0.0           py311hca03da5_1    defaults
pybind11-abi              4                    hd3eb1b0_1    defaults
pycosat                   0.6.6           py311h80987f9_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pyopenssl                 23.2.0          py311hca03da5_0    defaults
pysocks                   1.7.1           py311hca03da5_0    defaults
python                    3.11.8          hdf0ec26_0_cpython    conda-forge
python.app                3               py311h80987f9_0    defaults
python_abi                3.11                    4_cp311    conda-forge
readline                  8.2                  h1a28f6b_0    defaults
reproc                    14.2.4               hc377ac9_1    defaults
reproc-cpp                14.2.4               hc377ac9_1    defaults
requests                  2.31.0          py311hca03da5_0    defaults
ruamel.yaml               0.17.21         py311h80987f9_0    defaults
setuptools                68.2.2          py311hca03da5_0    defaults
sqlite                    3.41.2               h80987f9_0    defaults
tk                        8.6.13               h5083fa2_1    conda-forge
tqdm                      4.65.0          py311hb6e6a13_0    defaults
truststore                0.8.0           py311hca03da5_0    defaults
tzdata                    2023c                h04d1e81_0    defaults
urllib3                   1.26.18         py311hca03da5_0    defaults
wheel                     0.41.2          py311hca03da5_0    defaults
xz                        5.4.5                h80987f9_0    defaults
yaml-cpp                  0.8.0                h313beb8_0    defaults
zlib                      1.2.13               h53f4e23_5    conda-forge
zstandard                 0.19.0          py311h80987f9_0    defaults
zstd                      1.5.5                hd90d995_0    defaults

Additional Context

No response

zmbc commented 3 months ago

I see this as well with just:

name: my_env
channels:
  - conda-forge
dependencies:
  - python
  - pip
  - pip:
      - poetry

Running with --log-level DEBUG, the last line it prints before hanging is always DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443. I am guessing that no timeout is being set, as described in this issue; if a lot of requests are made, one is bound to fail, and it will just wait forever.

zmbc commented 3 months ago

This was fixed upstream in Poetry here: https://github.com/python-poetry/poetry/pull/5881

jxu10 commented 3 months ago

conda-lock solves after adding an explicit specifier for google-api-core[grpc] (dsub 0.4.8 specifies google-api-core>=2.7.3,<=2.11.0):

name: myenv
channels:
  - conda-forge
dependencies:
  - pip=24.0
  - python=3.11.8
  - pip:
      - dsub==0.4.8
      - google-api-core[grpc]>=2.7.3,<=2.11.0

It appears that the vendored Poetry has issues handling pip package extras, and the Poetry solver infinitely loops if both google-api-core and google-api-core[grpc] are sub-dependencies in the dependency tree.

I haven't tested, but #613 may help.