glibc incompatibility

[berceanu]

Following instructions from docs.cupy.dev with miniforge

Issue:

$ conda install cupy

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \ 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                       

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versionsThe following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.27=0
  - cupy -> nccl[version='>=2.7.8.1,<3.0a0'] -> __glibc[version='>=2.17']

Your installed version is: 2.27

Environment (conda list): Ubuntu 18.04.4 LTS

``` +-----------------------------------------------------------------------------+ | NVIDIA-SMI 440.100 Driver Version: 440.100 CUDA Version: 10.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla V100-SXM3... Off | 00000000:1E:00.0 Off | 0 | | N/A 40C P0 52W / 350W | 0MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 15 Tesla V100-SXM3... Off | 00000000:C1:00.0 Off | 0 | | N/A 40C P0 52W / 350W | 0MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ ``` ``` $ conda list # packages in environment at /data/storage/berceanu/software/pkg/miniforge3pic: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge brotlipy 0.7.0 py38h8df0ef7_1001 conda-forge bzip2 1.0.8 h516909a_3 conda-forge ca-certificates 2020.6.20 hecda079_0 conda-forge certifi 2020.6.20 py38h924ce5b_2 conda-forge cffi 1.14.3 py38h1bdcb99_1 conda-forge chardet 3.0.4 py38h924ce5b_1008 conda-forge conda 4.9.0 py38h924ce5b_1 conda-forge conda-package-handling 1.7.2 py38h8df0ef7_0 conda-forge cryptography 3.2 py38hb23e4d4_0 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge ld_impl_linux-64 2.35 h769bd43_9 conda-forge libffi 3.2.1 he1b5a44_1007 conda-forge libgcc-ng 9.3.0 h5dbcf3e_17 conda-forge libgomp 9.3.0 h5dbcf3e_17 conda-forge libstdcxx-ng 9.3.0 h2ae2ef3_17 conda-forge ncurses 6.2 he1b5a44_2 conda-forge openssl 1.1.1h h516909a_0 conda-forge pip 20.2.4 py_0 conda-forge pycosat 0.6.3 py38h8df0ef7_1005 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pyopenssl 19.1.0 py_1 conda-forge pysocks 1.7.1 py38h924ce5b_2 conda-forge python 3.8.6 h852b56e_0_cpython conda-forge python_abi 3.8 1_cp38 conda-forge readline 8.0 he28a2e2_2 conda-forge requests 2.24.0 pyh9f0ad1d_0 conda-forge ruamel_yaml 0.15.80 py38h8df0ef7_1003 conda-forge setuptools 49.6.0 py38h924ce5b_2 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge sqlite 3.33.0 h4cf870e_1 conda-forge tk 8.6.10 hed695b0_1 conda-forge tqdm 4.51.0 pyh9f0ad1d_0 conda-forge urllib3 1.25.11 py_0 conda-forge wheel 0.35.1 pyh9f0ad1d_0 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h516909a_0 conda-forge zlib 1.2.11 h516909a_1010 conda-forge ```

Details about conda and system ( conda info ):

``` $ conda info active environment : None shell level : 0 user config file : /data/storage/berceanu/.condarc populated config files : /data/storage/berceanu/software/pkg/miniforge3pic/.condarc conda version : 4.9.0 conda-build version : not installed python version : 3.8.6.final.0 virtual packages : __cuda=10.2=0 __glibc=2.27=0 __unix=0=0 __archspec=1=x86_64 base environment : /data/storage/berceanu/software/pkg/miniforge3pic (writable) channel URLs : https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch package cache : /data/storage/berceanu/software/pkg/miniforge3pic/pkgs /data/storage/berceanu/.conda/pkgs envs directories : /data/storage/berceanu/software/pkg/miniforge3pic/envs /data/storage/berceanu/.conda/envs platform : linux-64 user-agent : conda/4.9.0 requests/2.24.0 CPython/3.8.6 Linux/5.4.0-48-generic ubuntu/18.04.4 glibc/2.27 UID:GID : 1129:1130 netrc file : None offline mode : False ```

[berceanu]

Also, if I try

$ conda install nccl
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \ 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                       

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package libstdcxx-ng conflicts for:
python=3.8 -> libffi[version='>=3.2.1,<3.3.0a0'] -> libstdcxx-ng[version='>=4.9']
python=3.8 -> libstdcxx-ng[version='>=7.3.0|>=7.5.0']

Package libgcc-ng conflicts for:
python=3.8 -> libgcc-ng[version='>=7.3.0|>=7.5.0']
python=3.8 -> libffi[version='>=3.2.1,<3.3.0a0'] -> libgcc-ng[version='>=4.9']

Package _libgcc_mutex conflicts for:
python=3.8 -> libgcc-ng[version='>=7.5.0'] -> _libgcc_mutex[version='*|0.1',build='conda_forge|main']
nccl -> libgcc-ng[version='>=7.5.0'] -> _libgcc_mutex[version='*|0.1',build='conda_forge|main']The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.27=0
  - feature:|@/linux-64::__glibc==2.27=0
  - nccl -> __glibc[version='>=2.17']

Your installed version is: 2.27

[berceanu]

# .condarc

channels:
- conda-forge

Bumped NVIDIA Driver Version to 455.32.00, no change.

[leofang]

@jakirkham I think this should be an issue in nccl-feedstock, what do you think?

[jakirkham]

Actually I think this is a Conda issue. It's come up before, but I don't recall which issue number it was. This one looks similar ( https://github.com/conda/conda/issues/9944 ). Though lacks more context.

[jakirkham]

Would add I think this is picking up cudatoolkit version 11.0, which does require __glibc >=2.17. So that part looks right, but the check that Conda is doing doesn't seem correct. If you are ok using an older CUDA version, would suggest requiring cudatoolkit=10.2, which should bypass this constraint.

Edit: Would add it is also possible to override the __glibc version using CONDA_OVERRIDE_GLIBC. Admittedly that shouldn't be needed here, but hopefully that provides you another path forward in the interim.

[leofang]

Looks like Conda detected @berceanu has __cuda=10.2, so I assumed he installed cudatoolkit=10.2, or at least Conda chose that version for him. If it's not the case, yet another way to enforce 10.2 is to do conda install -c conda-forge cupy cudatoolkit=10.2. This should also work.

[berceanu]

$ conda install cupy cudatoolkit=10.2

PackagesNotFoundError: The following packages are not available from current channels:

  - cudatoolkit=10.2

[berceanu]

Edit: Would add it is also possible to override the __glibc version using CONDA_OVERRIDE_GLIBC. Admittedly that shouldn't be needed here, but hopefully that provides you another path forward in the interim.

@jakirkham, what should I set it to? So far I tried export CONDA_OVERRIDE_GLIBC="", as well as export CONDA_OVERRIDE_GLIBC="2.17", both without any luck.

[leofang]

Did you exclude the default channel? Currently cudatoolkit must come from default.

[berceanu]

Did you exclude the default channel? Currently cudatoolkit must come from default.

Ah yes, that was excluded by default in miniforge. Now it works, but only with 10.x. Any idea how to make 11.x work?

[leofang]

It's still ongoing as you noticed from the other thread. We haven't released CuPy for 11.x on Conda-Forge yet.

[leofang]

That said, it should be possible to do pip install cupy-cuda110 in your conda env if you have CUDA 11.0 installed locally.