search

conda-forge / cupy-feedstock

A conda-smithy repository for cupy.
BSD 3-Clause "New" or "Revised" License
5 stars 23 forks source link

Depend on libnvjitlink #222

Closed madsbk closed 1 year ago

madsbk commented 1 year ago

Solution to issue cannot be found in the documentation.

Issue

Install CuPy in a new conda env fails because it picks up a local version of libnvjitlink.so. mamba install libnvjitlink fixes the issue.

Reproduce:

# Install cupy
$ mamba create -n cupy-cuda12.2 -c conda-forge cupy

# import cupy
$ conda activate cupy-cuda12.2
$ python -c "import cupy"
Traceback (most recent call last):
  File "/home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2/lib/python3.11/site-packages/cupy/__init__.py", line 17, in <module>
    from cupy import _core  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2/lib/python3.11/site-packages/cupy/_core/__init__.py", line 3, in <module>
    from cupy._core import core  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cupy/_core/core.pyx", line 1, in init cupy._core.core
  File "/home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2/lib/python3.11/site-packages/cupy/cuda/__init__.py", line 8, in <module>
    from cupy.cuda import compiler  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2/lib/python3.11/site-packages/cupy/cuda/compiler.py", line 13, in <module>
    from cupy.cuda import device
  File "cupy/cuda/device.pyx", line 9, in init cupy.cuda.device
ImportError: /home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2/lib/python3.11/site-packages/cupy_backends/cuda/libs/../../../../../libcusparse.so.12: undefined symbol: __nvJitLinkGetLinkedCubin_12_2, version libnvJitLink.so.12

# Note libnvJitLink.so.12 is loaded from a local installation
$ ldd  /home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2/lib/libcusparse.so.12.1.2.129 
        linux-vdso.so.1 (0x00007fff437ca000)
        libnvJitLink.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libnvJitLink.so.12 (0x00007f1b36e0d000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1b36dea000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1b36de0000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1b36dda000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1b36c8b000)
        libgcc_s.so.1 => /home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2/lib/libgcc_s.so.1 (0x00007f1b36c6e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1b36a7c000)

# Installing libnvJitLink fixes the issue
$ mamba install libnvjitlink
$ ldd  /home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2/lib/libcusparse.so.12.1.2.129
        linux-vdso.so.1 (0x00007ffd1dbf9000)
        libnvJitLink.so.12 => /home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2/lib/libnvJitLink.so.12 (0x00007f8b94300000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8b942bd000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8b942b3000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8b942ad000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8b9415e000)
        libgcc_s.so.1 => /home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2/lib/libgcc_s.so.1 (0x00007f8b94141000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8b93f4f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8ba7271000)

Installed packages

# packages in environment at /home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cuda-cudart               12.2.128                      0    nvidia
cuda-nvrtc                12.2.128                      0    nvidia
cuda-nvtx                 12.2.128                      0    nvidia
cuda-version              12.2                 he2b69de_2    conda-forge
cupy                      12.2.0                   pypi_0    pypi
fastrlock                 0.8                      pypi_0    pypi
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libblas                   3.9.0           17_linux64_openblas    conda-forge
libcblas                  3.9.0           17_linux64_openblas    conda-forge
libcublas                 12.2.4.5                      0    nvidia
libcufft                  11.0.8.91                     0    nvidia
libcurand                 10.3.3.129                    0    nvidia
libcusolver               11.5.1.129                    0    nvidia
libcusparse               12.1.2.129                    0    nvidia
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
liblapack                 3.9.0           17_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libnvjitlink              12.2.128                      0    nvidia
libopenblas               0.3.23          pthreads_h80387f5_0    conda-forge
libsqlite                 3.43.0               h2797004_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
numpy                     1.25.2                   pypi_0    pypi
openssl                   3.1.2                hd590300_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
python                    3.11.4          hab00c5b_0_cpython    conda-forge
python_abi                3.11                    3_cp311    conda-forge
readline                  8.2                  h8228510_1    conda-forge
setuptools                68.1.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge

Environment info

active environment : cupy-cuda12.2
    active env location : /home/mkristensen/apps/miniconda3/envs/cupy-cuda12.2
            shell level : 1
       user config file : /home/mkristensen/.condarc
 populated config files : /home/mkristensen/.condarc
          conda version : 4.10.1
    conda-build version : not installed
         python version : 3.8.5.final.0
       virtual packages : __cuda=12.0=0
                          __linux=5.15.0=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/mkristensen/apps/miniconda3  (writable)
      conda av data dir : /home/mkristensen/apps/miniconda3/etc/conda
  conda av metadata url : https://repo.anaconda.com/pkgs/main
           channel URLs : https://conda.anaconda.org/rapidsai-nightly/linux-64
                          https://conda.anaconda.org/rapidsai-nightly/noarch
                          https://conda.anaconda.org/dask/label/dev/linux-64
                          https://conda.anaconda.org/dask/label/dev/noarch
                          https://conda.anaconda.org/rapidsai/linux-64
                          https://conda.anaconda.org/rapidsai/noarch
                          https://conda.anaconda.org/nvidia/linux-64
                          https://conda.anaconda.org/nvidia/noarch
                          https://conda.anaconda.org/pytorch/linux-64
                          https://conda.anaconda.org/pytorch/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /data/conda-package-cache
       envs directories : /home/mkristensen/apps/miniconda3/envs
                          /home/mkristensen/.conda/envs
               platform : linux-64
             user-agent : conda/4.10.1 requests/2.24.0 CPython/3.8.5 Linux/5.15.0-79-generic ubuntu/20.04.6 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
leofang commented 1 year ago

Hey Mads, you can't mix the nvidia channel with the conda-forge channel yet. cuSOLVER depends on libnvjitlink so on conda-forge the dependency is properly declared, but not on nvidia. (I am happy to point you to internal discussions if needed 🙂)

madsbk commented 1 year ago

Ahh ok, thanks