leofang
commented
7 months ago I think this has been fixed in #8200 and will be available in CuPy v13.1.0. Any chance you can build from the latest main branch and confirm?
Closed lucascolley closed 7 months ago
leofang
commented
7 months ago I think this has been fixed in #8200 and will be available in CuPy v13.1.0. Any chance you can build from the latest main branch and confirm?
lucascolley
commented
7 months ago Any chance you can build from the latest main branch and confirm?
What is the best way to do this with conda-forge? Just pip install git+https://github.com/cupy/cupy.git?
leofang
commented
7 months ago That should work because I see you already have CUDA Toolkit installed locally (/usr/local/cuda). You may want to use a fresh conda env to test this, to avoid installing CUDA packages from conda-forge. Also, setting this should help accelerate compilation:
CUPY_NVCC_GENERATE_CODE="arch=compute_61,code=sm_61"see https://docs.cupy.dev/en/stable/reference/environment.html#envvar-CUPY_NVCC_GENERATE_CODE.
lucascolley
commented
7 months ago Looks like my environment still needs some change:
Collecting git+https://github.com/cupy/cupy.git
Cloning https://github.com/cupy/cupy.git to /tmp/pip-req-build-5lgrbb8e
Running command git clone --filter=blob:none --quiet https://github.com/cupy/cupy.git /tmp/pip-req-build-5lgrbb8e
Resolved https://github.com/cupy/cupy.git to commit 028889eef3ba110829d677726348aa0f75aadb4e
Running command git submodule update --init --recursive -q
Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy<1.29,>=1.22 in /home/lucas/mambaforge/envs/cupy-test/lib/python3.11/site-packages (from cupy==13.0.0rc1) (1.26.4)
Collecting fastrlock>=0.5 (from cupy==13.0.0rc1)
Using cached fastrlock-0.8.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl.metadata (9.3 kB)
Using cached fastrlock-0.8.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl (52 kB)
Building wheels for collected packages: cupy
Building wheel for cupy (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [122 lines of output]
Generating cache key from header files...
Cache key (1545 files matching /tmp/pip-req-build-5lgrbb8e/cupy/_core/include/**): d156d4e64becd06cf23cd475b2ea91c4574dd026
Clearing directory: /tmp/pip-req-build-5lgrbb8e/cupy/.data
-------- Configuring Module: cuda --------
-------- Configuring Module: cusolver --------
-------- Configuring Module: cudnn --------
/tmp/tmp6v0u1th7/a.cpp:1:10: fatal error: cudnn.h: No such file or directory
1 | #include <cudnn.h>
| ^~~~~~~~~
compilation terminated.
command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-cc' failed with exit code 1
-------- Configuring Module: nccl --------
/tmp/tmp86566xpf/a.cpp:1:10: fatal error: nccl.h: No such file or directory
1 | #include <nccl.h>
| ^~~~~~~~
compilation terminated.
command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-cc' failed with exit code 1
-------- Configuring Module: nvtx --------
-------- Configuring Module: cutensor --------
/tmp/tmpoiv5w6z3/a.cpp:1:10: fatal error: cutensor.h: No such file or directory
1 | #include <cutensor.h>
| ^~~~~~~~~~~~
compilation terminated.
command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-cc' failed with exit code 1
-------- Configuring Module: cub --------
-------- Configuring Module: jitify --------
/home/lucas/mambaforge/envs/cupy-test/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lcuda: No such file or directory
collect2: error: ld returned 1 exit status
Cannot build a stub file.
Original error: command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-c++' failed with exit code 1
-------- Configuring Module: random --------
-------- Configuring Module: thrust --------
-------- Configuring Module: cusparselt --------
/tmp/tmpew_e5zf2/a.cpp:1:10: fatal error: cusparseLt.h: No such file or directory
1 | #include <cusparseLt.h>
| ^~~~~~~~~~~~~~
compilation terminated.
command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-cc' failed with exit code 1
-------- Configuring Module: dlpack --------
/home/lucas/mambaforge/envs/cupy-test/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/local/cuda/lib64/libcudart.so: undefined reference to `memcpy@GLIBC_2.14'
/home/lucas/mambaforge/envs/cupy-test/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/local/cuda/lib64/libcudart.so: undefined reference to `clock_gettime@GLIBC_2.17'
collect2: error: ld returned 1 exit status
**************************************************
*** WARNING: Cannot check compute capability
Cannot build a stub file.
Original error: command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-c++' failed with exit code 1
**************************************************
************************************************************
* CuPy Configuration Summary *
************************************************************
Build Environment:
Include directories: ['/tmp/pip-req-build-5lgrbb8e/cupy/_core/include/cupy/_cccl/libcudacxx', '/tmp/pip-req-build-5lgrbb8e/cupy/_core/include/cupy/_cccl/thrust', '/tmp/pip-req-build-5lgrbb8e/cupy/_core/include/cupy/_cccl/cub', '/tmp/pip-req-build-5lgrbb8e/cupy/_core/include', '/usr/local/cuda/include']
Library directories: ['/usr/local/cuda/lib64']
nvcc command : ['/usr/local/cuda/bin/nvcc']
hipcc command : (not found)
Environment Variables:
CFLAGS : -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/lucas/mambaforge/envs/cupy-test/include
LDFLAGS : -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/home/lucas/mambaforge/envs/cupy-test/lib -Wl,-rpath-link,/home/lucas/mambaforge/envs/cupy-test/lib -L/home/lucas/mambaforge/envs/cupy-test/lib
LIBRARY_PATH : (none)
CUDA_PATH : (none)
NVCC : (none)
HIPCC : (none)
ROCM_HOME : (none)
Modules:
cuda : Yes (version 12040)
cusolver : Yes
cudnn : No
-> Include files not found: ['cudnn.h']
-> Check your CFLAGS environment variable.
nccl : No
-> Include files not found: ['nccl.h']
-> Check your CFLAGS environment variable.
nvtx : Yes
cutensor : No
-> Include files not found: ['cutensor.h']
-> Check your CFLAGS environment variable.
cub : Yes (version 200200)
jitify : No
-> Cannot link libraries: ['cuda', 'nvrtc', 'cudart_static', 'pthread', 'rt', 'dl']
-> Check your LDFLAGS environment variable.
random : Yes
thrust : Yes (version 200200)
cusparselt: No
-> Include files not found: ['cusparseLt.h']
-> Check your CFLAGS environment variable.
dlpack : Yes
WARNING: Some modules could not be configured.
CuPy will be installed without these modules.
Please refer to the Installation Guide for details:
https://docs.cupy.dev/en/stable/install.html
************************************************************
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-req-build-5lgrbb8e/setup.py", line 93, in <module>
ext_modules = cupy_setup_build.get_ext_modules(True, ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-req-build-5lgrbb8e/install/cupy_builder/cupy_setup_build.py", line 475, in get_ext_modules
extensions = make_extensions(ctx, compiler, use_cython)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-req-build-5lgrbb8e/install/cupy_builder/cupy_setup_build.py", line 328, in make_extensions
raise Exception('Your CUDA environment is invalid. '
Exception: Your CUDA environment is invalid. Please check above error log.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for cupy
Running setup.py clean for cupy
error: subprocess-exited-with-error
× python setup.py clean did not run successfully.
│ exit code: 1
╰─> [122 lines of output]
Generating cache key from header files...
Cache key (1545 files matching /tmp/pip-req-build-5lgrbb8e/cupy/_core/include/**): d156d4e64becd06cf23cd475b2ea91c4574dd026
Clearing directory: /tmp/pip-req-build-5lgrbb8e/cupy/.data
-------- Configuring Module: cuda --------
-------- Configuring Module: cusolver --------
-------- Configuring Module: cudnn --------
/tmp/tmpe5f81sv_/a.cpp:1:10: fatal error: cudnn.h: No such file or directory
1 | #include <cudnn.h>
| ^~~~~~~~~
compilation terminated.
command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-cc' failed with exit code 1
-------- Configuring Module: nccl --------
/tmp/tmp5mssrxxm/a.cpp:1:10: fatal error: nccl.h: No such file or directory
1 | #include <nccl.h>
| ^~~~~~~~
compilation terminated.
command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-cc' failed with exit code 1
-------- Configuring Module: nvtx --------
-------- Configuring Module: cutensor --------
/tmp/tmph7vsxk2x/a.cpp:1:10: fatal error: cutensor.h: No such file or directory
1 | #include <cutensor.h>
| ^~~~~~~~~~~~
compilation terminated.
command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-cc' failed with exit code 1
-------- Configuring Module: cub --------
-------- Configuring Module: jitify --------
/home/lucas/mambaforge/envs/cupy-test/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lcuda: No such file or directory
collect2: error: ld returned 1 exit status
Cannot build a stub file.
Original error: command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-c++' failed with exit code 1
-------- Configuring Module: random --------
-------- Configuring Module: thrust --------
-------- Configuring Module: cusparselt --------
/tmp/tmpl2e5yyx5/a.cpp:1:10: fatal error: cusparseLt.h: No such file or directory
1 | #include <cusparseLt.h>
| ^~~~~~~~~~~~~~
compilation terminated.
command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-cc' failed with exit code 1
-------- Configuring Module: dlpack --------
/home/lucas/mambaforge/envs/cupy-test/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/local/cuda/lib64/libcudart.so: undefined reference to `memcpy@GLIBC_2.14'
/home/lucas/mambaforge/envs/cupy-test/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/local/cuda/lib64/libcudart.so: undefined reference to `clock_gettime@GLIBC_2.17'
collect2: error: ld returned 1 exit status
**************************************************
*** WARNING: Cannot check compute capability
Cannot build a stub file.
Original error: command '/home/lucas/mambaforge/envs/cupy-test/bin/x86_64-conda-linux-gnu-c++' failed with exit code 1
**************************************************
************************************************************
* CuPy Configuration Summary *
************************************************************
Build Environment:
Include directories: ['/tmp/pip-req-build-5lgrbb8e/cupy/_core/include/cupy/_cccl/libcudacxx', '/tmp/pip-req-build-5lgrbb8e/cupy/_core/include/cupy/_cccl/thrust', '/tmp/pip-req-build-5lgrbb8e/cupy/_core/include/cupy/_cccl/cub', '/tmp/pip-req-build-5lgrbb8e/cupy/_core/include', '/usr/local/cuda/include']
Library directories: ['/usr/local/cuda/lib64']
nvcc command : ['/usr/local/cuda/bin/nvcc']
hipcc command : (not found)
Environment Variables:
CFLAGS : -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/lucas/mambaforge/envs/cupy-test/include
LDFLAGS : -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/home/lucas/mambaforge/envs/cupy-test/lib -Wl,-rpath-link,/home/lucas/mambaforge/envs/cupy-test/lib -L/home/lucas/mambaforge/envs/cupy-test/lib
LIBRARY_PATH : (none)
CUDA_PATH : (none)
NVCC : (none)
HIPCC : (none)
ROCM_HOME : (none)
Modules:
cuda : Yes (version 12040)
cusolver : Yes
cudnn : No
-> Include files not found: ['cudnn.h']
-> Check your CFLAGS environment variable.
nccl : No
-> Include files not found: ['nccl.h']
-> Check your CFLAGS environment variable.
nvtx : Yes
cutensor : No
-> Include files not found: ['cutensor.h']
-> Check your CFLAGS environment variable.
cub : Yes (version 200200)
jitify : No
-> Cannot link libraries: ['cuda', 'nvrtc', 'cudart_static', 'pthread', 'rt', 'dl']
-> Check your LDFLAGS environment variable.
random : Yes
thrust : Yes (version 200200)
cusparselt: No
-> Include files not found: ['cusparseLt.h']
-> Check your CFLAGS environment variable.
dlpack : Yes
WARNING: Some modules could not be configured.
CuPy will be installed without these modules.
Please refer to the Installation Guide for details:
https://docs.cupy.dev/en/stable/install.html
************************************************************
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-req-build-5lgrbb8e/setup.py", line 93, in <module>
ext_modules = cupy_setup_build.get_ext_modules(True, ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-req-build-5lgrbb8e/install/cupy_builder/cupy_setup_build.py", line 475, in get_ext_modules
extensions = make_extensions(ctx, compiler, use_cython)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-req-build-5lgrbb8e/install/cupy_builder/cupy_setup_build.py", line 328, in make_extensions
raise Exception('Your CUDA environment is invalid. '
Exception: Your CUDA environment is invalid. Please check above error log.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed cleaning build dir for cupy
Failed to build cupy
ERROR: Could not build wheels for cupy, which is required to install pyproject.toml-based projectsYou may want to avoid using conda-forge compiler? I don't understand how CUDA was found but -lcuda (linking to the driver libcuda.so) would fail...
leofang
commented
7 months ago Another thing to try is to set the env var export CUDA_PATH=/usr/local/cuda and retry...
lucascolley
commented
7 months ago Another thing to try is to set the env var
export CUDA_PATH=/usr/local/cudaand retry...
Still fails with the same error sadly. I suppose I can try using pip instead, but I would like to get conda-forge working.
lucascolley
commented
7 months ago I think this has been fixed in https://github.com/cupy/cupy/pull/8200 and will be available in CuPy v13.1.0. Any chance you can build from the latest main branch and confirm?
Built in a pip env and can confirm that the problem is gone on main. Thanks! I would like to get conda-forge working if possible, but not urgent.
leofang
commented
6 months ago Thanks for confirming, @lucascolley! CuPy doesn't have Maxwell or Pascal GPUs in the CI, so we need to rely on users like you to help confirm a fix..
I would like to understand better: What do you mean by "getting conda-forge working"? We do maintain cupy-feedstock on conda-forge, so I'd like to know what's missing.
lucascolley
commented
6 months ago What do you mean by "getting conda-forge working"?
I just mean that I'd like to figure out what is wrong with my environment that is causing CuPy to error out. I understand that it's tricky to help as there are a lot of moving parts. Anyway, I am doing okay with my venv, so no worries if you have better things to be spending time on :)
leofang
commented
6 months ago If you write a dummy program that calls a driver API (thus requires linking to libcuda.so), and compile it with the conda-forge toolchain (gcc/nvcc), it might give you some clue. It works in cupy-feedstocks, because we link to the driver stub from cuda-driver-dev (for CUDA 12).
koolo233
commented
6 months ago Thank you @leofang and @lucascolley . pip install git+https://github.com/cupy/cupy.git works for me (Tesla P40 + CUDA 11.7).
Description
To Reproduce
Installation
Conda-Forge (
conda install ...)Environment
Additional Information
my env: