search

conda-forge / cupy-feedstock

A conda-smithy repository for cupy.
BSD 3-Clause "New" or "Revised" License
5 stars 23 forks source link

Library error when importing cupy #78

Closed darshats closed 3 years ago

darshats commented 3 years ago

Issue: Import cupy runs into the following error:

original error: /media/msmf/data/anaconda3/lib/python3.8/site-packages/cupy_backends/cuda/libs/cusparse.cpython-38-x86_64-linux-gnu.so: symbol cusparseSpMatSetValues version libcusparse.so.10 not defined in file libcusparse.so.10 with link time reference


Environment (conda list):

``` # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge ca-certificates 2020.11.8 ha878542_0 conda-forge certifi 2020.11.8 py39hf3d152e_0 conda-forge cudatoolkit 10.2.89 hfd86e86_1 cudnn 7.6.5 cuda10.2_0 cupy 8.1.0 py39hb4774a9_0 conda-forge cupyimg 0.1.dev0 pypi_0 pypi cython 0.29.21 pypi_0 pypi fast-upfirdn 0.5 pypi_0 pypi fastrlock 0.5 py39hf149a3a_1 conda-forge ld_impl_linux-64 2.35.1 hed1e6ac_0 conda-forge libblas 3.9.0 3_openblas conda-forge libcblas 3.9.0 3_openblas conda-forge libffi 3.2.1 he1b5a44_1007 conda-forge libgcc-ng 9.3.0 h5dbcf3e_17 conda-forge libgfortran-ng 9.3.0 he4bcb1c_17 conda-forge libgfortran5 9.3.0 he4bcb1c_17 conda-forge libgomp 9.3.0 h5dbcf3e_17 conda-forge liblapack 3.9.0 3_openblas conda-forge libopenblas 0.3.12 pthreads_h4812303_1 conda-forge libstdcxx-ng 9.3.0 h2ae2ef3_17 conda-forge nccl 2.8.3.1 h1a5f58c_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge numpy 1.19.4 py39h57d35e7_1 conda-forge openssl 1.1.1h h516909a_0 conda-forge pip 20.2.4 py_0 conda-forge python 3.9.0 h2a148a8_4_cpython conda-forge python_abi 3.9 1_cp39 conda-forge readline 8.0 he28a2e2_2 conda-forge setuptools 49.6.0 py39h079e4ff_2 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge sqlite 3.33.0 h4cf870e_1 conda-forge tk 8.6.10 hed695b0_1 conda-forge tzdata 2020d h516909a_0 conda-forge wheel 0.35.1 pyh9f0ad1d_0 conda-forge xz 5.2.5 h516909a_1 conda-forge zlib 1.2.11 h516909a_1010 conda-forge ```


Details about conda and system ( conda info ):

``` active environment : cuimg active env location : /media/msmf/data/anaconda3/envs/cuimg shell level : 2 user config file : /home/msmf/.condarc populated config files : /home/msmf/.condarc conda version : 4.8.3 conda-build version : 3.18.11 python version : 3.8.3.final.0 virtual packages : __cuda=11.0 __glibc=2.27 base environment : /media/msmf/data/anaconda3 (writable) channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /media/msmf/data/anaconda3/pkgs /home/msmf/.conda/pkgs envs directories : /media/msmf/data/anaconda3/envs /home/msmf/.conda/envs platform : linux-64 user-agent : conda/4.8.3 requests/2.24.0 CPython/3.8.3 Linux/5.4.0-42-generic ubuntu/18.04.5 glibc/2.27 UID:GID : 1000:1000 netrc file : None offline mode : False ```
leofang commented 3 years ago

Hi @darshats Sorry that no one replied. Is this still an issue? Any chance you have CUDA 10.1 installed locally (not from Conda-Forge)?

A possible workaround is to uninstall cupy and its dependencies installed from Conda-Forge, and re-install it from PyPI:

pip install cupy-cudaXXX

with XXX being your local CUDA version. Please refer to CuPy's documentation for it: https://docs.cupy.dev.

leofang commented 3 years ago

Possibly related: https://github.com/cupy/cupy/issues/3918, https://github.com/cupy/cupy/issues/4429, https://github.com/cupy/cupy/issues/4412

darshats commented 3 years ago

Hi @leofang - I managed to work around this, though I dont remember the steps now! It is possibly the cuda-10.1 issue. I did find that CUDA_PATH has to be set correctly for headers to be found on ubuntu else it leads to compile errors at runtime, something I think can be documented.

Thanks, Darshat

yichao96 commented 3 years ago

I also have this problem，python=3.6.9 cuda=10.1； but， when I downgrade the cuda version 10.0 and 'pip install cupy-cuda100' the import cupy error is OK!!

leofang commented 3 years ago

I did find that CUDA_PATH has to be set correctly for headers to be found on ubuntu else it leads to compile errors at runtime, something I think can be documented.

This should not happen if the compilation is done with NVRTC. With NVCC the situation is complicated. @darshats It'd be great if you could post a reproducer with the error output.

when I downgrade the cuda version 10.0 and 'pip install cupy-cuda100'

@LearnerGeorge Usually to use Conda-Forge's cudatoolkit (which is cupy's dependency) , the only thing you need to worry is to have the driver updated. The downgrade/upgrade can happen in Conda:

conda install -c conda-forge cudatoolkit=10.0 cupy  # or cudatoolkit=10.2/11.0

as long as the driver supports these runtime library versions.

darshats commented 3 years ago

Hi @leofang , I dont have a short program but the code is computing a gabor filter. LMK if you want to examine any env vars - just in case my env is corrupt. This is one of the call stack:

NVRTCError Traceback (most recent call last) /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupy/cuda/compiler.py in compile(self, options, log_stream) 515 nvrtc.addAddNameExpression(self.ptr, ker) --> 516 nvrtc.compileProgram(self.ptr, options) 517 mapping = None

cupy_backends/cuda/libs/nvrtc.pyx in cupy_backends.cuda.libs.nvrtc.compileProgram()

cupy_backends/cuda/libs/nvrtc.pyx in cupy_backends.cuda.libs.nvrtc.compileProgram()

cupy_backends/cuda/libs/nvrtc.pyx in cupy_backends.cuda.libs.nvrtc.check_status()

NVRTCError: NVRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

CompileException Traceback (most recent call last)

in in create_tiles(tile_size_pixel, pad, n_x_tiles, n_y_tiles, image, feature_folder) 75 roi_outer = image[y1:y2, x1:x2] 76 print(f'feature gen tile x={i},y={j} from top left {x1}:{y1}, bottom right {x2}:{y2}') ---> 77 outputs = feature_gen_tile(roi_outer) 78 79 blank_img = np.zeros((ydim, xdim), dtype=np.float64) in feature_gen_tile(roi) 10 outputs.append((img_grey, 'orig')) 11 print(f'grey image type {img_grey.dtype}') ---> 12 process_image(img_grey, 'grey', outputs, feature_df) 13 14 ## create features from HSV /media/msmf/data/Darshat/Python/cupyimg_featuregen_tiled.py in process_image(input_img, img_type, outputs, feature_df) 371 print('gabor...') 372 col_prefix = format(f'gabor_{img_type}') --> 373 ois = create_gabor_features(img, feature_df, col_prefix) 374 for item in ois: 375 outputs.append(item) /media/msmf/data/Darshat/Python/cupyimg_featuregen_tiled.py in create_gabor_features(img, feature_df, col_prefix) 112 for frequency in [15, 30]: 113 print(f'freq {frequency}') --> 114 output = do_gabor(img, frequency) 115 o = output.get() 116 output = None /media/msmf/data/Darshat/Python/cupyimg_featuregen_tiled.py in do_gabor(img, freq) 93 thetas = [i*PI/12 for i in range(12)] 94 for t in thetas: ---> 95 img_real, img_imag = gabor(img, frequency=freq, theta=t, sigma_x=15, sigma_y=15) 96 c = combine(img_real, img_imag) 97 ## accumulate highest responses /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupyimg/skimage/filters/_gabor.py in gabor(image, frequency, theta, bandwidth, sigma_x, sigma_y, n_stds, offset, mode, cval) 211 ) 212 --> 213 filtered = ndi.convolve(image, g, mode=mode, cval=cval) 214 215 return filtered.real, filtered.imag /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupyimg/scipy/ndimage/filters.py in convolve(input, weights, output, mode, cval, origin, use_weights_mask, dtype_mode) 199 ) 200 else: --> 201 return _correlate_or_convolve( 202 input, 203 weights, /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupyimg/scipy/ndimage/filters.py in _correlate_or_convolve(input, weights, output, mode, cval, origin, convolution, dtype_mode) 490 offsets = _filters_core._origins_to_offsets(origins, weights.shape) 491 kernel = _get_correlate_kernel(mode, weights.shape, int_type, offsets, cval) --> 492 output = _filters_core._call_kernel( 493 kernel, input, weights, output, weights_dtype=weights_dtype 494 ) /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupyimg/scipy/ndimage/_filters_core.py in _call_kernel(kernel, input, weights, output, structure, weights_dtype, structure_dtype) 150 output, temp = _util._get_output(output.dtype, input), output 151 args.append(output) --> 152 kernel(*args) 153 if needs_temp: 154 temp[...] = output[...] cupy/core/_kernel.pyx in cupy.core._kernel.ElementwiseKernel.__call__() cupy/core/_kernel.pyx in cupy.core._kernel.ElementwiseKernel._get_elementwise_kernel() cupy/_util.pyx in cupy._util.memoize.decorator.ret() cupy/core/_kernel.pyx in cupy.core._kernel._get_elementwise_kernel() cupy/core/_kernel.pyx in cupy.core._kernel._get_simple_elementwise_kernel() cupy/core/_kernel.pyx in cupy.core._kernel._get_simple_elementwise_kernel() cupy/core/core.pyx in cupy.core.core.compile_with_cache() /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupy/cuda/compiler.py in compile_with_cache(source, options, arch, cache_dir, extra_source, backend, enable_cooperative_groups, name_expressions, log_stream) 333 name_expressions, log_stream, cache_in_memory) 334 else: --> 335 return _compile_with_cache_cuda( 336 source, options, arch, cache_dir, extra_source, backend, 337 enable_cooperative_groups, name_expressions, log_stream, /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupy/cuda/compiler.py in _compile_with_cache_cuda(source, options, arch, cache_dir, extra_source, backend, enable_cooperative_groups, name_expressions, log_stream, cache_in_memory) 400 if backend == 'nvrtc': 401 cu_name = '' if cache_in_memory else name + '.cu' --> 402 ptx, mapping = compile_using_nvrtc( 403 source, options, arch, cu_name, name_expressions, 404 log_stream, cache_in_memory) /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupy/cuda/compiler.py in compile_using_nvrtc(source, options, arch, filename, name_expressions, log_stream, cache_in_memory) 171 cu_file.write(source) 172 --> 173 return _compile(source, options, cu_path, 174 name_expressions, log_stream) 175 else: /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupy/cuda/compiler.py in _compile(source, options, cu_path, name_expressions, log_stream) 155 name_expressions=name_expressions) 156 try: --> 157 ptx, mapping = prog.compile(options, log_stream) 158 except CompileException as e: 159 dump = _get_bool_env_variable( /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupy/cuda/compiler.py in compile(self, options, log_stream) 525 except nvrtc.NVRTCError: 526 log = nvrtc.getProgramLog(self.ptr) --> 527 raise CompileException(log, self.src, self.name, options, 528 'nvrtc' if not runtime.is_hip else 'hiprtc') 529 CompileException: /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupy/core/include/cupy/complex/complex.h(94): warning: __host__ annotation is ignored on a function("complex") that is explicitly defaulted on its first declaration /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupy/core/include/cupy/complex/complex.h(94): warning: __device__ annotation is ignored on a function("complex") that is explicitly defaulted on its first declaration /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupy/core/include/cupy/complex/complex.h(101): warning: __host__ annotation is ignored on a function("complex") that is explicitly defaulted on its first declaration /media/msmf/data/anaconda3/envs/cuty2/lib/python3.8/site-packages/cupy/core/include/cupy/complex/complex.h(101): warning: __device__ annotation is ignored on a function("complex") that is explicitly defaulted on its first declaration /tmp/tmpc1f_twtl/f0b6520f1d0c2886360a07a3b3a5562e_2.cubin.cu(9): catastrophic error: cannot open source file "math_constants.h" 1 catastrophic error detected in the compilation of "/tmp/tmpc1f_twtl/f0b6520f1d0c2886360a07a3b3a5562e_2.cubin.cu". Compilation terminated.
leofang commented 3 years ago

Excellent, @darshats! It's enough information for me. This is a known bug (math_constants.h is missing when there's no local installation of CUDA Toolkit (not cudatoolkit from Conda-Forge)) which we'll hopefully fix soon.