library cusparse not found

RamIdavalapaati commented 6 years ago

I have installed pyculib as pip install pyculib
When I import it in python shell then it is raising error that cusparse not found
```
>>> import pyculib
```

Error


Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/pyculib-1.0.1-py3.5.egg/pyculib/utils/libutils.py", line 17, in __new__
dll = open_cudalib(cls.lib)
File "/usr/local/lib/python3.5/dist-packages/numba/cuda/cudadrv/libs.py", line 48, in open_cudalib
raise OSError('library %s not found' % lib)
OSError: library cusparse not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "test_sort.py", line 3, in import pyculib File "/usr/local/lib/python3.5/dist-packages/pyculib-1.0.1-py3.5.egg/pyculib/init.py", line 49, in from . import blas, sparse, fft, rand, sorting File "/usr/local/lib/python3.5/dist-packages/pyculib-1.0.1-py3.5.egg/pyculib/sparse/init.py", line 2, in from .api import * File "/usr/local/lib/python3.5/dist-packages/pyculib-1.0.1-py3.5.egg/pyculib/sparse/api.py", line 6, in from .binding import (cuSparse, CUSPARSE_INDEX_BASE_ZERO, File "/usr/local/lib/python3.5/dist-packages/pyculib-1.0.1-py3.5.egg/pyculib/sparse/binding.py", line 1307, in cuSparse = _init_cuSparse() File "/usr/local/lib/python3.5/dist-packages/pyculib-1.0.1-py3.5.egg/pyculib/sparse/binding.py", line 1292, in _init_cuSparse name, func = _init_api_function(k, v) File "/usr/local/lib/python3.5/dist-packages/pyculib-1.0.1-py3.5.egg/pyculib/sparse/binding.py", line 1237, in _init_api_function lib = libcusparse() File "/usr/local/lib/python3.5/dist-packages/pyculib-1.0.1-py3.5.egg/pyculib/utils/libutils.py", line 20, in new e)) Exception: Cannot open library for cusparse: library cusparse not found


- I am using ubuntu-16.04 with the NVIDIA graphics card GeForce GTX 1080

Spexophis commented 6 years ago

Got the same problem. I am using window 10 pro.

stuartarchibald commented 6 years ago

Using conda will allow pyculibs dependencies to resolve correctly and a package called cudatoolkit will be installed which contains the missing CUDA libraries. Using pip only I think you'd need to install CUDA Toolkit from Nvidia directly and then set up some paths via environment variables as outlined here https://github.com/numba/numba/blob/master/README.rst#installing-numba.

Spexophis commented 6 years ago

Sorry, forget to say this. I was using conda install pyculib, which includes the cudatookit 9.0, but cusparse is still not found.

stuartarchibald commented 6 years ago

@billinair are you using cudatoolkit from the Numba channel?

charlineDS commented 6 years ago

pyculib is working with cuda 7.5 and VS2013 so you should downgrade cuda and Visual Studio : https://developer.nvidia.com/cuda-75-downloads-archive https://docs.microsoft.com/en-us/visualstudio/releasenotes/vs2013-community-vs

Then you should install cudatoolkit 7.5 conda install -c anaconda cudatoolkit==7.5

And then it should work :) Hope this help

Spexophis commented 6 years ago

@stuartarchibald I installed pyculib in the same environment with numba, namely the default environment of Anaconda.

Spexophis commented 6 years ago

@charlineuniv I have the old version in my old machine and worked with it before. But in the new computer, when I conda install pyculib, it automatically installs the cudatoolkit 9.0.

charlineDS commented 6 years ago

@billinair have you tried into a new conda environment, to install first cudatoolkit 7.5 and then pyculib ?

jimburnsphd commented 5 years ago

Mid Feb, 2019, issue still persists... as a new python user I did what every source instructed go to anaconda's main page, download and install. Then installed numba, cudatoolkit and pyculib, noticed python among some other things got downgraded to 3.6 but letting the "smart" tools resolve everything STILL doesn't seem to get things right! :) And people really used the term, 'dll hell' twenty years ago! It ain't got any better I tell ya! And as a complete newbie to Python, I'm spending a fortune searching and shifting through lots of useless garbage trying to find an answer! :(

stuartarchibald commented 5 years ago

@jimburnsphd is there any chance you could please expand on what you did so as to specifically describe the problem? Also if you have a reproducer for what you saw that would be really helpful. The Python 3.6 "downgrade" is because there hasn't been a rebuild of this package for Python 3.7 yet, as far as I can tell the constraint solver is correct in its behaviour. Conda provides a contained ecosystem under which CUDA should work fine providing you have suitable drivers and hardware installed on the host system. 'DLL hell' is avoided through the use of isolated environments. Here is an example (the machine it is running on has the Nvidia drivers and a Nvidia GPU installed):

$ conda create -n pyculib_example -q -y numba pyculib
Collecting package metadata: ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: <snip>/envs/pyculib_example

  added / updated specs:
    - numba
    - pyculib

The following NEW packages will be INSTALLED:

  blas               pkgs/main/linux-64::blas-1.0-mkl
  ca-certificates    pkgs/main/linux-64::ca-certificates-2019.1.23-0
  certifi            pkgs/main/linux-64::certifi-2018.11.29-py36_0
  cffi               pkgs/main/linux-64::cffi-1.11.5-py36he75722e_1
  cudatoolkit        pkgs/main/linux-64::cudatoolkit-10.0.130-0
  intel-openmp       pkgs/main/linux-64::intel-openmp-2019.1-144
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-8.2.0-hdf63c60_1
  libgfortran        pkgs/free/linux-64::libgfortran-3.0.0-1
  libgfortran-ng     pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-8.2.0-hdf63c60_1
  llvmlite           pkgs/main/linux-64::llvmlite-0.27.0-py36hd408876_0
  mkl                pkgs/main/linux-64::mkl-2019.1-144
  ncurses            pkgs/main/linux-64::ncurses-6.1-he6710b0_1
  numba              pkgs/main/linux-64::numba-0.42.0-py36h962f231_0
  numpy              pkgs/main/linux-64::numpy-1.13.3-py36ha266831_3
  openssl            pkgs/main/linux-64::openssl-1.1.1a-h7b6447c_0
  pip                pkgs/main/linux-64::pip-19.0.1-py36_0
  pycparser          pkgs/main/linux-64::pycparser-2.19-py36_0
  pyculib            pkgs/free/linux-64::pyculib-1.0.2-np113py36_2
  pyculib_sorting    pkgs/free/linux-64::pyculib_sorting-1.0.0-8
  python             pkgs/main/linux-64::python-3.6.8-h0371630_0
  readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5
  scipy              pkgs/main/linux-64::scipy-1.2.0-py36h7c811a0_0
  setuptools         pkgs/main/linux-64::setuptools-40.7.3-py36_0
  sqlite             pkgs/main/linux-64::sqlite-3.26.0-h7b6447c_0
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  wheel              pkgs/main/linux-64::wheel-0.32.3-py36_0
  xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

$ source activate pyculib_example

$ cat example.py
import numpy as np
import scipy.sparse.linalg
import pyculib

handle = pyculib.sparse.Sparse()
dtype = np.float32
m = n = 3
trans = 'N'

# Initialize the CSR matrix on the host and GPU.
row = np.array([0, 0, 0, 1, 1, 2])
col = np.array([0, 1, 2, 1, 2, 2])
data = np.array([0.431663, 0.955176, 0.925239, 0.0283651, 0.569277, 0.48015], dtype=dtype)

csrMatrixCpu = scipy.sparse.csr_matrix((data, (row, col)), shape=(m, n))
csrMatrixGpu = pyculib.sparse.csr_matrix((data, (row, col)), shape=(m, n))

print(csrMatrixCpu)
print(csrMatrixCpu.todense())

# Perform the analysis step on the GPU.
nnz = csrMatrixGpu.nnz
csrVal = csrMatrixGpu.data
csrRowPtr = csrMatrixGpu.indptr
csrColInd = csrMatrixGpu.indices

descr = handle.matdescr(0, 'N', 'U', 'G')
info = handle.csrsv_analysis(trans, m, nnz, descr, csrVal, csrRowPtr, csrColInd)

# Initialize the right-hand side of the system.
alpha = 1.0
rightHandSide = np.array([0.48200423, 0.39379725, 0.75963706], dtype=dtype)
gpuResult = np.zeros(m, dtype=dtype)

# Solve the system on the GPU and on the CPU.
handle.csrsv_solve(trans, m, alpha, descr, csrVal, csrRowPtr, csrColInd, info, rightHandSide, gpuResult)
cpuResult = scipy.sparse.linalg.dsolve.spsolve(csrMatrixCpu, rightHandSide, use_umfpack=False)

cpuDense = np.linalg.solve(csrMatrixCpu.todense(), rightHandSide)

print('gpu result = ' + str(gpuResult))
print('cpu result = ' + str(cpuResult))
print('cpu result = ' + str(cpuDense))(pyculib_example) 

$ python example.py 
  (0, 0)        0.431663
  (0, 1)        0.955176
  (0, 2)        0.925239
  (1, 1)        0.0283651
  (1, 2)        0.569277
  (2, 2)        0.48015
[[ 0.43166301  0.955176    0.92523903]
 [ 0.          0.0283651   0.56927699]
 [ 0.          0.          0.48015001]]
gpu result = [ 37.26496506 -17.86865234   1.58208275]
cpu result = [ 37.26496506 -17.86865234   1.58208275]
cpu result = [ 37.26496124 -17.86865044   1.58208275]

$ numba -s|grep -i cuda
__CUDA Information__
Found 1 CUDA devices
CUDA driver version                           : 10000
CUDA libraries:
cudatoolkit               10.0.130                      0

Tokariew commented 5 years ago

__OS Information__
Platform                                      : Windows-10-10.0.17763-SP0
Release                                       : 10
System Name                                   : Windows
Version                                       : 10.0.17763
OS specific info                              : 1010.0.17763SP0Multiprocessor Free

__Python Information__
Python Compiler                               : MSC v.1900 64 bit (AMD64)
Python Implementation                         : CPython
Python Version                                : 3.6.6
Python Locale                                 : en_GB cp1252

__LLVM information__
LLVM version                                  : 7.0.0

__CUDA Information__
Found 1 CUDA devices
id 0    b'GeForce GTX 1060 6GB'                              [SUPPORTED]
                      compute capability: 6.1
                           pci device id: 0
                              pci bus id: 1
Summary:
        1/1 devices are supported
CUDA driver version                           : 10010
CUDA libraries:
Finding cublas
        ERROR: can't locate lib
Finding cusparse
        ERROR: can't locate lib
Finding cufft
        ERROR: can't locate lib
Finding curand
        ERROR: can't locate lib
Finding nvvm
        named  nvvm64_33_0.dll
        finding libdevice for compute_20...     ok
        finding libdevice for compute_30...     ok
        finding libdevice for compute_35...     ok
        finding libdevice for compute_50...     ok

On Windows 10, 64 bits version numba didn't find the cudatoolkit libraries if they are newer than 7.5

jimburnsphd commented 5 years ago

@jimburnsphd is there any chance you could please expand on what you did so as to specifically describe the problem? Also if you have a reproducer for what you saw that would be really helpful

Expand? It doesn't get any more primative than Install numba (cudatookit I see is part of that install and doesn't need explicit installation), install pyculib. My numba -s looks almost identical to the one posted by Tokariew.

stuartarchibald commented 5 years ago

@jimburnsphd Yes, please expand so that I can try and reproduce and debug this. Information like which operating system you are running (but will guess at windows), which version of e.g. windows you are running, which CUDA drivers you having installed, which CUDA hardware you have in your machine, what versions of Python, Numba, cudatoolkit and PyCulib were installed, etc (basically numba -s output) is essential!

I'd really like to be able to help you get this working, occasionally bugs do appear and being able to reproduce what happened is key to resolving them. I think that if you are using windows and have cudatoolkit=10.x installed, then I may know what root cause of the problem is. Thanks for your help.

zen-ham commented 1 year ago

Issue still persists.

numba / pyculib

library cusparse not found #19