search

rusty1s / pytorch_sparse

PyTorch Extension Library of Optimized Autograd Sparse Matrix Operations
MIT License
1.01k stars 147 forks source link

OSError: libcusparse.so.11: cannot open shared object file: No such file or directory #271

Closed theekshanadis closed 1 year ago

theekshanadis commented 2 years ago

Hi,

I'm trying to install the latest version of torch-geometric. After trying multiple times the guide provided here, I'm still getting the below error while loading torch_sparse dependancy.

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/n10651811/.local/lib/python3.9/site-packages/torch_sparse/__init__.py", line 19, in <module> torch.ops.load_library(spec.origin) File "/home/n10651811/.local/lib/python3.9/site-packages/torch/_ops.py", line 220, in load_library ctypes.CDLL(path) File "/pkg/suse12/software/Python/3.9.1-GCCcore-9.3.0/lib/python3.9/ctypes/__init__.py", line 374, in __init__ self._handle = _dlopen(self._name, mode) OSError: libcusparse.so.11: cannot open shared object file: No such file or directory

Before installing I checked the cuda and pytorch versions in my machine and issued a custom command as follows,

pip install torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-1.11.0+11.3.html

Yet, I'm still getting the mentioned error.

Could you please help me to resolve this?

Many thanks Shaan.

rusty1s commented 2 years ago

Can you show me the log of running

pip install --verbose torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-1.11.0+11.3.html

In addition, what does nvcc --version report?

Queuecumber commented 1 year ago

Also having this error while following the readme instructions: pip install torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-1.13.0+cu117.html

Output of pip install --verbose --upgrade --force-reinstall --no-cache-dir torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-1.13.0+cu117.html

Using pip 22.3.1 from /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/pip (python 3.8)
Looking in links: https://data.pyg.org/whl/torch-1.13.0+cu117.html
Collecting torch-scatter
  Downloading https://data.pyg.org/whl/torch-1.13.0%2Bcu117/torch_scatter-2.1.0%2Bpt113cu117-cp38-cp38-linux_x86_64.whl (10.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.1/10.1 MB 30.9 MB/s eta 0:00:00
Collecting torch-sparse
  Downloading https://data.pyg.org/whl/torch-1.13.0%2Bcu117/torch_sparse-0.6.15%2Bpt113cu117-cp38-cp38-linux_x86_64.whl (4.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 57.5 MB/s eta 0:00:00
Collecting scipy
  Downloading scipy-1.9.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 33.8/33.8 MB 60.7 MB/s eta 0:00:00
Collecting numpy<1.26.0,>=1.18.5
  Downloading numpy-1.23.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.1/17.1 MB 64.1 MB/s eta 0:00:00
Installing collected packages: torch-scatter, numpy, scipy, torch-sparse
  Attempting uninstall: torch-scatter
    Found existing installation: torch-scatter 2.1.0+pt113cu117
    Uninstalling torch-scatter-2.1.0+pt113cu117:
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__init__.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/__init__.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_broadcasting.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_gather.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_multi_gpu.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_scatter.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_segment.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_zero_tensors.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/utils.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_broadcasting.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_gather.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_multi_gpu.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_scatter.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_segment.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_zero_tensors.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/utils.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/torch_scatter-2.1.0+pt113cu117.dist-info/
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/torch_scatter/
      Successfully uninstalled torch-scatter-2.1.0+pt113cu117
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.3
    Uninstalling numpy-1.19.3:
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/bin/f2py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/bin/f2py3
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/bin/f2py3.8
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/numpy-1.19.3.dist-info/
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/numpy.libs/
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/numpy/
      Successfully uninstalled numpy-1.19.3
  changing mode of /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/bin/f2py to 755
  changing mode of /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/bin/f2py3 to 755
  changing mode of /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/bin/f2py3.8 to 755
  Attempting uninstall: scipy
    Found existing installation: scipy 1.5.3
    Uninstalling scipy-1.5.3:
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/scipy-1.5.3.dist-info/
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/scipy.libs/
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/scipy/
      Successfully uninstalled scipy-1.5.3
  Attempting uninstall: torch-sparse
    Found existing installation: torch-sparse 0.6.15+pt113cu117
    Uninstalling torch-sparse-0.6.15+pt113cu117:
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__init__.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/__init__.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_add.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_cat.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_coalesce.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_convert.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_diag.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_ego_sample.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_eye.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_matmul.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_metis.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_neighbor_sample.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_overload.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_permute.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_saint.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_sample.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_spmm.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_spspmm.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_storage.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_tensor.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/test_transpose.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/__pycache__/utils.cpython-38.pyc
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_add.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_cat.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_coalesce.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_convert.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_diag.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_ego_sample.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_eye.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_matmul.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_metis.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_neighbor_sample.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_overload.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_permute.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_saint.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_sample.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_spmm.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_spspmm.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_storage.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_tensor.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/test_transpose.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/test/utils.py
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/torch_sparse-0.6.15+pt113cu117.dist-info/
      Removing file or directory /mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/torch_sparse/
      Successfully uninstalled torch-sparse-0.6.15+pt113cu117
Successfully installed numpy-1.23.5 scipy-1.9.3 torch-scatter-2.1.0+pt113cu117 torch-sparse-0.6.15+pt113cu117

error is 

Traceback (most recent call last):
  File "src/run_GNN_TSP.py", line 9, in <module>
    from torch_geometric.nn import GCNConv, ChebConv  # noqa
  File "/mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/torch_geometric/__init__.py", line 4, in <module>
    import torch_geometric.data
  File "/mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/torch_geometric/data/__init__.py", line 1, in <module>
    from .data import Data
  File "/mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/torch_geometric/data/data.py", line 20, in <module>
    from torch_sparse import SparseTensor
  File "/mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/torch_sparse/__init__.py", line 19, in <module>
    torch.ops.load_library(spec.origin)
  File "/mnt/fsx-outputs-chipdesign/mehrlich/src/grand-tsp/.venv/lib/python3.8/site-packages/torch/_ops.py", line 573, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcusparse.so.11: cannot open shared object file: No such file or directory

my machine doesn't have nvcc installed and that should not be required for a binary install

rusty1s commented 1 year ago

We use both PyTorch and CUDA as shared libs. Installing cudatoolkit via conda should be sufficient to resolve.

Queuecumber commented 1 year ago

You should consider depending on https://pypi.org/project/nvidia-cusparse-cu11/ this would make the deployment a lot easier and it matches how pytorch itself is now packaged

rusty1s commented 1 year ago

That's cool. Is this limited to cu117?

Queuecumber commented 1 year ago

I think (can ask internally if you want) that it is going to be 11.7 and up

github-actions[bot] commented 1 year ago

This issue had no activity for 6 months. It will be closed in 2 weeks unless there is some new activity. Is this issue already resolved?