rusty1s / pytorch_cluster

PyTorch Extension Library of Optimized Graph Cluster Algorithms
MIT License
816 stars 146 forks source link

Help!RuntimeError: Not complied with CUDA support #186

Closed MsCongs closed 5 months ago

MsCongs commented 1 year ago

Please help me about this issue. I have error like :

The following operation failed in the TorchScript interpreter. Traceback of TorchScript (most recent call last): File "/home/xucg/anaconda3/envs/diffdock/lib/python3.9/site-packages/torch_cluster/radius.py", line 118, in radius_graph

assert flow in ['source_to_target', 'target_to_source']
edge_index = radius(x, x, r, batch, batch,
             ~~~~~~ <--- HERE
                    max_num_neighbors if loop else max_num_neighbors + 1,
                    num_workers)

File "/home/xucg/anaconda3/envs/diffdock/lib/python3.9/site-packages/torch_cluster/radius.py", line 72, in radius ptr_y = torch.bucketize(arange, batch_y)

return torch.ops.torch_cluster.radius(x, y, ptr_x, ptr_y, r,
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
                                      max_num_neighbors, num_workers)

RuntimeError: Not compiled with CUDA support

My conda list is like following:

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
absl-py 1.4.0 pypi_0 pypi aiohttp 3.8.5 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi antlr4-python3-runtime 4.9.3 pypi_0 pypi async-timeout 4.0.3 pypi_0 pypi attrs 23.1.0 pypi_0 pypi biopandas 0.4.1 pypi_0 pypi biopython 1.81 pypi_0 pypi blas 1.0 mkl
brotlipy 0.7.0 py39h27cfd23_1003
bzip2 1.0.8 h7b6447c_0
ca-certificates 2023.05.30 h06a4308_0
certifi 2023.7.22 py39h06a4308_0
cffi 1.15.1 py39h5eee18b_3
charset-normalizer 3.2.0 pypi_0 pypi contextlib2 21.6.0 pypi_0 pypi contourpy 1.1.0 pypi_0 pypi cryptography 41.0.2 py39h22a60cf_0
cuda-cccl 11.7.58 hc415cf5_0 nvidia/label/cuda-11.7.0 cuda-command-line-tools 11.7.0 0 nvidia/label/cuda-11.7.0 cuda-compiler 11.7.0 0 nvidia/label/cuda-11.7.0 cuda-cudart 11.7.99 0 nvidia cuda-cudart-dev 11.7.60 h6a7c232_0 nvidia/label/cuda-11.7.0 cuda-cuobjdump 11.7.50 h28cc80a_0 nvidia/label/cuda-11.7.0 cuda-cupti 11.7.101 0 nvidia cuda-cuxxfilt 11.7.50 hb365495_0 nvidia/label/cuda-11.7.0 cuda-documentation 11.7.50 0 nvidia/label/cuda-11.7.0 cuda-driver-dev 11.7.60 0 nvidia/label/cuda-11.7.0 cuda-gdb 11.7.50 h4a0ac72_0 nvidia/label/cuda-11.7.0 cuda-libraries 11.7.1 0 nvidia cuda-libraries-dev 11.7.0 0 nvidia/label/cuda-11.7.0 cuda-memcheck 11.7.50 hc446b2b_0 nvidia/label/cuda-11.7.0 cuda-nsight 11.7.50 0 nvidia/label/cuda-11.7.0 cuda-nsight-compute 11.7.0 0 nvidia/label/cuda-11.7.0 cuda-nvcc 11.7.64 0 nvidia/label/cuda-11.7.0 cuda-nvdisasm 11.7.50 h5bd0695_0 nvidia/label/cuda-11.7.0 cuda-nvml-dev 11.7.50 h3af1343_0 nvidia/label/cuda-11.7.0 cuda-nvprof 11.7.50 h7a2404d_0 nvidia/label/cuda-11.7.0 cuda-nvprune 11.7.50 h7add7b4_0 nvidia/label/cuda-11.7.0 cuda-nvrtc 11.7.99 0 nvidia cuda-nvrtc-dev 11.7.50 heada363_0 nvidia/label/cuda-11.7.0 cuda-nvtx 11.7.91 0 nvidia cuda-nvvp 11.7.50 hd2289d5_0 nvidia/label/cuda-11.7.0 cuda-runtime 11.7.1 0 nvidia cuda-sanitizer-api 11.7.50 hb424887_0 nvidia/label/cuda-11.7.0 cuda-toolkit 11.7.0 0 nvidia/label/cuda-11.7.0 cuda-tools 11.7.0 0 nvidia/label/cuda-11.7.0 cuda-visual-tools 11.7.0 0 nvidia/label/cuda-11.7.0 cudatoolkit 11.5.1 hcf5317a_9 nvidia cycler 0.11.0 pypi_0 pypi deepspeed 0.5.9 pypi_0 pypi dllogger 1.0.0 pypi_0 pypi dm-tree 0.1.8 pypi_0 pypi e3nn 0.5.1 pypi_0 pypi einops 0.6.1 pypi_0 pypi fair-esm 2.0.0 pypi_0 pypi ffmpeg 4.3 hf484d3e_0 pytorch filelock 3.9.0 py39h06a4308_0
fonttools 4.42.0 pypi_0 pypi freetype 2.12.1 h4a9f257_0
frozenlist 1.4.0 pypi_0 pypi fsspec 2023.6.0 pypi_0 pypi gds-tools 1.3.0.44 0 nvidia/label/cuda-11.7.0 giflib 5.2.1 h5eee18b_3
gmp 6.2.1 h295c915_3
gmpy2 2.1.2 py39heeb90bb_0
gnutls 3.6.15 he1e5248_0
hjson 3.1.0 pypi_0 pypi idna 3.4 py39h06a4308_0
importlib-resources 6.0.1 pypi_0 pypi intel-openmp 2023.1.0 hdb19cb5_46305
jinja2 3.1.2 py39h06a4308_0
joblib 1.3.2 pypi_0 pypi jpeg 9e h5eee18b_1
kiwisolver 1.4.4 pypi_0 pypi lame 3.100 h7b6447c_0
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libcublas 11.10.3.66 0 nvidia libcublas-dev 11.10.1.25 h0c8ac2b_0 nvidia/label/cuda-11.7.0 libcufft 10.7.2.124 h4fbf590_0 nvidia libcufft-dev 10.7.2.50 h59a5ac8_0 nvidia/label/cuda-11.7.0 libcufile 1.7.1.12 0 nvidia libcufile-dev 1.3.0.44 0 nvidia/label/cuda-11.7.0 libcurand 10.3.3.129 0 nvidia libcurand-dev 10.2.10.50 hd49a9cd_0 nvidia/label/cuda-11.7.0 libcusolver 11.4.0.1 0 nvidia libcusolver-dev 11.3.5.50 hc6eba6f_0 nvidia/label/cuda-11.7.0 libcusparse 11.7.4.91 0 nvidia libcusparse-dev 11.7.3.50 hc644b96_0 nvidia/label/cuda-11.7.0 libdeflate 1.17 h5eee18b_0
libffi 3.4.4 h6a678d5_0
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libiconv 1.16 h7f8727e_2
libidn2 2.3.4 h5eee18b_0
libnpp 11.7.4.75 0 nvidia libnpp-dev 11.7.3.21 hb6476a9_0 nvidia/label/cuda-11.7.0 libnvjpeg 11.8.0.2 0 nvidia libnvjpeg-dev 11.7.2.34 h2e48410_0 nvidia/label/cuda-11.7.0 libpng 1.6.39 h5eee18b_0
libstdcxx-ng 11.2.0 h1234567_1
libtasn1 4.19.0 h5eee18b_0
libtiff 4.5.0 h6a678d5_2
libunistring 0.9.10 h27cfd23_0
libuv 1.44.2 h5eee18b_0
libwebp 1.2.4 h11a3e52_1
libwebp-base 1.2.4 h5eee18b_1
lightning-utilities 0.9.0 pypi_0 pypi lz4-c 1.9.4 h6a678d5_0
markupsafe 2.1.3 pypi_0 pypi matplotlib 3.7.2 pypi_0 pypi mkl 2023.1.0 h213fc3f_46343
mkl-service 2.4.0 py39h5eee18b_1
mkl_fft 1.3.6 py39h417a72b_1
mkl_random 1.2.2 py39h417a72b_1
ml-collections 0.1.1 pypi_0 pypi mpc 1.1.0 h10f8cd9_1
mpfr 4.0.2 hb69a4c5_1
mpmath 1.3.0 py39h06a4308_0
multidict 6.0.4 pypi_0 pypi ncurses 6.4 h6a678d5_0
nettle 3.7.3 hbbd107a_1
networkx 3.1 py39h06a4308_0
ninja 1.11.1 pypi_0 pypi nsight-compute 2022.2.0.13 0 nvidia/label/cuda-11.7.0 numpy 1.25.2 py39h5f9d8c6_0
numpy-base 1.25.2 py39hb5e798b_0
omegaconf 2.3.0 pypi_0 pypi openfold 1.0.0 pypi_0 pypi openh264 2.1.1 h4ff587b_0
openssl 3.0.10 h7f8727e_0
opt-einsum 3.3.0 pypi_0 pypi opt-einsum-fx 0.1.4 pypi_0 pypi packaging 23.1 pypi_0 pypi pandas 2.0.3 pypi_0 pypi pillow 10.0.0 pypi_0 pypi pip 23.2.1 py39h06a4308_0
psutil 5.9.5 pypi_0 pypi py-cpuinfo 9.0.0 pypi_0 pypi pycparser 2.21 pyhd3eb1b0_0
pyopenssl 23.2.0 py39h06a4308_0
pyparsing 3.0.9 pypi_0 pypi pysocks 1.7.1 py39h06a4308_0
python 3.9.17 h955ad1f_0
python-dateutil 2.8.2 pypi_0 pypi pytorch 1.11.0 py3.9_cuda11.5_cudnn8.3.2_0 pytorch pytorch-cuda 11.7 h778d358_5 pytorch pytorch-lightning 2.0.6 pypi_0 pypi pytorch-mutex 1.0 cuda pytorch pytz 2023.3 pypi_0 pypi pyyaml 6.0.1 pypi_0 pypi rdkit-pypi 2022.9.5 pypi_0 pypi readline 8.2 h5eee18b_0
requests 2.31.0 py39h06a4308_0
scikit-learn 1.3.0 pypi_0 pypi scipy 1.11.1 pypi_0 pypi setuptools 68.0.0 py39h06a4308_0
six 1.16.0 pypi_0 pypi spyrmsd 0.5.2 pypi_0 pypi sqlite 3.41.2 h5eee18b_0
sympy 1.12 pypi_0 pypi tbb 2021.8.0 hdb19cb5_0
threadpoolctl 3.2.0 pypi_0 pypi tk 8.6.12 h1ccaba5_0
torch-cluster 1.6.1 pypi_0 pypi torch-geometric 2.3.1 pypi_0 pypi torch-scatter 2.1.1 pypi_0 pypi torch-sparse 0.6.14 pypi_0 pypi torch-spline-conv 1.2.2 pypi_0 pypi torchaudio 0.11.0 py39_cu115 pytorch torchmetrics 1.0.3 pypi_0 pypi torchtriton 2.0.0 py39 pytorch torchvision 0.12.0 py39_cu115 pytorch tqdm 4.66.1 pypi_0 pypi triton 1.0.0 pypi_0 pypi typing_extensions 4.7.1 py39h06a4308_0
tzdata 2023.3 pypi_0 pypi urllib3 2.0.4 pypi_0 pypi wheel 0.38.4 py39h06a4308_0
xz 5.4.2 h5eee18b_0
yarl 1.9.2 pypi_0 pypi zipp 3.16.2 pypi_0 pypi zlib 1.2.13 h5eee18b_0
zstd 1.5.5 hc292b87_0

rusty1s commented 1 year ago

Looks like you installed torch-cluster without CUDA support. How did you install it in the first-place?

MsCongs commented 1 year ago

Looks like you installed torch-cluster without CUDA support. How did you install it in the first-place?

I install it using this command 'pip install torch-cluster -f https://data.pyg.org/whl/torch-1.11.0+cu117.html'

I think it should be right but turned out to be wrong.

rusty1s commented 1 year ago

If you look at the installation path of torch-cluster in site-packages/, do you see the binaries for CUDA? This should be called something like *_cuda.

MsCongs commented 1 year ago

If you look at the installation path of torch-cluster in site-packages/, do you see the binaries for CUDA? This should be called something like *_cuda.

No, I don't see the binaries for CUDA in the path of torch-cluster. How could I add this binaries in this path?

MsCongs commented 1 year ago

I solved this problem! I uninstalled the torch-cluster and installed it again using this command "conda install pytorch-cluster -c pyg" instead of original "pip install torch-cluster -f https://data.pyg.org/whl/torch-1.11.0+cu117.html" . Thank you for your help!

KrOverdulve commented 1 year ago

In my case the CPU version of torch_cluster was in my pip cache causing it to prefer that cached version to the one available at https://data.pyg.org/whl/torch-1.11.0+cu117.html. Adding the --no-cache-dir solved it for me. The command then becomes

pip install torch-cluster -f https://data.pyg.org/whl/torch-1.11.0+cu117.html --no-cache-dir
critopadolf commented 11 months ago

Unfortunately, while cuda seems to work fine in my pytorch install, I am getting the same "Not compiled with CUDA support" error as well.

torch version: 2.0.1+cu117 pip cmd: pip install torch-cluster -f https://data.pyg.org/whl/2.0.1+cu117.html --no-cache-dir

Looking in links: https://data.pyg.org/whl/2.0.1+cu117.html
Collecting torch-cluster
  Downloading torch_cluster-1.6.2.tar.gz (54 kB)
rusty1s commented 11 months ago

Can you try to manually remove the @torch.jit.script decorators in torch_cluster/radius.py in your installed version and report whether this resolves the issue?

critopadolf commented 11 months ago

Yes I can try it! Actually looking at the package I see there is only _radius_cpu.so. Could it be caused by cuda not being in my path? torch installed with cuda just fine so I don't know what's up here.

Thanks!

import torch
import torch_cluster
torch_cluster.radius(x = torch.FloatTensor([[0,0,0]]).cuda(), y = torch.FloatTensor([[0,0,0]]).cuda(), r = 0.01)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[5], line 3
      1 import torch
      2 import torch_cluster
----> 3 torch_cluster.radius(x = torch.FloatTensor([[0,0,0]]).cuda(), y = torch.FloatTensor([[0,0,0]]).cuda(), r = 0.01)

RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/home/user0/envs/pytorch-gpu-geometric/lib/python3.10/site-packages/torch_cluster/radius.py", line 83, in radius
        ptr_y = torch.bucketize(arange, batch_y)

    return torch.ops.torch_cluster.radius(x, y, ptr_x, ptr_y, r,
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
                                          max_num_neighbors, num_workers)
RuntimeError: Not compiled with CUDA support
critopadolf commented 11 months ago

pip install torch-cluster -f https://pytorch-geometric.com/whl/torch-2.0.1+cu117.html

This fixed it!

github-actions[bot] commented 5 months ago

This issue had no activity for 6 months. It will be closed in 2 weeks unless there is some new activity. Is this issue already resolved?