torch_sparse tells to be compiled for CUDA 10.1 even if I asked for 10.2

LucaMarconato commented 3 years ago

🐛 Bug

For some reasons in my machine I can't install torch 1.7.0, so I am trying either to use torch_geometric with torch 1.6.0 or 1.5.0. ~~Both, unfortunately, don't work for me~~ Edit: torch 1.6.0 works for me, but torch 1.5.0 no 😟 In the following I will describe the bug that I get with torch 1.5.0.

To Reproduce

I install torch on my Linux machine with the versions suggested in the torch website. I use

ENV_NAME=my_env
mamba create -c conda-forge -c bioconda -n $ENV_NAME snakemake python==3.8 -y
conda activate $ENV_NAME
mamba install pytorch==1.5.0 torchvision==0.6.0 cudatoolkit=10.2 -c pytorch -y

The versions are installed correctly, as confirmed by

python -c "import torch; print('torch version:', torch.__version__); assert torch.__version__ == '1.5.0'"
python -c "import torch; print('cuda version:', torch.version.cuda); assert torch.version.cuda == '10.2'"

I install torch geometric with

TORCH=1.5.0
CUDA=cu102
pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pip install torch-sparse -f https://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pip install torch-cluster -f https://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pip install torch-spline-conv -f https://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pip install torch-geometric

The installation goes smooth.

Nonetheless, when I run my torch code I get an error.

The code triggering the error:

from torch_geometric.data import Data, InMemoryDataset

The error:

Traceback (most recent call last):
  File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/runpy.py", line 192, in _run_module_as_main
...
    from spatial_uzh.ds.graphs import compute_graphs
  File "/data/l989o/tests/spatial_uzh_test_pipeline/spatial_uzh/ds/graphs.py", line 3, in <module>
    from torch_geometric.data import Data, InMemoryDataset
  File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_geometric/__init__.py", line 2, in <module>
    import torch_geometric.nn
  File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_geometric/nn/__init__.py", line 2, in <module>
    from .data_parallel import DataParallel
  File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_geometric/nn/data_parallel.py", line 5, in <module>
    from torch_geometric.data import Batch
  File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_geometric/data/__init__.py", line 1, in <module>
    from .data import Data
  File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_geometric/data/data.py", line 8, in <module>
    from torch_sparse import coalesce, SparseTensor
  File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_sparse/__init__.py", line 27, in <module>
    raise RuntimeError(
RuntimeError: Detected that PyTorch and torch_sparse were compiled with different CUDA versions. PyTorch has CUDA version 10.2 and torch_sparse has CUDA version 10.1. Please reinstall the torch_sparse that matches your PyTorch install.

Expected behavior

Torch geometric code should work. Until yesterday everything worked, but I had to delete my conda env and I didn't write down the versions that I was using. Now I can't create a working env anymore.

Environment

OS: RETTY_NAME="CentOS Linux 7 (Core)" Python version: 3.8.0 PyTorch version: 1.6.0 CUDA/cuDNN version: 10.2 GCC version: 7.2.0 How did you try to install PyTorch Geometric and its extensions (wheel, source): wheel Any other relevant information:

Additional context

Thanks for your help!

rusty1s commented 3 years ago

This is weird. I cannot reproduce it on my machine :( Not sure what's causing it. Does it work when using the "old" method with pip<20.3?

pip install torch-sparse==latest+cu102 -f https://pytorch-geometric.com/whl/torch-1.6.0.html

There also might be some caching issues. How about using pip with --no-cache-dir?

LucaMarconato commented 3 years ago

I have tested three scenarios:

I have tried reproducing the bug I had, since two hours ago I did a conda clean --all, maybe this contributed in changing the error message that I get now:

>>> from torch_geometric.data import Data, InMemoryDataset
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_geometric/__init__.py", line 2, in <module>
import torch_geometric.nn
File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_geometric/nn/__init__.py", line 2, in <module>
from .data_parallel import DataParallel
File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_geometric/nn/data_parallel.py", line 5, in <module>
from torch_geometric.data import Batch
File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_geometric/data/__init__.py", line 1, in <module>
from .data import Data
File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_geometric/data/data.py", line 8, in <module>
from torch_sparse import coalesce, SparseTensor
File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_sparse/__init__.py", line 12, in <module>
torch.ops.load_library(importlib.machinery.PathFinder().find_spec(
File "/home/l989o/.local/lib/python3.8/site-packages/torch/_ops.py", line 105, in load_library
ctypes.CDLL(path)
File "/data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/ctypes/__init__.py", line 369, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /data/l989o/miniconda3/envs/spatial_uzh_test/lib/python3.8/site-packages/torch_sparse/_version.so: undefined symbol: _ZN3c105ErrorC1ENS_14SourceLocationERKSs

Using pip 20.2.4 and then running the old method, which if I am right is

pip install torch-scatter==latest+${CUDA} -f https://pytorch-geometric.com/whl/torch-${TORCH}.html
pip install torch-sparse==latest+${CUDA} -f https://pytorch-geometric.com/whl/torch-${TORCH}.html
pip install torch-cluster==latest+${CUDA} -f https://pytorch-geometric.com/whl/torch-${TORCH}.html
pip install torch-spline-conv==latest+${CUDA} -f https://pytorch-geometric.com/whl/torch-${TORCH}.html
pip install torch-geometric

makes running from torch_geometric.data import Data, InMemoryDataset without error.

Finally I have tried the scenario in the first bullet point, but this time adding --no-cache-dir. and actually it works!

Thanks for your help!

rusty1s commented 3 years ago

Awesome :)

pyg-team / pytorch_geometric