Describe the bug
RuntimeError: CUSPARSE_STATUS_INVALID_VALUE at MinkowskiEngine/src/broadcast_kernel.cu:367
==========System==========
Linux-5.4.0-89-generic-x86_64-with-glibc2.17
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
3.8.13 (default, Mar 28 2022, 11:38:47)
[GCC 7.5.0]
==========Pytorch==========
1.10.1+cu102
torch.cuda.is_available(): True
==========NVIDIA-SMI==========
/usr/bin/nvidia-smi
Driver Version 460.56
CUDA Version 11.2
VBIOS Version 90.06.44.80.D4
Image Version G001.0000.02.04
==========NVCC==========
/usr/local/cuda-10.2/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89
==========CC==========
/usr/bin/c++
c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
==========MinkowskiEngine==========
0.5.4
MinkowskiEngine compiled with CUDA Support: True
NVCC version MinkowskiEngine is compiled: 10020
CUDART version MinkowskiEngine is compiled: 10020
Full output of the error:
Traceback (most recent call last):
File "train.py", line 36, in
do_train(params, debug=args.debug, visualize=args.visualize, device=device)
File "/home/client/git-back/Egonn/training/trainer.py", line 174, in do_train
loss.backward()
File "/home/client/miniconda3/envs/py3-mink/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/client/miniconda3/envs/py3-mink/lib/python3.8/site-packages/torch/autograd/init.py", line 154, in backward
Variable._execution_engine.run_backward(
File "/home/client/miniconda3/envs/py3-mink/lib/python3.8/site-packages/torch/autograd/function.py", line 199, in apply
return user_fn(self, *args)
File "/home/client/miniconda3/envs/py3-mink/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiBroadcast.py", line 87, in backward
grad_in_feat, grad_in_feat_glob = bw_fn(
RuntimeError: CUSPARSE_STATUS_INVALID_VALUE at /home/client/library/MinkowskiEngine/src/broadcast_kernel.cu:367
Describe the bug RuntimeError: CUSPARSE_STATUS_INVALID_VALUE at MinkowskiEngine/src/broadcast_kernel.cu:367
==========System========== Linux-5.4.0-89-generic-x86_64-with-glibc2.17 DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS" 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0] ==========Pytorch========== 1.10.1+cu102 torch.cuda.is_available(): True ==========NVIDIA-SMI========== /usr/bin/nvidia-smi Driver Version 460.56 CUDA Version 11.2 VBIOS Version 90.06.44.80.D4 Image Version G001.0000.02.04 ==========NVCC========== /usr/local/cuda-10.2/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Wed_Oct_23_19:24:38_PDT_2019 Cuda compilation tools, release 10.2, V10.2.89 ==========CC========== /usr/bin/c++ c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
==========MinkowskiEngine========== 0.5.4 MinkowskiEngine compiled with CUDA Support: True NVCC version MinkowskiEngine is compiled: 10020 CUDART version MinkowskiEngine is compiled: 10020
Full output of the error: Traceback (most recent call last): File "train.py", line 36, in
do_train(params, debug=args.debug, visualize=args.visualize, device=device)
File "/home/client/git-back/Egonn/training/trainer.py", line 174, in do_train
loss.backward()
File "/home/client/miniconda3/envs/py3-mink/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/client/miniconda3/envs/py3-mink/lib/python3.8/site-packages/torch/autograd/init.py", line 154, in backward
Variable._execution_engine.run_backward(
File "/home/client/miniconda3/envs/py3-mink/lib/python3.8/site-packages/torch/autograd/function.py", line 199, in apply
return user_fn(self, *args)
File "/home/client/miniconda3/envs/py3-mink/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiBroadcast.py", line 87, in backward
grad_in_feat, grad_in_feat_glob = bw_fn(
RuntimeError: CUSPARSE_STATUS_INVALID_VALUE at /home/client/library/MinkowskiEngine/src/broadcast_kernel.cu:367