search

chrischoy / DeepGlobalRegistration

[CVPR 2020 Oral] A differentiable framework for 3D registration
Other
470 stars 86 forks source link

RuntimeError: CUBLAS_STATUS_EXECUTION_FAILED at /tmp/pip-install-z25sch0o/minkowskiengine_101b60a34a8c4b73a1c33ae9cb85e161/src/math_functions_gpu.cu:46 #38

Open chalth opened 2 years ago

chalth commented 2 years ago

Traceback (most recent call last): File "train.py", line 79, in main(config) File "train.py", line 57, in main trainer.train() File "/home/lthach/DeepGlobalRegistration-master (1)/core/trainer.py", line 135, in train self._train_epoch(epoch) File "/home/lthach/DeepGlobalRegistration-master (1)/core/trainer.py", line 266, in _train_epoch loss.backward() File "/home/lthach/anaconda3/envs/dgr/lib/python3.7/site-packages/torch/tensor.py", line 245, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/lthach/anaconda3/envs/dgr/lib/python3.7/site-packages/torch/autograd/init.py", line 147, in backward allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag File "/home/lthach/anaconda3/envs/dgr/lib/python3.7/site-packages/torch/autograd/function.py", line 89, in apply return self._forward_cls.backward(self, *args) # type: ignore File "/home/lthach/anaconda3/envs/dgr/lib/python3.7/site-packages/MinkowskiEngine/MinkowskiConvolution.py", line 111, in backward coordinate_manager._manager, RuntimeError: CUBLAS_STATUS_EXECUTION_FAILED at /tmp/pip-install-z25sch0o/minkowskiengine_101b60a34a8c4b73a1c33ae9cb85e161/src/math_functions_gpu.cu:46

Hi, when i try to train the DGR model , i meet this problem , do you know how to solve it ? Thanks!