search

ylabbe / cosypose

Code for "CosyPose: Consistent multi-view multi-object 6D pose estimation", ECCV 2020.
MIT License
301 stars 89 forks source link

CUDA RuntimeError in run_cosypose_eval #70

Closed dkguo closed 2 years ago

dkguo commented 2 years ago

Hello,

I encountered cuda runtime error when I run:

python -m cosypose.scripts.run_cosypose_eval --config tless-siso —debug

Does anyone know how to solve this? Thank you.

Error message:

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/THC/generic/THCTensorMathPointwise.cu line=281 error=209 : no kernel image is available for execution on the device

Traceback (most recent call last):

File "/home/gdk/anaconda3/envs/cosypose/lib/python3.7/runpy.py", line 193, in _run_module_as_main

"__main__", mod_spec)

File "/home/gdk/anaconda3/envs/cosypose/lib/python3.7/runpy.py", line 85, in _run_code

exec(code, run_globals)

File "/media/gdk/Data/code/cosypose/cosypose/scripts/run_cosypose_eval.py", line 491, in 

main()

File "/media/gdk/Data/code/cosypose/cosypose/scripts/run_cosypose_eval.py", line 390, in main

preds = pred_runner.get_predictions(**pred_kwargs_n)

File "/media/gdk/Data/code/cosypose/cosypose/evaluation/pred_runner/multiview_predictions.py", line 137, in get_predictions

n_refiner_iterations=n_refiner_iterations,

File "/media/gdk/Data/code/cosypose/cosypose/integrated/pose_predictor.py", line 90, in get_predictions

n_iterations=n_coarse_iterations)

File "/home/gdk/anaconda3/envs/cosypose/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad

return func(*args, **kwargs)

File "/media/gdk/Data/code/cosypose/cosypose/integrated/pose_predictor.py", line 45, in batched_model_predictions

n_iterations=n_iterations, labels=labels)

File "/home/gdk/anaconda3/envs/cosypose/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call

result = self.forward(*input, **kwargs)

File "/media/gdk/Data/code/cosypose/cosypose/models/pose.py", line 106, in forward

model_outputs = self.net_forward(x)

File "/media/gdk/Data/code/cosypose/cosypose/models/pose.py", line 82, in net_forward

x = self.backbone(x)

File "/home/gdk/anaconda3/envs/cosypose/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call

result = self.forward(*input, **kwargs)

File "/media/gdk/Data/code/cosypose/cosypose/models/efficientnet.py", line 196, in forward

x = self.extract_features(inputs)

File "/media/gdk/Data/code/cosypose/cosypose/models/efficientnet.py", line 178, in extract_features

x = self._swish(self._bn0(self._conv_stem(inputs)))

File "/home/gdk/anaconda3/envs/cosypose/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call

result = self.forward(*input, **kwargs)

File "/media/gdk/Data/code/cosypose/cosypose/models/efficientnet_utils.py", line 53, in forward

return SwishImplementation.apply(x)

File "/media/gdk/Data/code/cosypose/cosypose/models/efficientnet_utils.py", line 40, in forward

result = i * torch.sigmoid(i)

RuntimeError: cuda runtime error (209) : no kernel image is available for execution on the device at /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/THC/generic/THCTensorMathPointwise.cu:281

0%| | 0/1 [01:08<?, ?it/s]

Destroy EGL OpenGL window.

dkguo commented 2 years ago

Solved by install newest version of pytorch conda install pytorch torchvision cudatoolkit=11.3 -c pytorch I’m using 3060 gpu, so pytorch 1.3.1 doesn’t support it anymore.