search

NVlabs / CenterPose

Single-Stage Keypoint-based Category-level Object Pose Estimation from an RGB Image (ICRA 2022)
Other
279 stars 33 forks source link

RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:425 #14

Open dbdxnuliba opened 1 year ago

dbdxnuliba commented 1 year ago

(CenterPose) dell1804@dell1804-G3-3590:~/center_pose_ws/CenterPose/src$ python demo.py --demo ../data/book.jpg --arch dlav1_34 --load_model ../models/CenterPose/book_v1_140.pth /home/dell1804/anaconda3/envs/CenterPose/lib/python3.6/site-packages/sklearn/utils/linearassignment.py:22: FutureWarning: The linearassignment module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead. FutureWarning) Fix size testing. training chunk_sizes: [1] The output will be saved to /home/dell1804/center_pose_ws/CenterPose/src/lib/../../exp/object_pose/default heads {'hm': 1, 'wh': 2, 'hps': 16, 'reg': 2, 'hm_hp': 8, 'hp_offset': 2, 'scale': 3} Creating model... loaded ../models/CenterPose/book_v1_140.pth, epoch 140 THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=11 : invalid argument Traceback (most recent call last): File "demo.py", line 156, in demo(opt, meta) File "demo.py", line 83, in demo ret = detector.run(image_name, meta_inp=meta) File "/home/dell1804/center_pose_ws/CenterPose/src/lib/detectors/base_detector.py", line 474, in run images, self.pre_images, pre_hms, pre_hm_hp, pre_inds, return_time=True) File "/home/dell1804/center_pose_ws/CenterPose/src/lib/detectors/object_pose.py", line 135, in process output = self.model(images, pre_images, pre_hms, pre_hm_hp)[-1] File "/home/dell1804/anaconda3/envs/CenterPose/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, kwargs) File "/home/dell1804/center_pose_ws/CenterPose/src/lib/models/networks/pose_dla_dcn.py", line 531, in forward x = self.dla_up(x) File "/home/dell1804/anaconda3/envs/CenterPose/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, *kwargs) File "/home/dell1804/center_pose_ws/CenterPose/src/lib/models/networks/pose_dla_dcn.py", line 441, in forward ida(layers, len(layers) - i - 2, len(layers)) File "/home/dell1804/anaconda3/envs/CenterPose/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, kwargs) File "/home/dell1804/center_pose_ws/CenterPose/src/lib/models/networks/pose_dla_dcn.py", line 415, in forward layers[i] = upsample(project(layers[i])) File "/home/dell1804/anaconda3/envs/CenterPose/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, *kwargs) File "/home/dell1804/center_pose_ws/CenterPose/src/lib/models/networks/pose_dla_dcn.py", line 387, in forward x = self.conv(x) File "/home/dell1804/anaconda3/envs/CenterPose/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, **kwargs) File "/home/dell1804/center_pose_ws/CenterPose/src/lib/models/networks/DCNv2/dcn_v2.py", line 128, in forward self.deformable_groups) File "/home/dell1804/center_pose_ws/CenterPose/src/lib/models/networks/DCNv2/dcn_v2.py", line 31, in forward ctx.deformable_groups) RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:425

(CenterPose) dell1804@dell1804-G3-3590:~/center_pose_ws/CenterPose/src$ nvidia-smi /usr/bin/nvidia-modprobe: unrecognized option: "-s"

ERROR: Invalid commandline, please run /usr/bin/nvidia-modprobe --help for usage information.

/usr/bin/nvidia-modprobe: unrecognized option: "-s"

ERROR: Invalid commandline, please run /usr/bin/nvidia-modprobe --help for usage information.

Sun Oct 9 11:10:47 2022
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.141.03 Driver Version: 470.141.03 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A | | N/A 50C P8 2W / N/A | 1083MiB / 3911MiB | 14% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 1165 G /usr/lib/xorg/Xorg 226MiB | | 0 N/A N/A 1846 G /usr/bin/gnome-shell 50MiB | | 0 N/A N/A 3778 G ...428520904353170423,131072 72MiB | | 0 N/A N/A 24592 C python 727MiB | +-----------------------------------------------------------------------------+

Python 3.6.15 | packaged by conda-forge | (default, Dec 3 2021, 18:49:41) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import torch torch.version '1.1.0'

dbdxnuliba commented 1 year ago

I has fixed the issue, by use torch==1.4.0 torchvision==0.5.0

OliviaZhang1996 commented 1 year ago

If the torch and torchvision changes, the version of DCNv2 should also be changed. The correct version of DCNv2 can be downloaded from the related branch. For example, if your torchvision=1.6.0, then you should download https://github.com/lucasjinreal/DCNv2_latest/archive/refs/heads/pytorch1.6.zip to replace the existing DCNv2 code. Then run sh ./make.sh. The make.sh can run smoothly after deleting the top two lines.