HybridPose: 6D Object Pose Estimation under Hybrid Representation (CVPR 2020)
MIT License
415
stars
64
forks
source link
[traine_core.py] from pretrained weights -- RuntimeError: cublas runtime error : the GPU program failed to execute at /tmp/pip-req-build-58y_cjjl/aten/src/THC/THCBlas.cu:331 #86
(hp) mona@mona-ThinkStation-P7:~/HP/HybridPose$ LD_LIBRARY_PATH=lib/regressor:$LD_LIBRARY_PATH python src/train_core.py --load_dir /home/mona/HP/HybridPose/saved_weights/linemod/ape/checkpoints/0.001/199 --object_name ape
number of model parameters: 12959563
loading checkpoint from /home/mona/HP/HybridPose/saved_weights/linemod/ape/checkpoints/0.001/199
Successfully loaded model from /home/mona/HP/HybridPose/saved_weights/linemod/ape/checkpoints/0.001/199
/home/mona/anaconda3/envs/hp/lib/python3.7/site-packages/torch/nn/functional.py:1350: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")
Traceback (most recent call last):
File "src/train_core.py", line 114, in <module>
trainer.generate_data(val_loader)
File "./trainers/coretrainer.py", line 572, in generate_data
pts2d_pred_loc, pts2d_pred_var = self.vote_keypoints(pts2d_map_pred, mask_pred)
File "./trainers/coretrainer.py", line 324, in vote_keypoints
mean, var = estimate_voting_distribution_with_mean(mask, pts2d_map, mean)
File "/home/mona/HP/HybridPose/lib/ransac_voting_gpu_layer/ransac_voting_gpu.py", line 400, in estimate_voting_distribution_with_mean
cov=torch.matmul(diff_pts.transpose(2,3), weighted_diff_pts) # b,vn,2,2
RuntimeError: cublas runtime error : the GPU program failed to execute at /tmp/pip-req-build-58y_cjjl/aten/src/THC/THCBlas.cu:331
and my system has the following info:
(hp) mona@mona-ThinkStation-P7:~$ nvidia-smi
Tue Oct 24 08:35:58 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.12 Driver Version: 535.104.12 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA RTX 6000 Ada Gene... On | 00000000:52:00.0 On | Off |
| 30% 59C P2 73W / 300W | 4008MiB / 49140MiB | 1% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 2417 G /usr/lib/xorg/Xorg 452MiB |
| 0 N/A N/A 2597 G /usr/bin/gnome-shell 68MiB |
| 0 N/A N/A 3098 G ...AAAAAAAACAAAAAAAAAA= --shared-files 57MiB |
| 0 N/A N/A 3447 G ...irefox/3252/usr/lib/firefox/firefox 357MiB |
| 0 N/A N/A 8414 C python 608MiB |
| 0 N/A N/A 8704 C python 654MiB |
| 0 N/A N/A 8973 C python 692MiB |
| 0 N/A N/A 9484 G ...sion,SpareRendererForSitePerProcess 111MiB |
| 0 N/A N/A 12323 C python 890MiB |
+---------------------------------------------------------------------------------------+
and
(hp) mona@mona-ThinkStation-P7:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
(hp) mona@mona-ThinkStation-P7:~$ python
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.2.0'
>>> import torchvision
>>> torchvision.__version__
'0.4.0a0'
>>>
I get this error:
and my system has the following info:
and
and
and