search

NVlabs / Deep_Object_Pose

Deep Object Pose Estimation (DOPE) – ROS inference (CoRL 2018)
Other
1.03k stars 287 forks source link

Loading DOPE model too slow #247

Open thehuongrbe opened 2 years ago

thehuongrbe commented 2 years ago

Hello, it tried run dope.launch with soup_60.pth but model loaded in over 500 seconds. i using GPU gerforce RTX 3060 and cuda version 11.4. help me, thank a lot

TontonTremblay commented 2 years ago

Damn dude that is long! I am not sure what to tell you, I have seen from system to system taking different amount of time, but never as much as this. I do not have a 3060 to test on though. So I am not sure how I can help you.

thehuongrbe commented 2 years ago

I try to wait until it's over and error Model loaded in 544.441734791 seconds. Running DOPE... (Listening to camera topic: '/camera/color/image_rect_color') Ctrl-C to stop THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=8 : invalid device function [ERROR] [1650507526.555579]: bad callback: <bound method Subscriber.callback of <message_filters.Subscriber object at 0x7fb2e1455150>> Traceback (most recent call last): File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback cb(msg) File "/opt/ros/melodic/lib/python2.7/dist-packages/message_filters/init.py", line 76, in callback self.signalMessage(msg) File "/opt/ros/melodic/lib/python2.7/dist-packages/message_filters/init.py", line 58, in signalMessage cb((msg + args)) File "/opt/ros/melodic/lib/python2.7/dist-packages/message_filters/init.py", line 225, in add self.signalMessage(msgs) File "/opt/ros/melodic/lib/python2.7/dist-packages/message_filters/init.py", line 58, in signalMessage cb((msg + args)) File "/home/nth/catkin_ws/src/DOPE-ROS-D435/dope/nodes/dope", line 248, in image_callback self.config_detect File "/home/nth/catkin_ws/src/DOPE-ROS-D435/dope/src/dope/inference/detector.py", line 251, in detect_object_in_image out, seg = net_model(image_torch) File "/home/nth/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(input, kwargs) File "/home/nth/.local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 141, in forward return self.module(*inputs[0], *kwargs[0]) File "/home/nth/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(input, kwargs) File "/home/nth/catkin_ws/src/DOPE-ROS-D435/dope/src/dope/inference/detector.py", line 91, in forward out1 = self.vgg(x) File "/home/nth/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, *kwargs) File "/home/nth/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/nth/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(input, **kwargs) File "/home/nth/.local/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 320, in forward self.padding, self.dilation, self.groups) RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

TontonTremblay commented 2 years ago

https://github.com/pytorch/pytorch/issues/15797 maybe this is related. I am not sure.