search

andyzeng / visual-pushing-grasping

Train robotic agents to learn to plan pushing and grasping actions for manipulation with deep reinforcement learning.
http://vpg.cs.princeton.edu/
BSD 2-Clause "Simplified" License
898 stars 315 forks source link

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED #22

Open suyashnigam opened 5 years ago

suyashnigam commented 5 years ago

Getting cuDNN error while trying to run the demo. Complete stack trace.

File "/home/nimble/VPG/visual-pushing-grasping/trainer.py", line 171, in forward output_prob, state_feat = self.model.forward(input_color_data, input_depth_data, is_volatile, specific_rotation) File "/home/nimble/VPG/visual-pushing-grasping/models.py", line 231, in forward flow_grid_before = F.affine_grid(Variable(affine_mat_before, requires_grad=False).cuda(), input_color_data.size()) File "/home/nimble/.local/lib/python2.7/site-packages/torch/nn/functional.py", line 2735, in affine_grid return vision.affine_grid_generator(theta, size) File "/home/nimble/.local/lib/python2.7/site-packages/torch/nn/_functions/vision.py", line 11, in affine_grid_generator ret = torch.cudnn_affine_grid_generator(theta, N, C, H, W) RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

cww97 commented 5 years ago

I have the same problem

Connected to simulation.
CUDA detected. Running with GPU acceleration.
Pre-trained model snapshot loaded from: /home/cww97/Documents/visual-pushing-grasping/downloads/vpg-original-sim-pretrained-10-obj.pth
Creating data logging session: /home/cww97/Documents/visual-pushing-grasping/logs/2019-06-17.00:31:08

Testing iteration: 0
Traceback (most recent call last):
  File "main.py", line 449, in <module>
    main(args)
  File "main.py", line 263, in main
    push_predictions, grasp_predictions, state_feat = trainer.forward(color_heightmap, valid_depth_heightmap, is_volatile=True)
  File "/home/cww97/Documents/visual-pushing-grasping/trainer.py", line 163, in forward
    output_prob, state_feat = self.model.forward(input_color_data, input_depth_data, is_volatile, specific_rotation)
  File "/home/cww97/Documents/visual-pushing-grasping/models.py", line 228, in forward
    flow_grid_before = F.affine_grid(Variable(affine_mat_before, requires_grad=False).cuda(), input_color_data.size())
  File "/home/cww97/anaconda3/envs/ur5/lib/python3.6/site-packages/torch/nn/functional.py", line 1516, in affine_grid
    return AffineGridGenerator.apply(theta, size)
  File "/home/cww97/anaconda3/envs/ur5/lib/python3.6/site-packages/torch/nn/_functions/vision.py", line 88, in forward
    torch._C._cudnn_affine_grid_generator_forward(theta, grid, N, C, H, W)
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED

I am running the code with a 2080,

Is this the problem of NVIDIA or the problem of the code

cww97 commented 5 years ago

Andy, could you tell us the exact version of your python?

some one said this may caused because of the difference between python3.6.2 and python 3.6.6

cww97 commented 5 years ago

@suyashnigam I fixed this by using the 0.4 branch with 2080, cuda10.0, torch=1.1(in fact every package the newest version). oh, three days passed

DKurome commented 4 years ago

cww97你弄好了吗，是因为nvidia的问题吗，我是2080ti cuda9 cudnn7.0.5,也是这个错误