RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

[YangHe712]

Hello when I do the training for people detection, the following error comes:

Traceback (most recent call last): File "./pytorch/train.py", line 643, in fire.Fire() File "/home/yangyang/anaconda3/envs/dl-second/lib/python3.6/site-packages/fire/core.py", line 127, in Fire component_trace = _Fire(component, args, context, name) File "/home/yangyang/anaconda3/envs/dl-second/lib/python3.6/site-packages/fire/core.py", line 366, in _Fire component, remaining_args) File "/home/yangyang/anaconda3/envs/dl-second/lib/python3.6/site-packages/fire/core.py", line 542, in _CallCallable result = fn(*varargs, kwargs) File "./pytorch/train.py", line 398, in train raise e File "./pytorch/train.py", line 245, in train ret_dict = net(example_torch) File "/home/yangyang/anaconda3/envs/dl-second/lib/python3.6/site-packages/torch/nn/modules/module.py", line 479, in call result = self.forward(*input, *kwargs) File "/home/yangyang/second.pytorch/second/pytorch/models/voxelnet.py", line 671, in forward voxel_features = self.voxel_feature_extractor(voxels, num_points) File "/home/yangyang/anaconda3/envs/dl-second/lib/python3.6/site-packages/torch/nn/modules/module.py", line 479, in call result = self.forward(input, kwargs) File "/home/yangyang/second.pytorch/second/pytorch/models/voxelnet.py", line 140, in forward x = self.vfe1(features) File "/home/yangyang/anaconda3/envs/dl-second/lib/python3.6/site-packages/torch/nn/modules/module.py", line 479, in call result = self.forward(*input, *kwargs) File "/home/yangyang/second.pytorch/second/pytorch/models/voxelnet.py", line 82, in forward x = self.norm(x.permute(0, 2, 1).contiguous()).permute(0, 2, File "/home/yangyang/anaconda3/envs/dl-second/lib/python3.6/site-packages/torch/nn/modules/module.py", line 479, in call result = self.forward(input, **kwargs) File "/home/yangyang/anaconda3/envs/dl-second/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 67, in forward exponential_average_factor, self.eps) File "/home/yangyang/anaconda3/envs/dl-second/lib/python3.6/site-packages/torch/nn/functional.py", line 1429, in batch_norm training, momentum, eps, torch.backends.cudnn.enabled RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

my env is: CUDA 9.0 on a Ubuntu 16.04 workstation.

BTW, the problem comes after the old GTX 960 graphic card was replaced by the new RTX 2070. Everything was OK before that.

[traveller59]

The RTX 2070 seems need CUDA 10, you can build pytorch with cuda 10 from source. Cound you run simple pytorch network training with cuda 9 + RTX?

[YangHe712]

The RTX 2070 seems need CUDA 10, you can build pytorch with cuda 10 from source. Cound you run simple pytorch network training?

I also notice this problem. CUDA 10 supports turing architecture. But we need to use CUDA 9 because of the SparseConvNet package, don't we? According to the Nvidia's reply, they guess CUDA 9 should work with Turing (ref: https://devtalk.nvidia.com/default/topic/1044361/value-quot-sm_75-quot-not-defined-with-cuda-9-0-installed-on-rtx-2070/).

Also, I run the following simple pytorch example with GPU:

import torch

dtype = torch.float

device = torch.device("cpu")

device = torch.device("cuda:0") # Uncomment this to run on GPU

N is batch size; D_in is input dimension;

H is hidden dimension; D_out is output dimension.

N, D_in, H, D_out = 64, 1000, 100, 10

Create random Tensors to hold input and outputs.

Setting requires_grad=False indicates that we do not need to compute gradients

with respect to these Tensors during the backward pass.

x = torch.randn(N, D_in, device=device, dtype=dtype) y = torch.randn(N, D_out, device=device, dtype=dtype)

Create random Tensors for weights.

Setting requires_grad=True indicates that we want to compute gradients with

respect to these Tensors during the backward pass.

w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True) w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-6 for t in range(500):

Forward pass: compute predicted y using operations on Tensors; these

# are exactly the same operations we used to compute the forward pass using
# Tensors, but we do not need to keep references to intermediate values since
# we are not implementing the backward pass by hand.
y_pred = x.mm(w1).clamp(min=0).mm(w2) 

# Compute and print loss using operations on Tensors.
# Now loss is a Tensor of shape (1,)
# loss.item() gets the a scalar value held in the loss.
loss = (y_pred - y).pow(2).sum()
print(t, loss.item())

# Use autograd to compute the backward pass. This call will compute the
# gradient of loss with respect to all Tensors with requires_grad=True.
# After this call w1.grad and w2.grad will be Tensors holding the gradient
# of the loss with respect to w1 and w2 respectively.
loss.backward()

# Manually update weights using gradient descent. Wrap in torch.no_grad()
# because weights have requires_grad=True, but we don't need to track this
# in autograd.
# An alternative way is to operate on weight.data and weight.grad.data.
# Recall that tensor.data gives a tensor that shares the storage with
# tensor, but doesn't track history.
# You can also use torch.optim.SGD to achieve this.
with torch.no_grad():
    w1 -= learning_rate * w1.grad
    w2 -= learning_rate * w2.grad

    # Manually zero the gradients after updating weights
    w1.grad.zero_()
    w2.grad.zero_()

it works normally.

thanks for your help in advance.

[YangHe712]

I think the error " CUDNN_STATUS_EXECUTION_FAILED " is due to the fact that CUDA 9.0 has problem recognizing the graphic card of RTX 20xx. To solve the problem, I first try to install CUDA 10.0 from the source, just as you suggested. But after I spent much time modifying the configuration, it still shows "segmentation fault (core dumped)". Then I turn to CUDA 9.2 because I hear from someone that 9.2 supports RTX 2080. Luckily it does work. So now the only problem reminded is the package SparseConvNet's dependency on CUDA 9. We finally sovled it by creating a symbolic link that targets the CUDA 9.2.

Now, the problem is solved.

[NagarajDesai1]

"We finally sovled it by creating a symbolic link that targets the CUDA 9.2." could you please tell me the exact steps which you performed to solve this issue? I am facing the same issue with cuda 9.2 and titan RTX GPU.