RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

[guocx1g]

result = self.forward(*input, **kwargs) File "/home/opt/anaconda3/envs/yolo/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 423, in forward return self._conv_forward(input, self.weight) File "/home/opt/anaconda3/envs/yolo/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 420, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch torch.backends.cuda.matmul.allow_tf32 = True torch.backends.cudnn.benchmark = True torch.backends.cudnn.deterministic = False torch.backends.cudnn.allow_tf32 = True data = torch.randn([1, 39, 1024, 1024], dtype=torch.float, device='cuda', requires_grad=True) net = torch.nn.Conv2d(39, 10, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1) net = net.cuda().float() out = net(data) out.backward(torch.randn_like(out)) torch.cuda.synchronize()

ConvolutionParams data_type = CUDNN_DATA_FLOAT padding = [1, 1, 0] stride = [1, 1, 0] dilation = [1, 1, 0] groups = 1 deterministic = false allow_tf32 = true input: TensorDescriptor 0x56396365ef20 type = CUDNN_DATA_FLOAT nbDims = 4 dimA = 1, 39, 1024, 1024, strideA = 40894464, 1048576, 1024, 1, output: TensorDescriptor 0x5639634faee0 type = CUDNN_DATA_FLOAT nbDims = 4 dimA = 1, 10, 1024, 1024, strideA = 10485760, 1048576, 1024, 1, weight: FilterDescriptor 0x56396365ed60 type = CUDNN_DATA_FLOAT tensor_format = CUDNN_TENSOR_NCHW nbDims = 4 dimA = 10, 39, 3, 3, Pointer addresses: input: 0x7ff870400000 output: 0x7ff87a000000 weight: 0x7ff967602a00

[SuhyeonHa]

For me, it occured when GPU memory was not enough. Please check it out.