when I inference the model，following error occurred，
Traceback (most recent call last):
File "tools/test_net.py", line 101, in
main()
File "tools/test_net.py", line 95, in main
output_folder=output_folder,
File "/home/xing.yue/PycharmProjects/FCOS/FCOS/fcos_core/engine/inference.py", line 83, in inference
predictions = compute_on_dataset(model, data_loader, device, inference_timer)
File "/home/xing.yue/PycharmProjects/FCOS/FCOS/fcos_core/engine/inference.py", line 30, in compute_on_dataset
output = model(images.to(device))
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, kwargs)
File "/home/xing.yue/PycharmProjects/FCOS/FCOS/fcos_core/modeling/detector/generalized_rcnn.py", line 49, in forward
features = self.backbone(images.tensors)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, *kwargs)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, kwargs)
File "/home/xing.yue/PycharmProjects/FCOS/FCOS/fcos_core/modeling/backbone/mobilenet.py", line 116, in forward
x = m(x)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, *kwargs)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, **kwargs)
File "/home/xing.yue/PycharmProjects/FCOS/FCOS/fcos_core/layers/misc.py", line 32, in forward
return super(Conv2d, self).forward(x)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
My environment is following:
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176
when I inference the model，following error occurred，
Traceback (most recent call last): File "tools/test_net.py", line 101, in
main()
File "tools/test_net.py", line 95, in main
output_folder=output_folder,
File "/home/xing.yue/PycharmProjects/FCOS/FCOS/fcos_core/engine/inference.py", line 83, in inference
predictions = compute_on_dataset(model, data_loader, device, inference_timer)
File "/home/xing.yue/PycharmProjects/FCOS/FCOS/fcos_core/engine/inference.py", line 30, in compute_on_dataset
output = model(images.to(device))
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, kwargs)
File "/home/xing.yue/PycharmProjects/FCOS/FCOS/fcos_core/modeling/detector/generalized_rcnn.py", line 49, in forward
features = self.backbone(images.tensors)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, *kwargs)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, kwargs)
File "/home/xing.yue/PycharmProjects/FCOS/FCOS/fcos_core/modeling/backbone/mobilenet.py", line 116, in forward
x = m(x)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, *kwargs)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, **kwargs)
File "/home/xing.yue/PycharmProjects/FCOS/FCOS/fcos_core/layers/misc.py", line 32, in forward
return super(Conv2d, self).forward(x)
File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
My environment is following: PyTorch version: 1.1.0 Is debug build: No CUDA used to build PyTorch: 9.0.176
OS: Ubuntu 16.04.6 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609 CMake version: version 3.18.4
Python version: 3.7 Is CUDA available: Yes CUDA runtime version: 9.0.176 GPU models and configuration: GPU 0: Tesla T4 GPU 1: Tesla T4
Nvidia driver version: 450.102.04 cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.5.0
Versions of relevant libraries: [pip3] numpy==1.19.2 [pip3] torch==1.1.0 [pip3] torchvision==0.3.0 [conda] blas 1.0 mkl defaults [conda] mkl 2019.0 118 defaults [conda] mkl-service 1.1.2 py37h90e4bf4_5 defaults [conda] mkl_fft 1.0.4 py37h4414c95_1 defaults [conda] mkl_random 1.0.1 py37h4414c95_1 defaults Pillow (8.2.0)
what's the problem?