CUDA error

[Prakhar-97]

Hi I have the right version of Cuda and still getting this issue while running the main file, can you help me with this

Environment : sys.platform: linux Python: 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0] CUDA available: True CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 10.0, V10.0.130 GPU 0: Tesla P100-PCIE-16GB GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 PyTorch: 1.4.0+cu100 PyTorch compiling details: PyTorch built with:

• GCC 7.3
• Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CUDA Runtime 10.0
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
• CuDNN 7.6.3
• Magma 2.5.1
• Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.5.0+cu100 OpenCV: 4.1.2 MMCV: 0.5.3 MMDetection: 1.2.0+0f33c08 MMDetection Compiler: GCC 7.5 MMDetection CUDA Compiler: 10.0

Error Traceback

Traceback (most recent call last): File "Table Structure Recognition/main.py", line 23, in result = inference_detector(model, i) File "/content/drive/My Drive/dundun/mmdetection/mmdet/apis/inference.py", line 86, in inference_detector result = model(return_loss=False, rescale=True, data) File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(*input, *kwargs) File "/content/drive/My Drive/dundun/mmdetection/mmdet/core/fp16/decorators.py", line 49, in new_func return old_func(args, kwargs) File "/content/drive/My Drive/dundun/mmdetection/mmdet/models/detectors/base.py", line 149, in forward return self.forward_test(img, img_metas, kwargs) File "/content/drive/My Drive/dundun/mmdetection/mmdet/models/detectors/base.py", line 130, in forward_test return self.simple_test(imgs[0], img_metas[0], kwargs) File "/content/drive/My Drive/dundun/mmdetection/mmdet/models/detectors/cascade_rcnn.py", line 324, in simple_test self.test_cfg.rpn) if proposals is None else proposals File "/content/drive/My Drive/dundun/mmdetection/mmdet/models/detectors/test_mixins.py", line 34, in simple_test_rpn proposal_list = self.rpn_head.get_bboxes(proposal_inputs) File "/content/drive/My Drive/dundun/mmdetection/mmdet/core/fp16/decorators.py", line 127, in new_func return old_func(args, *kwargs) File "/content/drive/My Drive/dundun/mmdetection/mmdet/models/anchor_heads/anchor_head.py", line 276, in get_bboxes scale_factor, cfg, rescale) File "/content/drive/My Drive/dundun/mmdetection/mmdet/models/anchor_heads/rpn_head.py", line 92, in get_bboxessingle proposals, = nms(proposals, cfg.nms_thr) File "/content/drive/My Drive/dundun/mmdetection/mmdet/ops/nms/nms_wrapper.py", line 54, in nms inds = nms_cuda.nms(dets_th, iou_thr) RuntimeError: CUDA error: no kernel image is available for execution on the device (launch_kernel at /pytorch/aten/src/ATen/native/cuda/Loops.cuh:103) frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fde7991b193 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so) frame #1: void at::native::gpu_index_kernel<_nv_dl_wrapper_t<nv_dl_tag<void ()(at::TensorIterator&, c10::ArrayRef, c10::ArrayRef), &(void at::native::index_kernel_impl<at::native::OpaqueType<8> >(at::TensorIterator&, c10::ArrayRef, c10::ArrayRef)), 1u>> >(at::TensorIterator&, c10::ArrayRef, c10::ArrayRef, __nv_dl_wrapper_t<_nv_dl_tag<void (*)(at::TensorIterator&, c10::ArrayRef, c10::ArrayRef), &(void at::native::index_kernel_impl<at::native::OpaqueType<8> >(at::TensorIterator&, c10::ArrayRef, c10::ArrayRef)), 1u>> const&) + 0x7bb (0x7fde7f58387b in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so)

[kshitijkapadni]

It seems that you are executing the code on google colab. The code is totally developed in google colab and we have never faced this issue. Most of the CUDA related issues were solved by Reseting the environment or Factory reset Runtime(You can try this). If you have correctly followed the MMdetection installation instructions everything should go fine. As per my knowledge, the issue is it is unable to find the GPU.

[Prakhar-97]

Yes, I am executing the code in google colab. I have followed the setup instructions as mentioned by you in Readme.md file , I have changed the Runtime type to GPU and even tried to run all the commands after Factory reset Runtime, still getting the same error. Is there anything I am missing? Also, are there changes to be made in the config file as the paths mentioned there do not seem to be universal.