search

facebookresearch / Detectron

FAIR's research platform for object detection research, implementing popular algorithms like Mask R-CNN and RetinaNet.
Apache License 2.0
26.22k stars 5.45k forks source link

CUDA driver version is insufficient for CUDA runtime version #899

Closed bruce2k closed 5 years ago

bruce2k commented 5 years ago

Expected results

Pass the inference demo

python tools/infer_simple.py \ --cfg configs/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml \ --output-dir /tmp/detectron-visualizations \ --image-ext jpg \ --wts https://dl.fbaipublicfiles.com/detectron/35861858/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml.02_32_51.SgT4y1cO/output/train/coco_2014_train:coco_2014_valminusminival/generalized_rcnn/model_final.pkl \ demo

Actual results

RuntimeError: [enforce fail at common_gpu.cc:98] error == cudaSuccess. 35 vs 0. Error at: /opt/conda/conda-bld/pytorch-nightly_1560316055483/work/caffe2/core/common_gpu.cc:98: CUDA driver version is insufficient for CUDA runtime version frame #0: c10::ThrowEnforceNotMet(char const, int, char const, std::string const&, void const*) + 0x59 (0x7fa791f4d8a9 in /home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libc10.so) frame #1: caffe2::CaffeCudaGetDevice() + 0x8f6 (0x7fa746dabdc6 in /home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so) frame #2: + 0x2cf8165 (0x7fa7485f4165 in /home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so) frame #3: + 0x68eb7 (0x7fa79266feb7 in /home/bruce/anaconda3/envs/maskrcnn/lib/python3.6/site-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-36m-x86_64-linux-gnu.so)

Detailed steps to reproduce

The command that you ran



### System information

* Operating system: Ubuntu 17.10
* Compiler version: gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu3.2) 
* CUDA version: CUDA Version 9.0.176
* cuDNN version: CUDNN_MAJOR 7
* NVIDIA driver version: NVIDIA-SMI 384.130                Driver Version: 384.130
* GPU models (for all devices if they are not all the same): 1080Ti
* `python --version` output: Python 3.6.8 :: Anaconda, Inc.
* Anything else that seems relevant: ?
caffe2 install command: conda install pytorch-nightly cuda90 -c pytorch
anshumankmr commented 5 years ago

Make a new environment using conda and try installing it again: A complete fix would be however ensuring your Nvidia drivers match the version of CUDA and CUDNN, preferably try uninstalling that nvidia driver and install the latest version, it might help.

bruce2k commented 5 years ago

yup , new environment works