gliese581gg / YOLO_tensorflow

tensorflow implementation of 'YOLO : Real-Time Object Detection'
Other
1.72k stars 656 forks source link

modprobe: ERROR: could not insert 'nvidia_378_uvm': Invalid argument #23

Closed halt9 closed 7 years ago

halt9 commented 7 years ago

When trying to load YOLO_small.ckpt, at line self.fc_32 = self.fc_layer(32,self.fc_30,1470,flat=False,linear=True) I get the error:

modprobe: ERROR: could not insert 'nvidia_378_uvm': Invalid argument
E tensorflow/stream_executor/cuda/cuda_driver.cc:509] failed call to cuInit: CUDA_ERROR_UNKNOWN
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: vm-gpu
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: vm-gpu
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 367.57.0
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:363] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module  367.57  Mon Oct  3 20:37:01 PDT 2016
GCC version:  gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)
"""
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 367.57.0
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:300] kernel version seems to match DSO: 367.57.0

Is the driver version hard coded into the checkpoint file? My driver is 367.57.0, as you can tell. I can run other tensorflow code fine. Using tensorflow-gpu=0.12.1. Using the same code on an exact same machine with a newer drive is fine, but this machine cannot have its drivers updated for reasons, so I'm wondering if it's possible to run this file with the older driver. I use CUDA 8.0 and cuDNN 5.1