ShaoqingRen / faster_rcnn

Faster R-CNN
Other
2.72k stars 1.22k forks source link

Problem in training with GeForce GTX1070 and CUDA 8.0 #130

Open oneQuery opened 7 years ago

oneQuery commented 7 years ago

Is there anybody who succeeded running _'script_faster_rcnnVOC...' with GeForce GTX1070 and CUDA 8.0?

When I use GeForce GTX 960M and CUDA 6.5, it doesn't have any problem. But there is a problem as I use the GPU, GeForce GTX1070, and CUDA 8.0 with like this error:

Error using caffe_
glog check error, please check log and clear mex

Error in caffe.Solver/step (line 56)
caffe_('solver_step', self.hSolver_self, iters);

Error in fast_rcnn_train>check_gpu_memory (line 216)
caffe_solver.step(1);

Error in fast_rcnn_train (line 89)
check_gpu_memory(conf, caffe_solver, num_classes, opts.do_val);

Error in Faster_RCNN_Train.do_fast_rcnn_train (line 7)
model_stage.output_model_file = fast_rcnn_train(conf, dataset.imdb_train, dataset.roidb_train, ...

Error in script_faster_rcnn_VOC2007_ZF (line 53)
model.stage1_fast_rcnn = Faster_RCNN_Train.do_fast_rcnn_train(conf_fast_rcnn, dataset,
model.stage1_fast_rcnn, opts.do_val);

And this is the log:

.
.
.
I1211 21:28:12.254725    96 net.cpp:743] Ignoring source layer fc8
I1211 21:28:12.254725    96 net.cpp:743] Ignoring source layer prob
F1211 21:28:42.700903    96 math_functions.cu:422] Check failed: status == CURAND_STATUS_SUCCESS (201 vs. 0)  CURAND_STATUS_LAUNCH_FAILURE
F1211 21:28:42.700903    96 math_functions.cu:422] Check failed: status == CURAND_STATUS_SUCCESS (201 vs. 0)  CURAND_STATUS_LAUNCH_FAILURE_

I think it has a problem with GPU, GPU driver and CUDA version. So I'm wondering is there anybody who succeeded running _'script_faster_rcnnVOC...' with GeForce GTX1070 and CUDA 8.0 (I'm using MATLAB 2016b and Window 8.1)

xzabg commented 7 years ago

@assess09 Have you solved it now?

rtgoring commented 7 years ago

GTX 1080 can't use CUDNN 5, recompile with that disabled. Use CUDA 7.5. Enable PTX.