search

smallcorgi / Faster-RCNN_TF

Faster-RCNN in Tensorflow
MIT License
2.34k stars 1.12k forks source link

Error in Running with GPU #179

Open xinboli opened 7 years ago

xinboli commented 7 years ago

Hi, I tried to run the code as follows

  1. I only installed tensorflow-gpu and ran the code as CUDA_VISIBLE_DEVICES=3 ./experiments/scripts/faster_rcnn_end2end.sh GPU 3 VGG16 pascal_voc or ./experiments/scripts/faster_rcnn_end2end.sh GPU 3 VGG16 pascal_voc

then the error presented (my input is a 3744-by-5616 rgb image)

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,3,3744,5616] [[Node: conv1_1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](_recv_Placeholder_0/_105, conv1_1/weights/read)]] [[Node: rpn_rois/_125 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_285_rpn_rois", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

  1. If I installed tensorflow (not gpu version) and ran the code ./experiments/scripts/faster_rcnn_end2end.sh CPU 3 VGG16 pascal_voc

it occupied all of the 40 CPU cores instead only in the specified 3

  1. If I installed tensorflow and tensorflow-gpu at the same time, and ran the code CUDA_VISIBLE_DEVICES=3 ./experiments/scripts/faster_rcnn_end2end.sh GPU 3 VGG16 pascal_voc or ./experiments/scripts/faster_rcnn_end2end.sh GPU 3 VGG16 pascal_voc

it occupied all of the 40 CPU cores too. But it only used c.a. 100MB GPU 3.

It seems that, it only could run in CPU model. I have 4 TITAN X, and 40 E5-2650.

Can anyone help? Thanks a lot

charan223 commented 6 years ago

Same issue here also, how did you resolve it?