search

prabindh / darknet

Convolutional Neural Networks
http://pjreddie.com/darknet/
Other
122 stars 46 forks source link

CUDA out of memory error even with CUDNN=0 #32

Closed vbadri closed 6 years ago

vbadri commented 7 years ago

Verified that the tiny yolo model runs just fine with both CUDNN=0 and CUDNN=1 However, cannot get the full yolo model to run. Error message below. A previous version of this branch ran just fine for me (in fact, it was the only one which would run since I'm using OpenCV 3). I see an earlier commenter fixed this issue by disabling CUDNN, but that didn't work for me.

I suspect this may simply be because I'm running a low-end GPU card. My CUDA configuration below from lspci | grep VGA. Has anyone with this configuration got the latest code working with the full yolo CNN?

01:00.0 VGA compatible controller: NVIDIA Corporation GK107GLM [Quadro K1100M] (rev a1)

layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 1 max 2 x 2 / 2 416 x 416 x 32 -> 208 x 208 x 32 2 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 3 max 2 x 2 / 2 208 x 208 x 64 -> 104 x 104 x 64 4 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 5 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 6 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 7 max 2 x 2 / 2 104 x 104 x 128 -> 52 x 52 x 128 8 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 9 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 10 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 11 max 2 x 2 / 2 52 x 52 x 256 -> 26 x 26 x 256 12 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 13 CUDA Error: out of memory darknet-cpp: ./src/cuda.c:38: void check_error(cudaError_t): Assertion `0' failed.

prabindh commented 7 years ago

I use it on a Quadro 1000M, which has just the same 2GB memory. I have used it without issues. Could you share the cfg file ?

vbadri commented 7 years ago

Command that failed. The cfg file was from the repo. So was the coco.data file. I got the weights from the original pjreddie site

./darknet-cpp detector test cfg/coco.data cfg/yolo.cfg /home/badri/documents/see/yolo.weights data/dog.jpg

prabindh commented 7 years ago

I see. Most likely you might used a version earlier than this (https://github.com/prabindh/darknet/commit/60e952ba694e3e0811db5868d70ad7ebfe676836), that introduced new layers. However, you can control the ratio of batch/subdivision in the cfg file to see if you are able to get a reasonable memory figure.

prabindh commented 6 years ago

Closing old issues. Please reopen if the issue is valid now with latest master.