search

Azure / ObjectDetectionUsingCntk

Fast R-CNN Object Detection on Azure using CNTK
129 stars 61 forks source link

cuDNN failure 4: CUDNN_STATUS_INTERNAL_ERROR #2

Closed MikeWise2718 closed 7 years ago

MikeWise2718 commented 7 years ago

I go this error message when running this. I suspect this might be an out-of-memory error as I have a rather small GPU unit (only 2 GB) but I do wish the error message was more informative. Not the fault of this software or CNTK though seems to be CUDA's fault. Googling yielded no answer either - but I did find more than a few people asking about the message, some thinking it might have been memory issues.

In any case the work around is to add a deviceId=-1 parameter to force a CPU-only invocation of CNTK to line 24 of 3_runCntk.py like this:

 cmdStr += " deviceID=-1 ImageH={} ImageW={}".format(cntk_padHeight, cntk_padWidth)

CNTK then ran in 188 seconds on my i7-4770T CPU (not the fastest either) - and this is fine for getting the demo to work.

It would be worth having this as a flag.

PatrickBue commented 7 years ago

Hi, yeah, looks like this is indeed caused by the GPU running out of memory. The ResNet model is pretty memory hungry. Glad you found a workaround.