CPU-RAM training crashes with CUDA Error

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )

http://pjreddie.com/darknet/

Other

21.72k stars 7.96k forks source link

CPU-RAM training crashes with CUDA Error #5186

Open xevolesi opened 4 years ago

xevolesi commented 4 years ago

Hi, @AlexeyAB !

I wanted to test #4386 and did this .cfg file: yolov3-default-608-norandom-mosaic-scale-cpumem.cfg.txt

It's successfully trained about 5000 iterations and then droped with:

Also, if I set random = 1 and dynamic_minibatch = 1, learning still crashes with a CUDA error.

My server config:

125 GB RAM;
4xRTX 2080ti

Could you please give me advice on how to configure the .cfg file so i can train YOLOv3 using CPU-RAM with random = 1?

AlexeyAB commented 4 years ago

It is in Beta currently. So may be just CPU-Memory is not enough. Or try to train without -map flag. You can check the mAP later by using command ./darknet detector map ...

xevolesi commented 4 years ago

Thank you. I'll try it.