AlexeyAB / Yolo_mark

GUI for marking bounded boxes of objects in images for training neural network Yolo v3 and v2
https://github.com/AlexeyAB/darknet
The Unlicense
1.8k stars 680 forks source link

running two darknet processes: "CUDA Error: out of memory" #70

Open WZRPW opened 6 years ago

WZRPW commented 6 years ago

hi there, I am testing the maximal number of processes of darknet.exe (realtime object detection with IP camera) can run on my PC. It worked OK to run one process of darknet.exe I got the following error message if I started another separate process of darknet.exe: ######################################################################## layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 18 Shortcut Layer: 15 CUDA Error: out of memory CUDA Error: out of memory: No error ######################################################################## I found that someone had experienced the memory error and increased subdivisions to solve the problem. https://github.com/AlexeyAB/Yolo_mark/issues/19 But I changed the subvision from 16 to 32 and 64(Batch = 64 as deafult in yolov3.cfg). In all these three cases, it still ran out of memory.

My PC information: CPU: Intel(R) Core(TM) i5-7400 CPU @3.00GHz Mem: 16GB Windows 10 GPU: Nvidia Geforce 730

darknet_resource_cost

WZRPW commented 6 years ago

update on this issue: I did the same test on a PC with Nvidia GTX 1080 ti (PC: Intel i5-8400, RAM 16GB). Two processes of darknet.exe worked very well for object detection. CPU usage was up to 87% in this case.