The loss suddenly becomes larger and the map becomes 0

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )

http://pjreddie.com/darknet/

Other

21.71k stars 7.96k forks source link

The loss suddenly becomes larger and the map becomes 0 #6593

Open yds5817 opened 4 years ago

yds5817 commented 4 years ago

Hello! The model I trained looked normal at the beginning, but then the loss would suddenly rise. What was going on？

./darknet detector train ./cfg/wujie.data ./cfg/yolov4-custom.cfg ./yolov4.conv.137 -dont_show -mjpeg_port 9080 -map -gpus 0,1,2,3
 DEBUG=1
 CUDA-version: 9000 (11000), cuDNN: 7.1.4, CUDNN_HALF=1, GPU count: 4
 CUDNN_HALF=1
 OpenCV version: 3.4.10d
0,1,2,3
 Prepare additional network for mAP calculation...
 0 : compute_capability = 610, cudnn_half = 0, GPU: TITAN Xp
net.optimized_memory = 0
mini_batch = 1, batch = 32, time_steps = 1, train = 0
   layer   filters  size/strd(dil)      input                output

stephanecharette commented 4 years ago

I recently had the same problem. Was asking on the discord channel just the other day. Never did figure it out.

I think that project was yolov4-tiny-3l, but I'm no longer 100% certain.

dsgh2 commented 4 years ago

Me too. I solved this problem by decreasing learning rate a little bit. But I am wondering why mAP only rises to a certain point and then goes to zero. It seemed to go well.

zyayoung commented 4 years ago

Same issue. The sudden drop will occur near the same iter even if restart training from the last normal checkpoint.

For more information: obj drop to almost exactly zero (< 1e-6) while classification and IoU become much lower than it should be but non-zero.

WANGCHAO1996 commented 3 years ago

Hello, how to solve this problem, class will be 1, will be 0, IOU is very low..