search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.73k stars 7.96k forks source link

Nan avg-loss get large during training #7880

Open only-zzx opened 3 years ago

only-zzx commented 3 years ago

have an issue with Nan avg-loss: while training my own dateset in p5,i found avg-loss getting bigger and bingger. but same dataset has good result in YOLOv4 traning. I chang the cfg parameter "batch=10" to fix my GPU cfg-file is here:yolov4-p5.txt

i use command:./darknet detector train cfg/voc.data cfg/yolov4-p5.cfg ./yolov4-p5.conv.232 -gpus 0,1,2,3 -map my 企业微信截图_16258087963150

lsd1994 commented 3 years ago

reduce learning rate

only-zzx commented 3 years ago

reduce learning rate

my learning_rate=0.0001. have to reduce more?

lsd1994 commented 3 years ago

I don't know your dataset and this model in detail, you can just try it. I got the similar chart with you and just solved by reducing learning rate.

bsysman commented 3 years ago

Has your problem been solved，I have the same problem as you。