Open willbattel opened 4 years ago
I lowered the LR from the default 0.00261
to 0.00200
and so far it seems to be running nominally. I'll keep watching it.
hello, i also meet this problem, i use yolov4-leaky.cfg and yolov4.weights to train model, after some iters, loss become nan. I lowered the LR from the default 0.00261 to 0.00100,, but no difference. can u solve this problem?@willbattel
Mine worked with LR of 0.002, so I'm not sure what the problem is in your case. I would try on other models, like normal v4, and see if you have the same issue or if it's only with v4 Leaky.
Maybe you need to use cd53paspp-gamma_final.weights as the preweights.because of the difference of leaky and swish.
I'm trying to train YOLOv4 and YOLOv4 Leaky on a custom dataset. YOLOv4 is working, but YOLOv4 Leaky is diverging during training (nan avg-loss). I pulled the cfg file from the Model Zoo and changed the parameters to the same values in my YOLOv4 cfg file. The only difference between the two files is the mish/leaky activations specified in the Model Zoo files.
Compare with YOLOv4 cfg - https://www.diffchecker.com/8ulGrJk7
Two ideas I had for maybe why this is happening,
Is darknet trying to train the alpha value in the activations? In other words, is darknet using LReLU or PReLU. If darknet is trying to train the activation layers (PReLU), then the base weights in
yolov4.conv.137
won't work. I didn't see any other weights to use for YOLOv4 Leaky so I used the same as YOLOv4.Is the learning rate too high for Leaky? Both cfg files use the same learning rate, but maybe that value isn't appropriate for YOLOv4 Leaky. Have you tested the learning rate on v4 Leaky? I read your paper for YOLOv4 and I'm wondering if the proper learn rate was found for YOLOv4 Leaky or only for YOLOv4.
Otherwise, I'm not sure why v4 Leaky is failing. Normal v4 is training perfectly fine and is configured nearly identically.