Loss goes not decrease

[VisionEp1]

Hi once again me,

I am currently training a yolov3 (not spp , with 608x608 , 12 anchors and focal_loss =1, classes = 3) for 40 000 iterations.

The avg loss is currently about 0.55

The loss was at 0.6 at 10.000 iteration

Train images: - 20 million However i had the same issues with 50 000+ iterations on train images: 25.000

Following questions:

1. is it normal that the loss basicaly stays the same ? (i know training images amount, but also with 25.000 images is the same result)
2. is there anyway to speedup the process (without loosing accuracy)
3. How long should i usually train ? (50.000, 100.000, 1million iterations)

When you see that average loss 0.xxxxxx avg no longer decreases at many iterations then you should stop training. what are many iterations?

4. what is a "good loss" in yolo ? (i know it depends on dataset etc, but to have a high result like 90% reading rate should the loss be about 0.1? or 0.01?etc)

thanks in advance

[AlexeyAB]

@VisionEp1 Hi,

1. It is normal. Usually, the more images - the higher loss - the higher accuracy (mAP) on the same test-dataset. So accuracy (mAP) is more important indicator than loss.

2. Use modern Volta/Turing GPU with CUDNN_HALF=1 in the Makefile, and/or use multi-GPU training after first 1000 iterations.

3. Usually you should train the same number of iterations as many images, i.e. ~20 million iterations. Or you should train at least 1 epoch = images/batch = 20mil/64 = 310 000 iterations.

4. Usually I get final avg loss about 0.5. But I saw final avg loss 0.1 - 2.0

[VisionEp1]

thanks another question: if the map increases slightly in between iterations, is this normal?

[AlexeyAB]

Yes, it is normal.