High values of avg loss and low mAP training small objects

[Lepiloff]

@AlexeyAB hi!!! I am training my own dataset with only one class. Dataset contains of 1k labeled images and 1k unlabeled(empty .txt file). Each images contains from 1 to 600 small objects. For train I made new yolov3.cfg file with such changes (in accordance with the recommendations for training small objects https://github.com/AlexeyAB/darknet#how-to-improve-object-detection):

batch=64
subdivisions=64
.....
max_batches = 2000
steps=1600,1800
...
[upsample]
stride=4
[route]
layers = -1, 11
...
filters = 18
...
classes = 1
...
add max=700 to each yolo layers
....
recalculate anchors for all yolo layer
anchors = 14,30,  25,25,  22,49,  38,37,  30,73,  50,54,  45,97,  77,73,  73,133
...
add random=1 to last yolo layer

My cf file https://github.com/Lepiloff/pyolo/blob/master/pyolo/cfg/yolov3.cfg And start train with command ./darknet detector train data/obj.data yolov3j.cfg darknet53.conv.74 After 2000 iteration I got avg loss 8.593376, 12.742638 and mAP

I tried to train 300 more iterations but the results got worse

After getting these bad results, I tried to do Fine-Tuning instead of Transfer-Learning using new yolov3-openimages.conv.81 and yolov3-openimages.cfg with the same parameters as last time but with stopbackward=1 After 2000 iterations the avg loss is about the same (~10) but the mAP is worse

Why is this so? Could you help me choose the right settings for this task?

[AlexeyAB]

@Lepiloff Hi,

Since there are 600 objects on one image - it seems there are very small objects. It's a good mAP = 59% for such small objects.

Just try to train with higher width=608 height=608

Or if you will use only this repository, you can try to train new model with width=832 height=832 batch=64 subdivisions=64 random=0 (or 608x608) especially for such small objects: https://github.com/AlexeyAB/darknet/files/3199657/yolo_v3_spp_pan.cfg.txt

More: https://github.com/AlexeyAB/darknet/issues/3114#issuecomment-494148968

Use the same pre-trained weights darknet53.conv.74

---

After getting these bad results, I tried to do Fine-Tuning instead of Transfer-Learning using new yolov3-openimages.conv.81 and yolov3-openimages.cfg with the same parameters as last time but with stopbackward=1

Objects in openimages dataset are much larger than in your dataset, so there stopbackward gives bad result.

[Lepiloff]

@AlexeyAB , thanks. More then 100 objects contains only about 15-20% of images.
1.I am trained with width=832 height=832 , is training with width=608 height=608 would be better for small objects?

2. Why my avg loss mach more bigger as required value - 0.5 , is this normal? which parameter is more important avg loss or mAP?
3. Was the right decision to add to all yolo layers max=600, at documentation you recommended add only at last layer?
4. About yolo_v3_spp_pan.cfg , all I need is to make only the specified settings (width=832 height=832 batch=64 subdivisions=64 random=0)? No additional specific changes (anchors, anything else)?

[AlexeyAB]

1. width=832 height=832 is better

2. this is normal, you have very small objects. mAP is more important

3. yes

4. set width=832 height=832 batch=64 subdivisions=64 max_batches = 4000 steps=3200,3600 random=0 also classes= and filters=(classes+5)x4 in each yolo-layer

[Lepiloff]

@AlexeyAB Thank you, I will try