using focal loss can impove mAP

[WZMIAOMIAO]

🚀 Feature

First of all, thank the author for sharing the quality code. I found that using focal loss can impove one point mAP(@IOU=0.5)

Training datasets

train_dataset: PASCAL VOC2012 train.txt val_dataset: PASCAL VOC2012 val.txt training epoch: 20 training initial lr: 0.001 end lr: 0.00001 training layer: only training predictor

Description

When I use default setting, I can get the following results.

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.582
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.809
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.644
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.199
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.464
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.666
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.464
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.693
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.717
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.378
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.621
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.779

When I use focal loss fuction, I can get the following results.

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.589
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.819
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.653
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.217
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.473
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.672
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.467
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.697
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.725
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.436
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.630
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.786

During using focal_loss, I change following setting:

"cls": 37.4 * 20,  # default 37.4
"obj": 64.3 * 45,  # default 64.3  
"fl_gamma": 1.5,  # default 0.0

In addition, I suggest adding a parameter(max_num) to NMS function. In some cases, it can speed up the test time.

def non_max_suppression(prediction, conf_thres=0.1, iou_thres=0.6,
                        multi_label=True, classes=None, agnostic=False, max_num=100):
    ...
    i = torchvision.ops.boxes.nms(boxes, scores, iou_thres)
    i = i[:max_num]  # only save top max_num objects
    ...

I hope it works for you. Thank you.

[glenn-jocher]

@WZMIAOMIAO oh that's interesting, thanks for the feedback! We've experimented with VOC training in YOLOv5, and have reached some excellent results after evolving hyperparameters on the VOC dataset. The best mAP we recorded was about 0.92 with YOLOv5x, with the smaller models like YOLOv5m (which is much smaller than YOLOv3-SPP) at about 0.90 mAP@0.5.

The command to reproduce with YOLOv5 is here. Dataset and pretrained weights are both auto-downloaded on first use. https://github.com/ultralytics/yolov5

python train.py --batch 64 --weights yolov5m.pt --data voc.yaml --img 512 --epochs 50 --hyp hyp.finetune.yaml

[WZMIAOMIAO]

Thank you. I'll learn your yolov5 later.

[www7890]

Thank you. I'll learn your yolov5 later.

Hi @WZMIAOMIAO , Is this conclusion from one single result of training? or a significant improvement?

[glenn-jocher]

@WZMIAOMIAO got it. To answer your other point, we have indeed included a maximum box parameter in the YOLOv5 NMS code settings section:

https://github.com/ultralytics/yolov5/blob/4d3680c81dad760e5d1ffe397ab2fc9169d9fb70/utils/general.py#L608-L614

    # Settings - YOLOv5 NMS
    min_wh, max_wh = 2, 4096  # (pixels) minimum and maximum box width and height
    max_det = 300  # maximum number of detections per image
    time_limit = 10.0  # seconds to quit after
    redundant = True  # require redundant detections
    multi_label = nc > 1  # multiple labels per box (adds 0.5ms/img)

[WZMIAOMIAO]

Thank you. I'll learn your yolov5 later.

Hi @WZMIAOMIAO , Is this conclusion from one single result of training? or a significant improvement?

This conclusion is only based on Pascal VOC dataset

[www7890]

Thank you. I'll learn your yolov5 later.

Hi @WZMIAOMIAO , Is this conclusion from one single result of training? or a significant improvement?

This conclusion is only based on Pascal VOC dataset

Got it, thanks!

[marina-neseem]

@WZMIAOMIAO How did you train on PASCAL dataset! I am trying to do so but I get very low mAP (around 20%) Did you just follow the instructions at "Train on a custom dataset"?

[WZMIAOMIAO]

@MarinaHesham Yes, but it should be noted that I trained only three predictors(freeze_layer=True) and modified the lost gain. If you follow my configuration, you can get the same results. In my experiment, I found that if you train all the weights, you need to readjust the gain of each loss, and the effect is not necessarily good. Personally, I think it's about calculating the loss function. I have tried to modify the method of calculating the loss function and the gain of each loss, which can get a slightly better result. I didn't spend too much time here because I didn't have much experience in tuning parameters. If you want to use focal loss, I suggest modifying the function that calculates the loss. You can refer to RetinaNet, the official implementation of Pytorch. https://github.com/pytorch/vision/blob/master/torchvision/models/detection/retinanet.py

[nanhui69]

Thank you. I'll learn your yolov5 later.

Hi @WZMIAOMIAO , Is this conclusion from one single result of training? or a significant improvement?

This conclusion is only based on Pascal VOC dataset

@WZMIAOMIAO What precision can you achieve in VOC dataset?

[nanhui69]

@MarinaHesham Yes, but it should be noted that I trained only three predictors(freeze_layer=True) and modified the lost gain. If you follow my configuration, you can get the same results. In my experiment, I found that if you train all the weights, you need to readjust the gain of each loss, and the effect is not necessarily good. Personally, I think it's about calculating the loss function. I have tried to modify the method of calculating the loss function and the gain of each loss, which can get a slightly better result. I didn't spend too much time here because I didn't have much experience in tuning parameters. If you want to use focal loss, I suggest modifying the function that calculates the loss. You can refer to RetinaNet, the official implementation of Pytorch. https://github.com/pytorch/vision/blob/master/torchvision/models/detection/retinanet.py

@WZMIAOMIAO what precision can you get ?? what's the train repo v3 or v5?

[glenn-jocher]

Hi @nanhui69, it seems there might be some confusion. I'm Glenn Jocher, the author of the Ultralytics YOLOv3 repo. The discussion you're referring to with @WZMIAOMIAO is not something I can directly comment on as it pertains to their personal experiments and results.

However, if you're experiencing low mAP on the PASCAL VOC dataset, I would suggest reviewing the training hyperparameters and ensuring that your dataset is correctly annotated and formatted. It's also important to use a suitable pre-trained model and potentially adjust the learning rate and loss gains as per your dataset's specific characteristics.

For detailed instructions on training on a custom dataset, you can refer to our official documentation at https://docs.ultralytics.com. If you're still facing issues, please open a new issue on the GitHub repo with details of your training setup, and we'll be happy to help you troubleshoot.