Did you test GIoU with yolov3-spp.cfg?

[AlexeyAB]

@nathantsoi Hi, Great work!

Whether in all cases you get an increase of AP@[.5, .95] and mAP@0.75, and decrease of mAP@0.5 by using GIoU instead of MSE?

Did you test GIoU with yolov3-spp.cfg https://github.com/AlexeyAB/darknet/blob/master/cfg/yolov3-spp.cfg

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nathantsoi]

good idea @AlexeyAB, i did not try this, but it would be interesting to do!

[iraadit]

Is it something you think about doing?

[nathantsoi]

Absolutely, just a bit short on time at the moment, if someone wants to try it in the meantime? It's a great idea!

On Tue, Jun 25, 2019, 7:22 AM Renaud Bouckenooghe notifications@github.com wrote:

Is it something you think about doing?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/generalized-iou/g-darknet/issues/12?email_source=notifications&email_token=AADBTGXQPFBFLYQPVATBNXDP4H5XTA5CNFSM4HTDM522YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYP473Y#issuecomment-505401327, or mute the thread https://github.com/notifications/unsubscribe-auth/AADBTGVCHFKHX4XEQ3G7BMDP4H5XTANCNFSM4HTDM52Q .

[HoracceFeng]

Hi, I am also looking forward to see the results and comparison between the original yolov3 and this GIOU version. It would be better if you can provide a table with the comparison. Thx~

[nathantsoi]

Ok, I have tested SPP w/ MSE loss vs SPP w/ GIoU loss. Results are positive. Note that I have not done a search for a better normalizer value to balance between classification and regression component of the loss. Results can likely be further improved with some limited search for a better normalizer.

Also note that unlike in the original Yolo work, we report mAP as calculated in the MS COCO Benchmark and described in our paper under section 2. Related Work, in the first paragraph. Essentially, this is the average mAP at a range of IoU thresholds. This is done for the reasons described in the same section of the paper noted above.

Baseline (MSE Loss) MSE Loss, IoU Metric: 0.329 mAP MES Loss, GIoU Metric: 0.318 mAP

GIoU Loss GIoU Loss, IoU Metric: 0.338 mAP, a 2.74% improvement GIoU Loss, GIoU Metric: 0.328 mAP, a 3.14% improvement

[AlexeyAB]

@nathantsoi That's great!

GIoU Loss GIoU Loss, IoU Metric: 0.338 mAP, a 2.74% improvement GIoU Loss, GIoU Metric: 0.328 mAP, a 3.14% improvement

Can you explain please, what is the difference between GIoU Loss, IoU Metric and GIoU Loss, GIoU Metric? Did you switch between them before training or after? And what parameter in cfg-file do you use to switch between them?

To switch between MSE and GIoU we use iou_loss=giou / mse in cfg. But how to switch GIoU Metric to IoU Metric?

[nathantsoi]

Ok, let me recap a bit, then address your specific questions. Some of this is probably obvious to some folks, but may not be to others. Also, note that a similar overview of this (with illustrations 😄) is available on the project website: https://giou.stanford.edu

Loss (aka cost function):

• During training this is used to determine how far a prediction is from the ground truth. This is used to see how well the network is learning during training and the derivative of this function is back-propagated and updates the weights and biases in the network.
• The Yolo loss function could be considered a multi-task loss, since it has 2 components: one part is for regression (where are the bounding boxes?) and the other part is for classification (what type of object is in the bounding box?). With GIoU, we only change the regression component of this loss, we do not change the classification component at all.
• With regards to the regression component: Yolo's normal network loss is MSE (mean squared error). Our work proposes using GIoU as a loss.
• Note that since the loss is used during training, we train the network using the same loss throughout. Switching losses during training is an idea, but beyond the scope of this work. We do not switch losses during training.

Metric (aka evaluation)

• Better described in this context as an "evaluation metric," the IoU metric and GIoU metric gives us a way of quantifying our results. Let's talk about the standard IoU metric first, then I will describe how the GIoU metric is different.
• Note that the GIoU-based mAP metric (aka GIoU metric) is a new metric we propose. For direct comparison against existing works, the standard IoU-based mAP (aka IoU metric or just mAP) should be used.
• The MS COCO toolkit provides code to calculate the widely used AP (average precision) and mAP (mean Average Precision) metrics. These calculations always use IoU.
• Let's first talk about the difference between AP and mAP, then talk about IoU vs GIoU in this context. Note that we can use any combination of ((AP or mAP) and (IoU or GIoU)) when reporting the evaluation metric.
• When calculating AP, an IoU threshold must be chosen. That is why AP is sometimes shown as AP-50 or AP-75, this means, AP with an IoU threshold of .5 or .75 respectively. See this article for more details. mAP averages the AP values from 0.5 to 0.95 in 0.05 increments. Note that when reported elsewhere AP and mAP always use IoU.
• For the GIoU metric, we modify the MS COCO toolkit (here) to use GIoU in place of IoU. We compute AP or mAP the same way as described above, but with GIoU in place of IoU.
• You'll notice the GIoU metric values are slightly lower than the IoU metric values. This is because of the "C" term in the GIoU formulation, which gives us some idea of the difference in shape properties and distance between prediction and ground truth, even in cases of 0 overlap. Check the formulation in the paper for more detail.
• The IoU or GIoU metric is computed after training the network. Once a network is trained on MSE or GIoU loss, we run the evaluation code as described here: https://github.com/generalized-iou/g-darknet#coco

To answer your specific questions @AlexeyAB:

Did you switch between them before training or after? loss: One loss chosen per network and only the chosen loss is used during training. Therefore, when testing 2 losses, 2 networks must be trained independently. metric: Both IoU (normal COCO metric) and GIoU (our new proposed metric) can be used to evaluate a trained network. Evaluation Instructions for COCO

And what parameter in cfg-file do you use to switch between them? loss: iou_loss metric: There is no parameter, but the evaluation scripts can be modified to run either or both. Run [scripts/coco_all_map.py] for both IoU and GIoU or edit [scripts/coco_eval.py] to use IoU (default) or GIoU

I hope that helps. Let me know if I missed or can clarify anything.

[AlexeyAB]

@nathantsoi Thanks for the very detailed answer! All clear. In addition to the new loss, you also offer a new accuracy metric AP that is based on GIoU instead of common IoU.

[nathantsoi]

No worries, thx for your great question, @AlexeyAB!

Also, I mentioned this in the main readme, but thanks again for all the hard work you've done on your fork and for the help getting started working on Darknet! I think it would be great to get GIoU loss integrated in your code base (which has many other nice optimizations). Maybe the evaluation code as well? Let me know what you think, and if this is a reasonable idea, what the right way forward is. I can try to submit a PR if you would like, or better highlight where I've implemented the loss?

[AlexeyAB]

@nathantsoi Hi,

Thanks for suggestion. I already implemented GIoU in my fork, you can see it there in column Done - this is the roadmap to Yolo v4: https://github.com/AlexeyAB/darknet/projects/1

Also there is comparison of different models with new features on custom small dataset: https://github.com/AlexeyAB/darknet/issues/3114#issuecomment-494148968

If you can add any suggestion, or if you can test these models on MS COCO or other competitions, that would be just great! Currently I just don't have a time, I don't have enough computing power and just today I had serious difficulties in my life (health). Thanks again for the openness!

[nathantsoi]

First, I am sorry to hear about your health difficulties, @AlexeyAB. No problem at all about the timing. I hope you make swift and full recovery!

Re. Yolo: I am glad to hear that GIoU could be helpful! I like all the tests you ran and I would be glad to help run some. One suggestion is that after training, computing the standard IoU based mean AP (aka AP), AP-50 and AP-75 for each network could be helpful so results can be directly compared with other networks (via the coco challenge: https://competitions.codalab.org/competitions/5181#results)

Hope you get well soon!

[LukeAI]

Why is it that there is an increase in mAP and AP75 but a decrease in AP50 with GIoU loss? I'm wondering if somehow, correct classifications are slightly less likely but the bounding boxes are tighter?

[AlexeyAB]

@LukeAI

correct classifications are slightly less likely but the bounding boxes are tighter?

Yes.