DerrickXuNu / OpenCOOD

[ICRA 2022] An opensource framework for cooperative detection. Official implementation for OPV2V.
https://mobility-lab.seas.ucla.edu/opv2v/
Other
644 stars 99 forks source link

Some wrong in the eval_utils.py #101

Closed happy0415 closed 1 year ago

happy0415 commented 1 year ago

First of all, thank you for the amazing work you have done!

However, I have noticed that there might be a potential issue in the calculate_ap function in the eval_utils.py file. When calculating the AP, all samples need to be sorted by their score. However, the boxes obtained by the calculate_ap function are only sorted within a single batch in the calculate_tp_fp function. This means that the sequence is currently only locally sorted, but not globally sorted.

I came across a solution in the CoAlign, which provides the correct modification to address this issue. The suggested modification involves saving the corresponding score in calculate_tp_fp and sorting again in calculate_ap. After testing, the modified implementation can potentially improve the accuracy by 10 to 20.

DerrickXuNu commented 1 year ago

Thanks for pointing out the issue! Yes, indeed, the score is only sorted locally, and sorting them globally can actually increase the AP. However, on the other hand, since all methods using opencood all use locally sorted evaluation, it should be a fair comparison. If you plan to use global sorting evaluation in your paper, you may need re-run the other methods for testing as well, which shouldn't be much work as the checkpoints are already there.