search

rafaelpadilla / review_object_detection_metrics

Object Detection Metrics. 14 object detection metrics: mean Average Precision (mAP), Average Recall (AR), Spatio-Temporal Tube Average Precision (STT-AP). This project supports different bounding box formats as in COCO, PASCAL, Imagenet, etc.
Other
1.09k stars 215 forks source link

How to make PR curve monotonically decreasing #62

Closed ashra-main closed 3 years ago

ashra-main commented 3 years ago

Hii, Thank you so much for this, saved me a ton of time!

I have an issue with the Pascal VOC implementation: I was reading the Pascal VOC 2012 doc which defines AP like,

image

I was wondering why the outputs are not monotonically decreasing when I run this code?

image

rafaelpadilla commented 3 years ago

Hi @arashash,

Thank you for your comment.

This behavior is not caused by the tool. Our tool produces the exact same results as the official PASCAL VOC 2012 development kit. This is a natural behavior caused by the metric itself, and you would see the same behavior in plots produced with the MATLAB tool. Such zig-zag shape may appear in many cases and is totally correct.

The paper refers to the "computation" of the AP. Not the curve itself. The computation is not made on the plotted curve but in its interpolated monotonic version.

(I assume that you are aware of the computation of the Precision x Recall curve, so I won't review it here. In case you need a detailed explanation, I suggest taking a look at this example and the example at section "5. A Numerical Example" in the paper.)

The curve is indeed zig-zagged, but the computation is made on the monotonic version of the curve, which is represented by the red dotted line in the plot below:

image

I hope it answers your doubt.

Best regards,

ashra-main commented 3 years ago

Woow thanks a million. This cleared up everything for me.

rafaelpadilla commented 3 years ago

Glad to help, @arashash :)

I will close this issue. If you have further questions, you can reopen it or open a new one.