COCO interpolation and integration

karl-gardner commented 2 years ago

Search before asking

[X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hello Glenn,

I have been diving into the mAP code recently and I see that you put the following source: https://github.com/rafaelpadilla/Object-Detection-Metrics

I have read this source and how rafael evaluates the mAP for PASCAL VOC dataset. However, that uses the 10 point interpolation or the all point interpolation. I am wondering where you specifically get the specific lines of code where you accumulate the maximums and integrate the composite trapezoidal rule with np.trapz( ):

I am wondering where I can find more information on this specifically?

Thanks,

Karl Gardner

Additional

No response

glenn-jocher commented 2 years ago

@karl-gardner this is my implementation I believe, but the 101 points comes from the pycocotools inteprolation point count.

karl-gardner commented 2 years ago

@glenn-jocher I understand that COCO uses 101 points for the interpolation count and I also understand how you are accumulating the FPs and TPs in line 61 of the metrics.py file:

Accumulate FPs and TPs fpc = (1 - tp[i]).cumsum(0) tpc = tp[i].cumsum(0)

https://github.com/ultralytics/yolov5/blob/master/utils/metrics.py#L61

However, one thing that I don't understand is after you append sentinel values to the mrec and mpre variables (I don't know what m is here) you compute the maximums of the precisions which you label "compute the precision envelope":

https://github.com/ultralytics/yolov5/blob/master/utils/metrics.py#L110

But when rafael padilla did the 11 point interpolation he used the 11 points to find the maximum values:

https://github.com/rafaelpadilla/Object-Detection-Metrics

With doing this with all points, does this mean you are doing an all point interpolation first like rafael padilla did:

Thanks,

Karl Gardner

glenn-jocher commented 2 years ago

@karl-gardner sorry I don't quite understand your question, but if you have a proposed change please submit a PR with before and after mAPs and speeds to quantify the differences.

karl-gardner commented 2 years ago

@glenn-jocher I do not have a proposed change actually. I am just trying to understand the steps involved in the AP calculation. Would you be able to review the following steps with a practical example for your average precision calculation and confirm if it looks correct?

A practical example Considering the set of 12 images in the figure below:

Each image, except (a), (g), and (j), has at least one target object of the class cat, whose locations are limited by the green rectangles. There is a total of 12 target objects limited by the green boxes. Images (b), (e), and (f) have two ground-truth samples of the target class. An object detector predicted 12 objects represented by the red rectangles (labeled with letters A to L) and their associated confidence levels are represented in percentages. Images (a), (g), and (j) are expected to have no detection. Conversely, images (b), (e), and (f) have two ground-truth bounding boxes. To evaluate the precision and recall of the 12 detections it is necessary to establish an IOU threshold t, which will classify each detection as TP or FP. In this example, let us first consider as TP the detections with IOU > 50%, that is t=0.5.

Step 1: Accumulate true positives, false positives, precision and recall Lines of code in metrics.py: https://github.com/ultralytics/yolov5/blob/master/utils/metrics.py#L62-L63

As stated before, AP is a metric to evaluate precision and recall in different confidence values. Thus, it is necessary to count the amount of TP and FP classifications given different confidence levels.

Step 2: Graphical representations of the precision x values presented in case t= 0.5 Lines of code in metrics.py: https://github.com/ultralytics/yolov5/blob/master/utils/metrics.py#L106-L107

Step 3: In the All-point interpolation approach, all points are considered. Lines of code in metrics.py: https://github.com/ultralytics/yolov5/blob/master/utils/metrics.py#L110

Step 4: 101-point precision defined in COCO challenge. Lines of code in metrics.py: https://github.com/ultralytics/yolov5/blob/master/utils/metrics.py#L115-L116

Step 5: Integrate using np.trapz() at each of the 101 points in the curve Lines of code in metrics.py: https://github.com/ultralytics/yolov5/blob/master/utils/metrics.py#L116

Thanks,

Karl Gardner

RohitSingh1226 commented 2 years ago

@karl-gardner is there a way to approach points via spline-interpolation

github-actions[bot] commented 2 years ago

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

Wiki – https://github.com/ultralytics/yolov5/wiki
Tutorials – https://docs.ultralytics.com/yolov5
Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

Ultralytics HUB – https://ultralytics.com/hub
Vision API – https://ultralytics.com/yolov5
About Us – https://ultralytics.com/about
Join Our Team – https://ultralytics.com/work
Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

ultralytics / yolov5