dnns92
commented
2 years ago Interestingly enough, the average recall can go beyond 1, but not beyond 2. One can check this by using a label as a prediction. This leads to recall = 1.0 as it should be. Using the same label twice as a prediction will lead to recall=2.0, Using the same label 3 times however does not lead to recall=3.0.
Using the same label 3 times: Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 1.00000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 1.00000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 1.00000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.00000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 1.00000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 1.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 2.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 2.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 2.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.00000
In this code recall is computed as a number_of_true_positive_detections / number_of_not_ignored_groundtruths, which is wrong in case there are multiple true positive detections of the same groundtruth object. Number_of_detected_groundtruths / number_of_not_ignored_groundtruths would be the correct way to compute this metric.
https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L372
https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L384
https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L390
I noticed the issue when an evaluation returned AR > 1.