Question: Is it possible to get the AR(Average Recall) using this tool?

[yqtianust]

Hi,

I am trying to get the AR(average recall) defined by coco . I am wondering to know is it possible that I can get this value directly from the results returned after running python pascalvoc.py.

I noticed that it return a list 'recall' when running python pascalvoc.py. However, I am not sure the relationship between this 'recall' and the AR defined by coco. Can you give me some explanation?

Thanks.

[rafaelpadilla]

Dear @yqtianust ,

The AR as implemented on COCO is not available here yet. We are planning to extend our code to do it. We should release a new version of our library soon with the COCO metrics.

You could use our code to implement it though. As stated at the COCO`s evaluation instruction: The AR is the maximum recall given a fixed number of detections per image (1, 10 or 100), averaged over categories and IoUs. 11 IOUs are used (50%, 55%, 60%, 65%, ..., 95%). But we highly recommend you to check carefully your results with their official implementation.

Regards

[yqtianust]

Dear @rafaelpadilla Thanks for replying.

I will try to implement it. Actually, I am struggling with two concepts:

1. I don't understand why is maximum:

The AR is the maximum recall given a fixed number of detections per image

2. For given a fixed number of detection, do these detection only contains the detection of one specific category or it contains all?

Any suggestion?

Thanks

[rafaelpadilla]

@yqtianust,

1. I suggest you to read this paper to get more familiar with the AR. Also make some online research to find better explanations.

2. You should take a look at how COCO does in their official code. Their page says: "AR is the maximum recall given a fixed number of detections per image, averaged over categories and IoUs. AR is related to the metric of the same name used in proposal evaluation but is computed on a per-category basis." See it here.

Regards

[yqtianust]

@rafaelpadilla Thanks.

I think I got the point.

First, we need get the max recall for a single class, using max_r = max(metricsPerClass['recall']); acc_r = acc_r+max_r

Second, for a single threshold, we need get the average (over all classes) of max recall, avg_recall = acc_ar/num_valid_classes

Last, we need compute the average over all threshold.

Using the above method, and IoU =[0.5:0.05:0.95], I got the AR = 0.5987 while cocoapi got 0.595 For comparison, I got AP using your code = 0.574, while cocapi got 0.573

I guess I am correct?

[rafaelpadilla]

Dear @yqtianust ,

Our implementation by default uses the Accumulated AP, as done by PASCAL VOC challenge, I would need to look at their code to see if they implement the 11-Point AP or the Accumulated AP.

Our implementation also has the 11-point average precision implementation. For that, change the "method" parameter from "method=MethodAveragePrecision.EveryPointInterpolation" to "method=MethodAveragePrecision.ElevenPointInterpolation".

I highly recommend you to check COCO implementation to see how COCO performs the interpolation on AP.

Hope it helps

[yqtianust]

@rafaelpadilla Thanks.

Definitely, I will try to read the source code. However, it seems it combines both python and cpp. It hard for me, as a newbie.

I notice a line in cocoapi, eval.py line26,

recThrs - [0:.01:1] R=101 recall thresholds for evaluation

[rafaelpadilla]

@yqtianust ,

In a near future, I will make another function with coco's metrics. :)

Regards, Rafael

Please, close this issue if your question regarding this issue was answered properly.

[yqtianust]

@rafaelpadilla Thanks for your answer.