mAP question

[andikira]

Hello @Cartucho i have some question about mAP. as far as i know the mAP is method for evaluate object detection task, but i have confuse for the result. I try to set a different threshold and i got mAP and predicted object. when i set the threshold very low (0.01) i got higher mAP but more false prediciton and when i set the threshold to 0.5 i got lower mAP but fewer false prediction, like pic below

i'm newbie in object detection, but i think the more false prediction mean lower mAP, am i right ? another question, is the mAP doesn't represent the object detection performance ? or there are another way to evaluate object detection task ?

i'm sorry when this question is not proper to ask here, if yes i will close/delete it asap. thank you.

[Cartucho]

It's perfectly fine to ask here. Give me a sec and I will explain it to you.

Could you please provide the plot of the AP of the class Monosit in both cases? That will help me explain!

[andikira]

Thanks for your response.

monosit AP for threshold 0.01 and 0.5

[Cartucho]

So, first of all, I recommend you to see this video: mean Average Precision

Basically, the mAP is a single-number metric used to evaluate rankings. Well, this is great for object recognition detectors since the predictions are associated with a confidence level and allows us to rank these predictions from higher to lower confidence and then get the mAP value.

In practice the higher the confidence of a detection (from 0% to 100%) the more important it will be. Specifically what happens at rank 1 is twice as important as what happens at rank 2. So it tells how good is your detector taking into account the confidence for each prediction.

The AP is calculated by the area (shown in blue) of each of the plots above. Each of the points in that graph correspond to a prediction and they are ordered by confidence. These points go down when there is a false detection and up when there is a true detection. In the gaph of the left the blue dot didn't go down 33 times (corresponding to the true detections) and in the graph of the right 32 times.

As you can see from the left plot the false predictions are all concentrated in the end and probably associated with low confidence levels, meaning that in terms of mAP you have a very good model. In this case if you find the right threshold for this class that will remove those last points in the end (try for example a threshold of 0.1) you can even get a higher AP. If you get creative you can even find the right threshold for each class.

Are there any other way to evaluate object detection task ?

mAP is the standard metric used in the research papers. Your model seems to be working very well (in fact it even seems too good to be true). You can also have a look at other metrics like ROC curve.

[andikira]

This is a great explanation, now i have more good intuition about mAP

In this case if you find the right threshold for this class that will remove those last points in the end (try for example a threshold of 0.1) you can even get a higher AP. If you get creative you can even find the right threshold for each class.

I have try 0.1, 0.05, 0,03, 0,02, and 0,01 threshold, the best mAP is 0.02 (93.33%) this is better 0.01% than 0.01 (93.32%), but i think mAP doesn't significantly decrease to the increasement of false prediction (for low rank/confidence) , am i right ?

and what do you think about F1-value for object detection evaluation ?

[Cartucho]

Yeah, you are right, it didn't make much difference since they are the last ranks!

Well, it truly depends on your application, the F1 value is used in the ROC curves so watch some videos about it (there are great ones on youtube). Basically, it depends on how many false detections do you want to allow.

First, try to understand what's Precision and Recall. F1 is just a way to balance them both.

[andikira]

ok thank you so much sir, i'll learn more

[hibaalqaysi]

In this case if you find the right threshold for this class that will remove those last points in the end (try for example a threshold of 0.1) you can even get a higher AP. If you get creative you can even find the right threshold for each class.

Hello @Cartucho , How is it possible to change/adjust the conf. threshold in your repo please?