How to calculate PrecisionRateAtN?

[MAX-OTW]

Hi, @Tobias-Fischer @stschubert @oravus ,thank you very much for your work！It helps me a lot！ Here I would like to ask about the calculation of PrecisionRateAtN. I think this is also a measure of VPR performance that can be considered for inclusion in this tutorial. In another work VPR-Bench，https://github.com/MubarizZaffar/VPR-Bench/blob/77ad30f3684476717188c67d7f3f66be19f832af/performance_comparison.py#L71C1-L104C1, this is the code for calculating RecallRateAtN, I would like to ask you how to calculate PrecisionRateAtN? I tried several times, but failed. Can you provide the relevant code? Please note this: Recall@N=TP@N/(TP@N＋ FN@N);Precision@N=TP@N/(TP@N+ FP@N) Hope to receive your reply soon :) Thanks & Best Regards!

[stschubert]

Hi @MAX-OTW, thank you for your question.

Let us start with a brief summary of Recall@N (also called recall@K). It is defined as follows [1]:

[T]he recall@K measures the rate of query images with at least one actually matching database image.

So, the metric assumes for each query image a matching database image, and evaluates if at least one of the retrieved N database images per query image is an actual match. If yes, it is counted as true positive ($TP$), otherwise it is considered a false negative ($FN$), so that $TP+FN=|Q|$ on a whole dataset ($|Q|\ldots$ number of query images). Finally, given $TP$ and $FN$, $Recall@N=\frac{TP}{TP+FN}$.

Now, let's continue with Precision@N: Let's assume, we again retrieve N database images for each query image. For $precision=\frac{TP}{TP+FP}$, we now need true positives ($TP$) and false positives ($FP$). For $TP$, we could use the definition above: If at least one of the retrieved N database images is an actual match, it is considered a true positive $TP$. But for $FP$, I don't see an unambiguous definition:

• On the one hand, we could consider all candidates of our N database images that are not actual matches as $FP$. However, in this scenario an increasing $N$ would decrease the Precision@N performance, especially if N actually matching database images do not exist for a current query. This behaviour would be different to Recall@N, where the performance increases with increasing N.
• On the other hand, we could consider query images without an actual match among its best N database images a FP, but then $Precision@N=Recall@N$, so that we don't need Precision@N.

In summary, I unfortunately don't see a reasonable, unambiguous definition of Precision@N. I hope, this answers your question. Please let me know if I understood something wrong, so that there might be a better answer to your question.

Best, Stefan

[1] Schubert, S., Neubert, P., Garg, S., Milford, M. and Fischer, T. (2023). "Visual Place Recognition: A Tutorial". IEEE Robotics & Automation Magazine

[stschubert]

@MAX-OTW,

I also saw your comment in the VPR-Bench repo:

Recall@N and Precision@N: These metrics are widely applied in the field of information retrieval. They reveal how well the model performs in terms of FPs as well as FNs. I found them in Google search, as below： https://towardsdatascience.com/unveiling-the-precision-n-and-recall-n-in-recommender-system-7a4c6b69d060 ；https://medium.com/@m_n_malaeb/recall-and-precision-at-k-for-recommender-systems-618483226c54

In the second link, the author states:

Precision at k is the proportion of recommended items in the top-k set that are relevant. Its interpretation is as follows. Suppose that my precision at 10 in a top-10 recommendation problem is 80%. This means that 80% of the recommendation I make are relevant to the user.

As stated in my comment above, according to this example, Precision@N for visual place recognition (VPR) would require at least N actually matching database images for each query image, which cannot be guaranteed as there might exist only one matching database image per query. So, Precision@N might be suited for information retrieval, but is rather impractical for VPR.

Best, Stefan