Doubts in AUC-PR

[divyagupta25]

I have a couple of questions in AUC-PR (Section 3.4.1).

• For a given query and a confidence score threshold, why are the incorrect images (based on ground truth) which were not retrieved by the algorithm (score less than threshold) not termed as true negatives?
• Why is AUC-PR not affected if the second or third best match is a correctly retrieved image? Why do you say that it neither penalizes nor rewards retrieval of more than one correct match per query? Won't the precision be high if multiple correct matches are retrieved? And in case we are looking at only a single correct match, then will the precision values be only 0 or 1?
• How does it reward a VPR system only for retrieving a single correct match per query from the reference database?

I guess all questions are related, and I do not understand the single correct match thing.

[MubarizZaffar]

Hey, thanks for your questions. See below my response, hope it helps clarify.

For a given query and a confidence score threshold, why are the incorrect images (based on ground truth) which were not retrieved by the algorithm (score less than threshold) not termed as true negatives?

• I see that you are approaching the thresholding from the perspective of a single query with a list of retrieved images. But in the case of AUC-PR in VPR-Bench, after all the N queries have their corresponding best matches retrieved, then the problem becomes a binary classification problem (1: correct retrieval, 0: incorrect retrieval) with N corresponding best matching scores. There a true-negative would mean a query that does not have a correct match in the database for which we predict an incorrect retrieval, but there are no such queries, i.e., all N queries have a corresponding correct match.

Why is AUC-PR not affected if the second or third best match is a correctly retrieved image? Why do you say that it neither penalizes nor rewards retrieval of more than one correct match per query? Won't the precision be high if multiple correct matches are retrieved?

• Yes, it could be affected, if the binary decision-making that I refer to above is based on the Top-K retrieved images instead of the Top-1. The latter is useful for applications where false-positives are not desirable as the best match. For our implementation of AUC-PR in VPR-Bench, it is based on Top-1, but Recall@K, which is another metric, is based on the Top-K retrieved images.

And in case we are looking at only a single correct match, then will the precision values be only 0 or 1?

• The problem becomes a binary classification problem with N queries. The precision is not 0/1, but it is based on the failure/success of retrieval, i.e., 0/1. Note again: the matching scores are used to rank the queries (with matching scores), not the references.

How does it reward a VPR system only for retrieving a single correct match per query from the reference database?

• I am not sure if I understand this question. AUC-PR in VPR-Bench only cares about the best matching reference, which can either be a correct (1) or an incorrect match (1). There could be multiple correctly retrieved references, but for our instance of AUC-PR for VPR-Bench we only look at the best match (i.e., the reference image with the highest matching score).

I note that your questions are not directly related to issues in the code-base, but are questions about the paper and evaluations in VPR. GitHub issues may not be the best mode for these questions, please feel free to send us an email. Many thanks.