Closed celsofranssa closed 1 year ago
MRR@k is only equal to Recall@k if you have exactly one positive per query. Unfortunately, there are many meanings for Recall, but at least for trec eval (which ranx follows) it always uses positives_found/total_positives, even if k is smaller than the amount of positives you have.
For the other definitions, one I have seen called as R_cap (using positives_found/min(k,total_positives)) or Success ( the function "found any positives @ k").
Recall@1 is equal to MRR@1 only if you have exactly 1 relevant document per query, otherwise they are not as Recall takes into consideration the total number of relevant documents (retrieved + non-retrieved).
You can find the definition of both metrics in ranx's documentation (Recall - MRR) and Wikipedia (Recall - MRR).
I see, Thank you for your answers.
Describe the bug
MRR@1
should be equal toRecall@1
. However, these metrics diverge for the case below.To Reproduce
or use Google Colab.
Expected behavior
mrr@1
=recall@1
Am I missing something?