search

recommenders-team / recommenders

Best Practices on Recommendation Systems
https://recommenders-team.github.io/recommenders/intro.html
MIT License
19.36k stars 3.12k forks source link

[FEATURE] Add r-precision metric #2067

Closed daviddavo closed 7 months ago

daviddavo commented 9 months ago

Description

R-precision can be defined as the precision@R, with R being the number of relevant documents for the query [1]. This means that for a certain user $u$ which has $R$ relevant items, we would ask the recommendation engine for the top $R$ recommendations, and see which of them are true positives.

[1]: C. D. Manning, P. Raghavan, and H. Schütze, ‘Chapter 8. Evaluation in information retrieval’, in Introduction to information retrieval, New York: Cambridge University Press, 2008, pp. 151–175.

Expected behavior with the suggested feature

Adding R-Precision will allow evaluating systems with some characteristics.

Other comments

Implementation idea:

  1. Get the number of relevant documents (R) for all users
  2. Get the top-k recommendations of the model where $k = \max R$
  3. Calculate the R-precision for each user, and return the mean as in other metrics
miguelgfierro commented 9 months ago

@daviddavo would you be able to contribute to this?