search

AstraZeneca / rexmex

A general purpose recommender metrics library for fair evaluation.
https://rexmex.readthedocs.io/
275 stars 26 forks source link

Adjusted mean rank #9

Open cthoyt opened 2 years ago

cthoyt commented 2 years ago

@mberr's adjusted mean rank address some of the problems with the mean rank, including its size dependence. Reference: https://arxiv.org/abs/2002.06914

benedekrozemberczki commented 2 years ago

We will add it to the library! Thank you @cthoyt :)

cthoyt commented 2 years ago

Reference implementation in PyKEEN here: https://github.com/pykeen/pykeen/blob/139c3d1db77c9e443d03db5383a163297345adfd/src/pykeen/evaluation/rank_based_evaluator.py#L468-L478

mberr commented 2 years ago

I am one of the authors and happy to contribute an implementation. To this end, I have a question regarding the existing metrics, e.g., https://github.com/AstraZeneca/rexmex/blob/8e5d0ad392ecd6caf83a8b09a1f59bb4b8f89735/rexmex/metrics/ranking.py#L64-L73

If I understand the API correctly, you assume that there is a single recommendation, a sequence of items, and some of them are relevant (relevant_items). The mean in mean rank is then computed over the relevant items.

This seems to be a bit different than the evaluation protocol frequently used in (Knowledge Graph) Link Prediction, where we have multiple recommendations (e.g. for each head-relation pair), each with relevant items (the tail entities), and compute the mean over all of these.