Open cthoyt opened 2 years ago
We will add it to the library! Thank you @cthoyt :)
Reference implementation in PyKEEN here: https://github.com/pykeen/pykeen/blob/139c3d1db77c9e443d03db5383a163297345adfd/src/pykeen/evaluation/rank_based_evaluator.py#L468-L478
I am one of the authors and happy to contribute an implementation. To this end, I have a question regarding the existing metrics, e.g., https://github.com/AstraZeneca/rexmex/blob/8e5d0ad392ecd6caf83a8b09a1f59bb4b8f89735/rexmex/metrics/ranking.py#L64-L73
If I understand the API correctly, you assume that there is a single recommendation
, a sequence of items, and some of them are relevant (relevant_items
). The mean in mean rank is then computed over the relevant items.
This seems to be a bit different than the evaluation protocol frequently used in (Knowledge Graph) Link Prediction, where we have multiple recommendations (e.g. for each head
-relation
pair), each with relevant items (the tail
entities), and compute the mean over all of these.
@mberr's adjusted mean rank address some of the problems with the mean rank, including its size dependence. Reference: https://arxiv.org/abs/2002.06914