Implement ranking strategies as followed by ConvKB and by Old Dogs New tricks paper

[sumitpai]

Background and Context In AmpliGraph, while evaluating corruptions, if the test set triple gets same score as any of the corruptions, we assign the worst rank. There are other approaches followed in literature. We should implement all these strategies to allow the user to compare model performance using all three approaches. Description Let's look at each of the three strategies in detail with an example.

Assume there are only 10 corruptions, and assume that all the corruptions get the same score as the test triple. The ranks assigned by the three strategies are as follows:

• Assign the worst rank i.e. the test set triple gets a rank of 11. This is followed by most papers in the literature. This is the strictest approach and it drives down the mrr by a large margin if there are many ties.
• Assign the middle rank i.e. the test set triple gets a rank of 6. We found this strategy being used by ICLR 2020 paper¹. This approach seems to be fair towards the model in resolving the ties as it assigns the middle rank to break ties.
• Assign the best rank i.e. the test set triple gets a rank of 1. This approach is followed by ConvKB². This approach is overly biased and helps the model achieve a very good mrr in case of ties.

References:

1. Ruffinelli et al., You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings, ICLR 2020
2. Nguyen et al., A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network, NAACL 2018

[wradstok]

I think it may be interesting to look at this paper as well: https://arxiv.org/abs/2002.06914.

They call worst/middle/best ranking pessimistic/realistic/optimistic and also list methods used for many existing models. Additionally, they introduce a new ranking mechanism called Adjusted Mean Rank (AMR) which should make it possible to compare results between datasets or different train/test splits on the same dataset.

[sumitpai]

Fixed by pull request #214