Closed JuliaGast closed 5 months ago
For the benchmark, we should only track one metric per dataset. Of course for our development process, we can look at different metric and see what makes sense the most. I think MRR would be the most strict metric as if you evaluate with many (potentially all negatives), the model has to rank the positive edge as top 1. While hits@10, it just needs to be top 10.
Not sure how to do time aware MRR correctly, can you walk us through it at the meeting?
knowledge graph on OGB is computed with MRR: https://ogb.stanford.edu/docs/linkprop/
This is solved in pull request #12