Code for time-aware filtered MRR computation

JuliaGast / TGB2

Temporal Graph Benchmark project repo

1 stars 0 forks source link

Code for time-aware filtered MRR computation #7

Closed JuliaGast closed 5 months ago

JuliaGast commented 6 months ago

[x] time-aware filtered mrr computation: integrate
[x] do we also log hits@1, 3, 10? I think this would be useful.

shenyangHuang commented 5 months ago

For the benchmark, we should only track one metric per dataset. Of course for our development process, we can look at different metric and see what makes sense the most. I think MRR would be the most strict metric as if you evaluate with many (potentially all negatives), the model has to rank the positive edge as top 1. While hits@10, it just needs to be top 10.

Not sure how to do time aware MRR correctly, can you walk us through it at the meeting?

shenyangHuang commented 5 months ago

knowledge graph on OGB is computed with MRR: https://ogb.stanford.edu/docs/linkprop/

shenyangHuang commented 5 months ago

This is solved in pull request #12