search

facebookresearch / ssl-relation-prediction

Simple yet SoTA Knowledge Graph Embeddings.
Other
109 stars 19 forks source link

Quick check with the new evaluation code #17

Closed weihua916 closed 2 years ago

weihua916 commented 2 years ago

Hi! This is OGB Team.

We recently released https://github.com/snap-stanford/ogb/releases/tag/1.3.4 which improves the MRR calculation. This should not change your result if your model makes different predictive scores for different triplets, but would penalize those models that give the same predictive scores for different triplets (which is not ideal for link prediction).

Could you update your ogb package to 1.3.4 and run your model again just to confirm everything stays the same (the dataset stays the same. only evaluator changed)? Just one seed should be enough. Thanks!

yihong-chen commented 2 years ago

@weihua916 Thanks for the notice. I have rerun the experiments on ogbl-biokg and ogbl-wikikg with one seed. The results stay the same with ogb 1.3.4.

weihua916 commented 2 years ago

That's good to confirm. Thank you!