What's wrong with FB15k and WN18?

[GittiHab]

You state

Used in the paper, but do not use these datasets for your research: FB15k and WN18.

What is the problem with using these datasets? They are commonly used in this domain.

[dbadrian]

I recommend you to read the paper for an in-depth discussion.

tl;dr: The general reasoning is that the authors propose that both datasets contain exploitable properties, possibly skewing results making them useless/"misleading" for research. The recommendation given is to use the reduced datasets FB15k-237 and WN18RR instead. So, you need to make up your mind yourself if you want to avoid the datasets or if knowing about it allows you to interpret results differently.

[TimDettmers]

Thanks for the comment @dbadrian! This is exactly right, if you read our paper you find that these datasets contain a large portion of inverse relations that enable leakage from the training set to the validation set. Theoretically, you can use a symmetric model like DistMult and get a score of 0.91 Hits@1 without training any entities — this is not how the task is meant to be! Using FB15k-237 and WN18RR makes sure that any model trained on these dataset does not exploit test set leakage.