How to deal with unseen entities in WN18RR and FB15K237?

TimDettmers / ConvE

Convolutional 2D Knowledge Graph Embeddings resources

MIT License

675 stars 163 forks source link

How to deal with unseen entities in WN18RR and FB15K237? #21

Closed boyang-nlp closed 6 years ago

boyang-nlp commented 6 years ago

Hi, I have noticed that test.txt/valid.txt of WN18RR and FB15K237 contains entities that are not appear at train.txt,. It means that these entities will never be updated during the training, how to deal with these unseen entities?

TimDettmers commented 6 years ago

Thank you for noting this an opening an issue. Other people also recognized this issue and so I copy a reply from previous issues:

I was unaware of this when I designed the dataset. Currently, it would be consistent with other work, if you just treat these entities as any other. This falsely inflates the error rates, but if everyone treats this in the same manner there are no problems — we just need to be able to compare scores between models.

Please also see #6 for an extended discussion.