search

intfloat / SimKGC

ACL 2022, SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models
188 stars 36 forks source link

Accuracy of Wiki-trans #6

Closed EurusNotes closed 2 years ago

EurusNotes commented 2 years ago

Hi, Liang.

I was wondering why the accuracy of the backward predictions in Wiki-trans is very poor? In the other datasets, while there is some difference between the forward predictions and backward predictions, the difference is not as large. What do you think is the reason for this?

intfloat commented 2 years ago

This is mainly due to the characteristics of the Wikidata5M datasets.

For example, given a triple (jacques villeneuve sr., place of birth, berthierville quebec), backward prediction requires predicting jacques villeneuve sr. given (?, place of birth, berthierville quebec). A lot of entities share the same place of birth berthierville quebec, and Wikidata misses many such facts due to its incompleteness. As a result, many predictions by the model are false negatives. This is one flaw of current automatic evaluation metrics.

In Section 5.4 of the paper, we conduct a small-scale human evaluation to quantitatively examine this phenomenon.

EurusNotes commented 2 years ago

Liang, Thank you for your reply, it's very helpful.