search

bys0318 / SQUIRE

EMNLP 22' (Oral): SQUIRE: A Sequence-to-sequence Framework for Multi-hop Knowledge Graph Reasoning
31 stars 5 forks source link

Link Prediction #10

Open wanggang1111111 opened 4 months ago

wanggang1111111 commented 4 months ago

The link prediction results based on embedding models in the paper are so high, why is the Hit@10 result of the TransE model only 0.47?

bys0318 commented 4 months ago

I guess there is a difference in the evaluation setting. In our experiment, we measure the prediction accuracy only on the tail entity (t) in the test triplets, following previous works on multi-hop reasoning. Meanwhile, in the line of KG completion researches, they usually measure the accuracies on both the head and tail entities in the test triplets, and report their average. This accuracy is often lower than the accuracy on only the tail entities, since predicting the head entities is a harder task (1-N).

wanggang1111111 commented 4 months ago

Since this code involves multi-hop knowledge graph reasoning, why not consider conducting experiments on multi-hop knowledge graph reasoning datasets such as MetaQA?

bys0318 commented 2 months ago

Correct me if i'm wrong, but I think MetaQA mainly focuses on testing how well a QA system can take a natural-language (NL) form multi-hop query and ground the NL query to entities/relations in the KG to answer it. It is a different problem setting from our paper.