search

RichardHGL / WSDM2021_NSM

Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals. WSDM 2021.
130 stars 22 forks source link

MetaQA-3hop test coverage 100% #14

Closed lihuiliullh closed 2 years ago

lihuiliullh commented 2 years ago

Hi, may I know how you check the MetaQA-3hop test coverage? In the paper, the coverage is 99%. But I check it myself, the coverage is 100%. I want to know whether there is a difference between our metrics. I simple find all the 3hop entities to the topic entity, and if any answer node is contained in these 3hop entities, I think this test question is covered.

RichardHGL commented 2 years ago

No, the coverage is calculated with the extracted subgraph. The extracted subgraph for every question may not contain all entities in 3 hop. And it's not based on test set, it's based on all questions (train + dev + test)