search

UKPLab / coling2018-graph-neural-networks-question-answering

Accompanying code for our COLING 2018 paper "Modeling Semantics with Gated Graph Neural Networks for Knowledge Base Question Answering"
Apache License 2.0
172 stars 29 forks source link

What might be the reason to cause the performance difference between on WebQ and WebQSP #5

Closed marcwww closed 5 years ago

marcwww commented 5 years ago

Really appreciate it for your contribution to the domain of semantic parsing.

May I ask the question as shown in the title of this issue?

Notice that in "Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base" the F1 score on WebQuestion is over 50., and this gated GNN method get over 25. F1 score on WebQSP

Thanks a lot. Your answer will be of great help.

daniilsorokin commented 5 years ago

Hi,

the difference is mainly due to a mismatch between the Wikidata that we use in our experiments and Freebase that was used to get the previous results.

See also Section 5.3 in the paper: "Overall, in 32% of the cases the error was caused by the gap between the KB and the data set. This lets us put the current results into a perspective with the previously reported numbers for the Freebase KB. If we approximately adjust our results for this kind of errors, we achieve between 0.469 and 0.51 F-score."

Best, Daniil