search

AskNowQA / LC-QuAD

A data set of natural language queries with corresponding SPARQL queries
GNU General Public License v3.0
91 stars 30 forks source link

Literals in the evaluation #15

Open yhshu opened 3 years ago

yhshu commented 3 years ago

I’m working on a KBQA system on LC-QuAD 1.0. I would like to ask some questions about the evaluation. Should I include the literal in golden answers when I execute the golden SPARQL against DBpedia 1604? I noticed that all lambda variables in golden SPARQLs are ‘?uri’. Since there’s no FILTER in SPARQL for literals, does it mean answers are just URIs? Thanks.

saist1993 commented 3 years ago

In my opinion, the best way to evaluate your system would be to execute the gold SPARQL query (the one given in the dataset) and compare it with the answers retrieved by your system.

Purely from the structure/template of the SPARQL, there might be literals as answers. However, we removed all SPARQL and its associated question while creating this dataset, which returned a literal.