How to calculate denotation accuracy for queries for Overnight?

[lukovnikov]

Various papers report denotation accuracy on Overnight, which requires executing the logical form on a database. (The Overnight paper specifies that "We created a database for each domain by randomly generating facts using entities and properties in the domain(with type-checking).") However, it is unclear how to easily calculate denotation accuracy from the given data. Could you please point me in the right direction?

EDIT: I found SimpleWorld

[subendhu19]

I have a similar question for domains in the overnight dataset. I have a text file of predicted canonical forms from which I want to compute the denotation accuracy. The canonical forms in this file are predicted by a simplification model that simplifies general utterances into a fixed canonical form so that they can then be parsed by the grammar rules. So now, I need to run the fixed grammar rules to parse the predicted canonical utterances and then execute the logical forms to get the denotation accuracy. What commands should I run for this?

[ppasupat]

I'm not familiar with the overnight package. That said, there are other works on Overnight that evaluated on denotations. The best shot might be Neural Semantic Parsing over Multiple Knowledge-bases which has examples of how to run the evaluation

[subendhu19]

Thank you. The link provides an evaluator to execute the logical forms and it works. However, I'm still looking for a command to parse canonical utterances with the fixed grammar files to produce the logical forms. Loading the grammar files in the "simple" mode isn't producing any logical form candidates in overnight domains.

[ppasupat]

It looks like the authors of the overnight paper also has a Codalab here: https://worksheets.codalab.org/worksheets/0x269ef752f8c344a28383240f7bb2be9c

There might be some runs that demonstrate how to parse input sentences into logical forms, but I'm not totally sure.

[subendhu19]

Thank you for pointing it out. There are no clear instructions in the codalab workbook. It appears most of the runs also involve some sort of training.

I tried the following command and it parses some sentences but not all of them.

./run @mode=simple -Grammar.inPaths overnight/basketball.grammar -Grammar.tags general parse -Builder.parser FloatingParser

To be clear, the grammar files are all there. I just need a way to use the grammar in those files, take a file where each line contains a natural sentence as input and generate the formulae according to the grammar into each line in an output file. Assuming overnight grammar files are like any other grammar files, is there a command to do this?

[makeshn]

@subendhu19 did you figure out how to do it? If you could please share that, it would be great! Thanks.