Closed kl2806 closed 6 years ago
The original dataset from the 90s marked queries as requiring context or not and our data is derived from work at UW, which only used the examples that should not have required context. Unfortunately, there are some mistakes in the data that mean cases that require context are in our dataset (finding them and removing them or editing them is on the list of known issues in this repository).
Getting from our data back to the original should be possible by using approximate string matching (exact match won't work because we made some fixes to sentences, and because our variable extraction removed some variability, e.g. instead of sometimes having 'american airlines' and sometimes having 'aa', we always have 'aa').
If you are specifically interested in contextual SQL, check our Alane Suhr's work - https://github.com/clic-lab/atis and http://alanesuhr.com/atis.pdf
Many of the utterances in ATIS highly depends on the history of the interaction, and one of the core challenges is figuring how how to reason about this context. For example, the utterance "which ones arrive at 7pm" requires the model to resolve references to previous utterances (What does "ones" refer to?).
Looking at the dataset, it seems like 1) the utterances do not have these kind of references, eg "ones" does not appear in the dataset and 2) there isn't any information on which interaction the utterances come from since they are grouped by SQL query. Were there any modifications to the original utterances and is there any way to reconstitute the structure of the original interactions from the data? Thanks!