GindaChen
commented
3 years ago Looking back at the original text:
To map English sentences to query sketches, we have implemented our own semantic parser on top of the Sempre ... For the linguistic processor, we leverage the pre-trained models of the Stanford CoreNLP [Manning et al. 2014] library for part-of-speech tagging and named entity recognition.
It really sounds like SQLizer wasn't directly using Sempre to do semantic parsing, but that isn't really possible because of how sempre is structured. Either they inject the grammar into sempre, or they are just simply using the stanford corenlp without any of the logical features. Also,
Given an utterance u, our parser generates all possible query sketches S i and assigns each S i a score that indicates the likelihood that S i is the intended interpretation of u. This score is calculated based on a set of pre-defined features. More precisely, given an utterance u and weight vector w, the parser maps each query sketch S i to a d-dimensional feature vector ϕ(u, S i ) ∈ R d and computes the likelihood score for S i as the weighted sum of its features Sqlizer uses approximately 40 features that it inherits from the Sempre framework. Examples of features include the number of grammar rules used in the derivation, the length of the matched input, whether a particular rule was used in the derivation, the number of skipped words in a part-of-speech tag etc.
Sempre never associate the score with features other than (1) the number of occurance / some weighted frequency in the training set and (2) the calculated likelihood. I'm now wondering what these 40 features should represent here that they had described.
The sempre does not seem to provide a pre-trained mode (I will ask them soon) to utilize their FreeBase copy directly. Instead, we will need to train on the dataset ourselves.
There is an active training process on the cloudlab machine, ultimately training with 6 iterations, each with 512 examples using their given dataset (called free917).
There are a few problems:
Training is very slow. They seem to provide a single-threaded trainer to train the model (I could be wrong as there are just too many parameters to set). It's running 1 hour with 50 examples only... We will see if there is a way to accelerate the training process, otherwise, we will just have to wait for another 50+ hours for the model to come hot.
Parsing will be hard. For example, suppose we have an utterance
Our expected target result (i.e. the answer in the training set) is the following:
But Sempre will send us back some logical derivations as follows:
These entities does not guarantee the original text will be presented in the derivation, nor will it guarantee they are indeed feasible.