search

taoyds / sparc

scripts and baselines for SParC: Yale & Salesforce Semantic Parsing and Text-to-SQL in Context Challenge
https://yale-lily.github.io/sparc
73 stars 12 forks source link

When an adequate template exists for a given SQL input, does it mean the English output will have a perfect, 100% LCR? #2

Open nico3865 opened 4 years ago

nico3865 commented 4 years ago

Hi,

First, thanks for releasing this SQL-to-text template baseline! (over here: https://github.com/brucepang/template-based-sql2response-model).

I'm reading the CoSQL paper (https://arxiv.org/pdf/1909.05378.pdf) and I read that

"this [SQL-to-text template baseline] approach is limited by the small coverage of the training templates and its LCR is only around 40%."

My question is, how is this SQL-to-text template baseline performing on the subset of SQL queries for which it does have adequate templates ?

--> i.e. when it has good templates for a given SQL query, what is the LCR?

I'm trying to estimate whether the low LCR is really due to a low coverage of templates only, or if there are other causes, too.

--> Even just a ballpark number is going to help a lot to decide if building more templates is worth my time!

Another related question is: did you spend very little time building templates, or on the contrary, you spent an enormous amount of time building templates and you see no end to this? ... Is building templates worth my time, you think, to improve the LCR for SQL-to-text?

Thank you so much for clarifying this issue!

nico3865 commented 4 years ago

I'm assuming that when an adequate template is found for a given SQL query, then the LCR score for that question is 100%.

Is that correct?