search

salesforce / WikiSQL

A large annotated semantic parsing corpus for developing natural language interfaces.
BSD 3-Clause "New" or "Revised" License
1.62k stars 322 forks source link

Leaderboard for weakly supervised WikiSQL task #33

Closed agarwl closed 5 years ago

agarwl commented 5 years ago

Since the results on the supervised learning benchmark are quite close to being saturated, I think having a leaderboard for models trained using only weak-supervision would be more relevant benchmark (for example, Memory Augmented Program Synthesis from Liang et. al beats some of the old entries in the strongly-supervised leaderboard using only weak supervision.)

vzhong commented 5 years ago

Thanks for your feedback! Currently this distinction is made in the paper. We can also make this distinction in the leaderboard, however this doesn't seem like a common use case for WikiSQL (I think?). We can reconsider this issue in the future.