Sqlite3 could not execute *-db.sql

[songhe17]

I wanted to run atis-db.sql in python with sqlite3 to create the db file but failed due to syntax errors.

[jkkummerfeld]

All of the databases were constructed in MySQL and exported to text files. I don't know the differences in syntax between MySQL and sqlite3, so I'm not sure how to make it work there. We'd be happy to accept a pull request with sqlite3 versions of the databases!

Here is the information about how we exported the databases (from the atis sql file):

-- MySQL dump 10.13  Distrib 5.7.17, for Linux (x86_64)
--
-- Host: localhost    Database: atis
-- ------------------------------------------------------
-- Server version       5.7.17

[rizar]

There is a script for converting MySQL dumps to SQLite dumps, and it kind of works:

https://gist.githubusercontent.com/esperlu/943776/raw/be469f0a0ab8962350f3c5ebe8459218b915f817/mysql2sqlite.sh

[jkkummerfeld]

Using that script I've added sqlite versions of the data.

[rizar]

Thanks! (and also thanks for the great initiative to collect all lang2sql datasets in one place, by the way!). But for the record I should note there are also big databases (Yelp, Scholar, etc.) that you do not include in your repository, which need to be manually converted. The script that I mentioned above gives reasonable results.

[jkkummerfeld]

Good to know. There is also one (restaurants) that is not in any SQL format.

I should also note, a few new datasets have been released in this space since we put this together:

• SParC
• CoSQL

And some additional work on evaluation:

• https://github.com/google-research/language/tree/master/language/xsp
• https://arxiv.org/abs/2010.02840

[rizar]

Thank you, Jonathan. I was aware of these links, except for the last one (which is very interesting, by the way).

In fact what I'm trying to do now is evaluating a SPIDER-trained RAT-SQL-style model using the methodology proposed by Suhr et al (which is also the penultimate link in your list). A key challenge is figuring out the proper primary and foreign key relations to feed into the model. We will do an open-source release all is done.

[jkkummerfeld]

Cool - looking forward to it!