search

simonw / airtable-export

Export Airtable data to YAML, JSON or SQLite files on disk
https://datasette.io/tools/airtable-export
Apache License 2.0
112 stars 15 forks source link

--sqlite DBFILE option for exporting to SQLite #10

Closed simonw closed 3 years ago

simonw commented 3 years ago

Unlike the other options this one will take an argument - the path to the SQLite database file to export to.

simonw commented 3 years ago

This may not work because the first positional argument, output_path is currently required and must be a directory.

simonw commented 3 years ago

Rather than wait until all records have been retrieved, I'm going to have the --sqlite option write batches of 100 records to the database as they arrive. This will allow you to run Datasette against the database and hit refresh to see the records coming in.

simonw commented 3 years ago

Will need to deal with weird column names - just hit an error against this generated SQL:

   [airtable_id] TEXT PRIMARY KEY,
   [ID] INTEGER,
   [Location] TEXT,
   [soft-dropped-column: Vaccines available?] TEXT,
simonw commented 3 years ago

An annoyance of the Airtable API is that it's not possible to distinguish many-to-one fields from many-to-many fields - a many-to-many shows up as values that are JSON lists with a single string in them.

I'm going to ignore that issue for the moment and just tolerate it.

simonw commented 3 years ago

Looks like that SQL error wasn't a problem after all - SQLite is apparently fine with columns called [soft-dropped-column: Vaccines available?]