multiprocessio / datastation

App to easily query, script, and visualize data from every database, file, and API.
https://datastation.multiprocess.io
Other
2.9k stars 112 forks source link

All result writes go through writer interface #267

Closed eatonphil closed 2 years ago

eatonphil commented 2 years ago

Before this files would write with a JSON encoder directly. This PR abstracts the write process so that the encoding can be altered without all input format readers knowing about the encoding.

This allows dsq to implement the write interface with a SQLite writer that (in many cases) bypasses conversion to JSON. This ends up speeding up dsq ~2x in those cases. See https://github.com/multiprocessio/dsq/pull/66 for the SQLiteWriter details.