This adds some new arguments:
--db-type [athena,duckdb] (defaulting to athena)
--load-ndjson-dir DIR (tells DuckDB where to find source ndjsons)
A light abstraction layer has been added in databases.py to choose the correct backend based on the args.
Mostly the SQL is the same. Some light tweaks for standardization, plus some compatibility user-defined functions injected into duckdb allow both backends to work on the same SQL.
Checklist
[x] Consider if documentation (like in docs/) needs to be updated
[x] Consider if tests should be added
[x] Run pylint if you're making changes beyond adding studies
[x] Update template repo if there are changes to study configuration
This adds some new arguments: --db-type [athena,duckdb] (defaulting to athena) --load-ndjson-dir DIR (tells DuckDB where to find source ndjsons)
A light abstraction layer has been added in databases.py to choose the correct backend based on the args.
Mostly the SQL is the same. Some light tweaks for standardization, plus some compatibility user-defined functions injected into duckdb allow both backends to work on the same SQL.
Checklist
docs/) needs to be updated