How to export to parquet directly? Seems like command line is needed at the moment

[jaanli]

Hi! This is amazing work and at @onefact we're using it across the board.

Right now we couldn't find a way to export .duckdb outputs to .parquet format.

Here is an example:

• https://github.com/onefact/data_build_tool_for_datathinking.org/blob/main/datathinking/models/cityofnewyork.us/service_requests.sql results in /tmp/service_requests.duckdb

• this duckdb database must then be converted to parquet format using duckdb /tmp/service_requests.duckdb and then:

COPY service_requests TO '/tmp/service_requests.parquet' (FORMAT 'PARQUET', CODEC 'ZSTD');

Is exporting to parquet supported, or is there a place to add this code? Not sure if this is the appropriate place to raise this issue -- still learning about this dope ecosystem!

(Next up, uploading to aws s3!)

[jwills]

We support writing to parquet (or csv, or json) files directly via the external materialization: https://github.com/jwills/dbt-duckdb/tree/master#writing-to-external-files

[jwills]

If you want to control the COPY operations yourself, dbt supports that via hooks: https://docs.getdbt.com/docs/build/hooks-operations

[jaanli]

That was it!! Thank you so much @jwills -- this is a huge help for a large class at Princeton 🙏 and several nonprofits we are working with @onefact. Really grateful for your support!!!