Native cloud storage export for supported warehouses

[IrakliJani]

Description

Add the ability to offload query results to cloud storage using supported warehouses' native export feature. This would allow users to export results to cloud storage like GCS (preferred for BigQuery) or S3.

The feature would be accessible through the Lightdash UI, making it easy for users to export their data without relying on Lightdash infrastructure.

[djayatillake]

It might be a good idea to have a configurable bucket to put these results in - so results generate subfolders or something similar. IT teams can then make the whole bucket accessible to anyone on their network etc, or if the convention is bucket/username then they can even choose to say people only have access to their own results.

Maybe for V2 but it could perhaps be good for Lightdash to administrate this kind of access to a bucket if given one to use by the IT team.

[jw-aleksandr-zolotukhin]

Just an idea: Would be great to use that option (when enabled) to steer "Export CSV" feature from the tables.

This would eliminate row limit and executed way faster by DWH instead of Lightdash engine (I'm assuming it's generating file row by row right now).

[ZeRego]

I guess the downside of this approach is losing all the value formatting. eg: currency, rounding

[djayatillake]

I don't think that really matters, for the use case.

[stale[bot]]

Is this issue still relevant? There have been no updates for 60 days, please close the issue or keep the conversation going!

[IrakliJani]

+1 customer, still relevant