lightdash / lightdash

Self-serve BI to 10x your data team ⚡️
https://lightdash.com
MIT License
3.94k stars 419 forks source link

Native cloud storage export for supported warehouses #4851

Open IrakliJani opened 1 year ago

IrakliJani commented 1 year ago

Description

Add the ability to offload query results to cloud storage using supported warehouses' native export feature. This would allow users to export results to cloud storage like GCS (preferred for BigQuery) or S3.

The feature would be accessible through the Lightdash UI, making it easy for users to export their data without relying on Lightdash infrastructure.

djayatillake commented 1 year ago

It might be a good idea to have a configurable bucket to put these results in - so results generate subfolders or something similar. IT teams can then make the whole bucket accessible to anyone on their network etc, or if the convention is bucket/username then they can even choose to say people only have access to their own results.

Maybe for V2 but it could perhaps be good for Lightdash to administrate this kind of access to a bucket if given one to use by the IT team.

jw-aleksandr-zolotukhin commented 1 year ago

Just an idea: Would be great to use that option (when enabled) to steer "Export CSV" feature from the tables. image

This would eliminate row limit and executed way faster by DWH instead of Lightdash engine (I'm assuming it's generating file row by row right now).

ZeRego commented 1 year ago

I guess the downside of this approach is losing all the value formatting. eg: currency, rounding

djayatillake commented 1 year ago

I don't think that really matters, for the use case.

stale[bot] commented 1 year ago

Is this issue still relevant? There have been no updates for 60 days, please close the issue or keep the conversation going!

IrakliJani commented 1 year ago

+1 customer, still relevant