New destination: Azure data lake storage Gen 2

wavermartijn commented 3 years ago

Tell us about the new destination you’d like to have

I would like to have Azure data lake storage Gen 2 as a destination

Describe the context around this new destination

This destination will help setting up data pipelines in the Azure cloud.
Multiple sources will probably end up in this destination. The amount of data will potentially be high.

Describe the alternative you are considering or using

The alternative would store files on some local disk first and a different process will then pick it up and push it to the Azure Data lake storage. Or we could look for/develop a alternative service that does the ingestion of the data.

┆Issue is synchronized with this Asana task by Unito

eduardgruy commented 3 years ago

+1. Would be great to have ADLS destination that supports multiple formats. Ideally Deltalake/CSV/Parquet. In our scenario ADLS would be used as a kind of landing zone so appending changes to a delta lake table would be perfect.

hammadhasandogar commented 3 years ago

+1 required by many of our clients for collection of data from sources and files atleast to the landing zone.

airbytehq / airbyte