search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.76k stars 4.04k forks source link

[Roadmap] Destination Data Lake Databricks to Certified #34874

Open nataliekwong opened 8 months ago

nataliekwong commented 8 months ago

Destination Databricks gains the Destinations V2 treatment . This includes the async framework (faster!) and better Typing and Deduplication, error handling, and more

kendru commented 7 months ago

Will this include supporting Destinations V2 for Databricks?

evantahler commented 7 months ago

Will this include supporting Destinations V2 for Databricks?

Yes it will!

kendru commented 4 months ago

Is this still on the roadmap? The lack of support for transformations in the current Databricks destination is a major impediment for my team. We would be happy to help contribute if the there is any portion that is open to community contributions.

evantahler commented 4 months ago

Yep! It's coming in the next few months. Stay tuned!

kendru commented 4 months ago

Yep! It's coming in the next few months. Stay tuned!

Thanks! I just saw that it was moved to "Future / Not Prioritized", and I wanted to make sure. In addition to OSS contributions, my team is also open to beta testing.

alexbegg commented 4 months ago

I wanted to share that I am in the process of migrating from Redshift to Databricks and I would like to beta test a Destinations V2 version of the Databricks connector when available. I previously beta tested the Redshift Destinations V2 connector.

nataliekwong commented 1 month ago

We've released a modernized Databricks destination (v3.1.0) with several new features, including typing & deduping (Destinations V2), refreshes, and resumable full refresh!

We are now working towards certification in the coming months. If you have any feedback on v3.1.0, feel free to share it here.