New features for Databricks connector

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

https://airbyte.com

Other

16.11k stars 4.12k forks source link

New features for Databricks connector #7232

Open tuliren opened 3 years ago

tuliren commented 3 years ago

Tell us about the problem you're trying to solve

Provide separate bucket paths for staging vs production table
Support partitioning by specific columns
- This includes partitioning during table creation and new data ingestion

klogdog commented 2 years ago

Fivetran doesn't require access to s3 for staging. They only require the Databricks portion of information , which are server address, http path, and the personal access token.

We should be able to make the Databricks connector as robust and easy. Removing the need for s3 access and relying on Databricks for staging would simplify the process for airbyte users and relieve them of the headache of permission errors in AWS.

Steiniche commented 1 year ago

@tuliren I believe this is fixed by using https://docs.airbyte.com/integrations/destinations/databricks/ and writing directly to hive or Unity Catalog.