dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
288 stars 116 forks source link

Add Databricks plugin for Unity Catalog + SQL Warehouse compute #236

Open FlorianSchroevers opened 8 months ago

FlorianSchroevers commented 8 months ago

Description & motivation

resolves: #269

This change adds a Databricks plugin that can be used for Databricks using SQL Warehouse compute with Unity Catalog enabled, which is not possible using the spark plugin. I have used this fork of the repository successfully within my organization for several months now. It also moves the alter table ... recover partitions definition from the spark plugin to common, since it's named 'default'.

Differences between databricks plugin and spark plugin:

I'd be happy to make changes to my fork if necessary.

Checklist

CodeGeek1212 commented 3 months ago

Love to use this one! Any chance this one is going to main?

dataders commented 2 months ago

@FlorianSchroevers @CodeGeek1212 @grindheim have y'all been using these macros already by chance by adding to your dbt project's macros/ directory?

I haven't yet had the chance to test them or this PR, but would like to understand how "ready for primetime" they are. Because setting up the CI infrastructure is a non-trivial amount of work in and of itself