[Bug] Databricks SQL Warehouse Compatibility

fivetran-joemarkiewicz commented 5 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the issue

If a user is leveraging the a Databricks SQL Warehouse, then they will see the below error. This is due to the incremental strategy insert_overwrite not being compatibile with Databricks SQL Warehouse runtimes.

Relevant error log or model output

spark.sql.sources.partitionOverwriteMode is not available

Expected behavior

The incremental strategy works regardless of Databricks runtime.

dbt Project configurations

Any

Package versions

Latest

What database are you using dbt with?

databricks

dbt Version

Any

Additional Context

We likely will consider the following since we cannot differentiate between databricks runtimes in dbt when using adapter calls.

Leverage the merge strategy with some failsafes in place to avoid the downsides of the merge strategy
Remove the incremental strategy

Are you willing to open a PR to help address this issue?

[x] Yes.
[ ] Yes, but I will need assistance and will schedule time during our office hours for guidance
[ ] No.

fivetran-joemarkiewicz commented 5 months ago

Providing an update here - we have been able to find a way to differentiate between Databricks SQL Warehouse and Databricks All Purpose Clusters in the above PR! I had gone back and forth on either leveraging the merge strategy or just opting to remove the incremental strategy for SQL Warehouse runtimes. As we are not 100% confident in the merge incremental strategy for the fivetran_platform__audit_table model at this time, we are moving forward with making this model not incremental for SQL Warehouse Databricks runtimes.

fivetran-joemarkiewicz commented 4 months ago

As of the latest release v1.7.2, this compatibility issue has been addressed. As such, I will close out this issue.

fivetran / dbt_fivetran_log