databricks / dbt-databricks

A dbt adapter for Databricks.
https://databricks.com
Apache License 2.0
226 stars 119 forks source link

Set unique temp table suffix to allow parallel incremental executions #811

Closed huangxingyi-git closed 1 month ago

huangxingyi-git commented 1 month ago

Resolves #

Description

Set unique table suffix to allow parallel incremental execution For some specific cases (eg. backfill very large amount of data), we need to execute parallel multiple dbt run of specific incremental(replace_where) model in which we pass the date (or country) as var argument. For example, we have a model we run every day using Airflow for which we pass the a date relative to the Airflow scheduler. FYI https://github.com/dbt-labs/dbt-athena/pull/650/files

If we want to process by batch of N days in parallel using Airflow concurrency, we need the tmp table create by each of the dbt run to be unique. Else, you are going to end up with N insert attempting to run with the same __dbt_tmp name, creating conflict and ultimately creating failure.

issue

Checklist

benc-db commented 1 month ago

Apologies for the delay; I've been busy getting ready for 1.9.0 release and Coalesce next week. Code looks straightforward, so I will just verify that it does not break functional tests.