Set unique table suffix to allow parallel incremental execution
For some specific cases (eg. backfill very large amount of data), we need to execute parallel multiple dbt run of specific incremental(replace_where) model in which we pass the date (or country) as var argument.
For example, we have a model we run every day using Airflow for which we pass the a date relative to the Airflow scheduler.
FYI
https://github.com/dbt-labs/dbt-athena/pull/650/files
If we want to process by batch of N days in parallel using Airflow concurrency, we need the tmp table create by each of the dbt run to be unique. Else, you are going to end up with N insert attempting to run with the same __dbt_tmp name, creating conflict and ultimately creating failure.
Apologies for the delay; I've been busy getting ready for 1.9.0 release and Coalesce next week. Code looks straightforward, so I will just verify that it does not break functional tests.
Resolves #
Description
Set unique table suffix to allow parallel incremental execution For some specific cases (eg. backfill very large amount of data), we need to execute parallel multiple dbt run of specific incremental(replace_where) model in which we pass the date (or country) as var argument. For example, we have a model we run every day using Airflow for which we pass the a date relative to the Airflow scheduler. FYI https://github.com/dbt-labs/dbt-athena/pull/650/files
If we want to process by batch of N days in parallel using Airflow concurrency, we need the tmp table create by each of the dbt run to be unique. Else, you are going to end up with N insert attempting to run with the same __dbt_tmp name, creating conflict and ultimately creating failure.
issue
Checklist
CHANGELOG.md
and added information about my change to the "dbt-databricks next" section.