dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
406 stars 228 forks source link

[dbt-spark] Microbatch Incremental Strategy #1109

Closed MichelleArk closed 1 month ago

MichelleArk commented 1 month ago

As part of https://github.com/dbt-labs/dbt-core/issues/10624, dbt-spark needs to implement a microbatch incremental strategy.

In spark, this will use the insert_overwrite strategy and explicitly require a partition_by field.

The base tests from dbt-adapters 1.6 should also be inherited and included as part of this work.