Microbatch Strategy - Githubissues

dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks

https://getdbt.com

Apache License 2.0

404 stars 227 forks source link

Microbatch Strategy #1108

Closed MichelleArk closed 1 month ago

MichelleArk commented 1 month ago

resolves #1109 docs dbt-labs/docs.getdbt.com/#

Problem

As part of https://github.com/dbt-labs/dbt-core/issues/10624, dbt-spark needs to implement a microbatch incremental strategy.

Solution

In spark, this will use the insert_overwrite strategy and explicitly require a partition_by field.
testing against same profiles as insert_overwrite testing
testing against parquet, since it supports timestamp partitioning

Checklist

[x] I have read the contributing guide and understand what's expected of me
[x] I have run this code in development and it appears to resolve the stated issue
[x] This PR includes tests, or tests are not required/relevant for this PR
[x] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

github-actions[bot] commented 1 month ago

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-spark contributing guide.