dbt-labs / dbt-postgres

Apache License 2.0
34 stars 14 forks source link

dbt-postgres 'microbatch' strategy #146

Closed MichelleArk closed 2 months ago

MichelleArk commented 2 months ago

resolves https://github.com/dbt-labs/dbt-postgres/issues/149 docs dbt-labs/docs.getdbt.com/#

Problem

As part of https://github.com/dbt-labs/dbt-core/issues/10624, dbt-postgres needs to implement a default microbatch strategy.

Solution

Although it is not write-optimized for partitioned data, we will use merge under-the-hood in dbt-postgres for now, as dbt-postgres does not yet support partitioning.

Checklist

github-actions[bot] commented 2 months ago

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-postgres contributing guide.

MichelleArk commented 2 months ago

Confirmed the failing tests are expected on tests/functional/adapter/test_incremental_microbatch.py::TestPostgresMicrobatch::test_run_with_event_time. Going to go ahead and merge on failing CI here as this will just be intermittent until https://github.com/dbt-labs/dbt-core/issues/10624 is merged.