search

dbt-labs / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
https://dbt-athena.github.io
Apache License 2.0
228 stars 100 forks source link

[Feature] Support for new microbatching incremental strategy #715

Closed jessedobbelaere closed 3 days ago

jessedobbelaere commented 2 months ago

Is this your first time submitting a feature request?

Describe the feature

Previously, for large time-series models we needed a custom insert-by-period macro. In this post dbt announced support for the "microbatching" strategy coming to dbt-core 1.9.

This issue is a placeholder to start preparing for the dbt 1.9 release with this new incremental strategy. Work can be tracked in this epic https://github.com/dbt-labs/dbt-core/issues/10624

Describe alternatives you've considered

No response

Who will this benefit?

Datasets that are too big to run on initial full load without batching (Athena query times out)

Are you interested in contributing this feature?

No response

Anything else?

Closes #190 and probably #697

amychen1776 commented 3 days ago

We are planning to ship this out so I'm going to close out this issue for now