dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.68k stars 1.61k forks source link

Write microbatch compiled + run code to separate target files #10743

Closed MichelleArk closed 5 days ago

MichelleArk commented 1 week ago

Resolves https://github.com/dbt-labs/dbt-core/issues/10714

Problem

During execution of microbatch models, each batch is re-compiled with batch-level time filters. Currently, this results in the compiled/run .sql files becoming clobbered on each write when there should be one file per compiled/run batch.

Solution

Pass an additional argument, split_suffix (naming suggestions welcome!) that splits the write path by the suffix and writes to a top-level directory without the suffix.

Checklist

github-actions[bot] commented 1 week ago

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 95.45455% with 1 line in your changes missing coverage. Please review.

Project coverage is 88.98%. Comparing base (db69473) to head (6da854a). Report is 5 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #10743 +/- ## ========================================== + Coverage 88.97% 88.98% +0.01% ========================================== Files 181 181 Lines 22956 22995 +39 ========================================== + Hits 20424 20461 +37 - Misses 2532 2534 +2 ``` | [Flag](https://app.codecov.io/gh/dbt-labs/dbt-core/pull/10743/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dbt-labs) | Coverage Δ | | |---|---|---| | [integration](https://app.codecov.io/gh/dbt-labs/dbt-core/pull/10743/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dbt-labs) | `86.16% <95.45%> (+0.01%)` | :arrow_up: | | [unit](https://app.codecov.io/gh/dbt-labs/dbt-core/pull/10743/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dbt-labs) | `62.41% <63.63%> (-0.03%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dbt-labs#carryforward-flags-in-the-pull-request-comment) to find out more. | [Components](https://app.codecov.io/gh/dbt-labs/dbt-core/pull/10743/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dbt-labs) | Coverage Δ | | |---|---|---| | [Unit Tests](https://app.codecov.io/gh/dbt-labs/dbt-core/pull/10743/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dbt-labs) | `62.41% <63.63%> (-0.03%)` | :arrow_down: | | [Integration Tests](https://app.codecov.io/gh/dbt-labs/dbt-core/pull/10743/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dbt-labs) | `86.16% <95.45%> (+0.01%)` | :arrow_up: |