dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.92k stars 1.63k forks source link

[Epic] state:modified should Actually (only) select the modified resources #9562

Open graciegoheen opened 9 months ago

graciegoheen commented 9 months ago

We currently call out a lot of caveats to state comparison in our docs.

There are many scenarios where executing dbt list --select state:modified over or under selects the appropriate resources. This leads to:

While it's unlikely we can cover ever single edge case here, we should improve state:modified so that it Ideally actually only selects the modified resources.

This epic will track the relevant bugs/enhancements related to state:modified.

### Issues
- [ ] https://github.com/dbt-labs/dbt-core/issues/10002
- [ ] https://github.com/dbt-labs/dbt-core/issues/9548
- [ ] https://github.com/dbt-labs/dbt-core/issues/9563
- [ ] https://github.com/dbt-labs/dbt-core/issues/9564
- [ ] https://github.com/dbt-labs/dbt-core/issues/4304
- [ ] https://github.com/dbt-labs/dbt-core/issues/10138
- [ ] https://github.com/dbt-labs/dbt-core/issues/9573
- [ ] https://github.com/dbt-labs/dbt-core/issues/8727
matt-winkler commented 8 months ago

What would be lost by disallowing env-aware logic outside the dbt_project.yml?

patkearns10 commented 8 months ago

Lots of examples here: https://discourse.getdbt.com/t/why-are-ci-jobs-state-method-building-more-less-models-nodes-than-expected/7504 An example of env_var logic here , which seems impossible to keep out of state:modified

graciegoheen commented 3 months ago

For those following along, would love to hear your thoughts over in this discussion https://github.com/dbt-labs/dbt-core/discussions/10518