dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.28k stars 1.54k forks source link

[CT-2260] [Feature] Explainable state modified #7109

Open z3z1ma opened 1 year ago

z3z1ma commented 1 year ago

Is this your first time submitting a feature request?

Describe the feature

Problem: I want to be able to explain a state:modified selection with fine grained diffs. I can code up a lot of ways to solve this but let me riff on what I think would be ideal.

Solution: There are methods returning primitive types that should ideally be wrapper types that duck-type the effect of the primitive. Specifically, the methods I am calling out are planted all over the sort-of core dataclasses, and they return bools when, in an ideal world, they should return a specific class which implements __bool__ but is, by effect of not being a binary primitive, significantly more extensible. We can do this whilst maintaining the contract with dependents of the method.

All of the same_* methods from same_body to same_database_representation These should pass around a Comparator which has a baked in method called output diff which leverages difflib to generate a log-able diff of the node. This means Comparator's will carry references to old and new, so that it is available as needed. And lastly, the diff is opt-in based on where we decide to plant the method calls. I can see the -vv verbosity level being the impetus to output the diff.

Describe alternatives you've considered

So long as the same_* methods are restricted to bools, there is no alternative.

Who will this benefit?

Anyone troubleshooting why a model is running when they don't think or know if it was modified by providing a highly detailed unified_diff.

Are you interested in contributing this feature?

Yes

Anything else?

Slack Thread: https://getdbt.slack.com/archives/C50NEBJGG/p1675216311733109

jtcohen6 commented 1 year ago

Thanks for opening the issue @z3z1ma! I know folks will be excited about this :)

(Internal note: Even though this has to do with node selection, which is generally Team:Execution, the changes will largely be in updates to our node classes, so Team:Language feels like the better fit.)

z3z1ma commented 1 year ago

In progress PR: #7102