search

dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.65k stars 1.6k forks source link

[Feature] Extend partial parsing behavior for env vars to project vars also #10578

Open dbeatty10 opened 1 month ago

dbeatty10 commented 1 month ago

Is this your first time submitting a feature request?

Describe the feature

Avoid full parses and use partial parsing when project vars have not changed.

The implementation would more granularly track where those inputs are being used — that is, extending what we've done for env vars to project vars. I believe the relevant partial parsing code related to environment variables is here.

Describe alternatives you've considered

No response

Who will this benefit?

This will allow dbt to complete the parsing phase faster by using partial parsing to only parse the changed files (or files related to those changes).

Are you interested in contributing this feature?

No response

Anything else?

From https://github.com/dbt-labs/docs.getdbt.com/issues/1066#issuecomment-2015122367 and https://github.com/dbt-labs/dbt-core/pull/9844#issuecomment-2038133642

donjin-master commented 3 weeks ago

Hey @dbeatty10, From past few days i am trying to implement the tracking for --vars. Please correct me if i am thinking in wrong direction.

  1. We want to track all the --vars which we passed in the cli?
  2. If we were considering cli, then to track the changes we would be saving the cli_vars in manifest and we can track all the changes like we did for env_var.
  3. This point is a question, what about the ordering of the cli_vars. Do we require ordering of the cli_vars, it could create a different filehash from prev one if order changed?

Do add up any points which i should be taken care of in implementing this one.

dbeatty10 commented 2 weeks ago

@donjin-master this one is complicated enough that we'll probably wait until we can implement this ourselves rather than accept a community contribution.

Were you hoping to create a solution for this one specifically because it is relevant to one of your workflows? Or are you just generally looking to contribute to something to dbt-core? If the latter, here's where you can find issues that we've labeled for community contributors: