dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.38k stars 1.56k forks source link

[CT-1747] Enable flagging nodes/configs for deferred validation #6511

Open gshank opened 1 year ago

gshank commented 1 year ago

Currently we have an issue with snapshot config, where the validation requires that certain config attributes be set. Since all node configs are validated twice -- once at model parsing time and once at schema parsing time (if a schema config exists) -- if somebody attempts to set the snapshot config in a schema file it will fail on the model parsing step.

In a more recent initiative we have config that might be set in both/either model and schema file and are attempting to validate that all of the config is in place. If the feature is turned on in model config and the other config settings are in the schema file, the complete validation would fail on model parsing.

What we need to handle situations like this is to be able to flag certain validation situations to happen after all of the schema parsing has been done. For example if a snapshot node is constructed from a sql file but doesn't contain some of the required config, set a flag. If a schema file runs that validation and it succeeds, we can then turn the flag off. But if we reach the end of schema file parsing and the flag is still set, re-run validation and issue an error.

jtcohen6 commented 1 year ago

Specific case we're interested in:

We can fall back to doing this not at parse time, since we'll catch it at runtime anyway. It's conceivable that we'd want this capability for other cases in the future, where we want to validate compatibility between configs/attributes that are being set in different places.