nicholasyager / dbt-loom

A dbt-core plugin to weave together multi-project dbt-core deployments
The Unlicense
114 stars 18 forks source link

Feature: Support cross-project Group evaluation #60

Closed nicholasyager closed 4 months ago

nicholasyager commented 5 months ago

Description and Motivation

Within dbt-core, it is possible to assign models and other resources to groups. These groups can be used to restrict other downstream resources from ref-ing the resource if they are not part of the same group. Due to the way that dbt-core instantiates ModelNodes based on ModelNodeArgs, it has not been possible to enforce group reference constraints across project boundaries, and ref-ing protected nodes from across projects has historically yielded error messages lacking the group name of the upstream node (#59).

This pull request unblocks cross-project group evaluation by extending our internal LoomModelNodeArgs construct to include a group string, and wraps the existing ModelNode.from_args() classmethod with an internal model_node_wrapper, which inject the upstream node's group string into the resulting ModelNode. I also added a similar wrapper to ensure that downstream projects have their upstream-projects injected as valid group names during group validity checks. The metadata will NOT be passed into the downstream manifest, but this is a good first step.

Along the way, I found another defect in dbt-core's evaluation of access evaluation for private models, which prevents grouped private models in projects with restricted access from using generic tests (https://github.com/dbt-labs/dbt-core/issues/10134). This means that we cannot enforce checks against protected models as described in #43. I'll restore this functionality once the underlying defect has been resolved in dbt-core.

nicholasyager commented 5 months ago

@z3z1ma, can you please give this branch a test to confirm if it has the intended outcome? Thanks!

z3z1ma commented 4 months ago

Sorry for not getting back to you here. Thanks for the fix 💯 much appreciated sir!