Make node resolution environment aware

[zendesk-sova]

Is your feature request related to a problem? Please describe. Hi folks, Thanks for the great work, I wonder is it possible to make node resolution process to be environment (target) aware?

So in our use case lower environments (targets) like dev, ci are using one set of Snowflake tables for defer and we can put it in the manifest, but for higher environments (target) like prod the same model should be resolved to a different fully qualified name. If I rephrase the question can we hack generate_schema_name and generate_database_name macros to node resolution?

Describe alternatives you've considered Not quite sure if there is a solution other than maintain N different manifests one for each environment, which doesn't look like an optimal and potentially error prone method.

Additional context I'm ready to contribute to the solution with enough guidance provided.

[nicholasyager]

Hi @zendesk-sova 👋🏻 Thanks for submitting this feature request. I think I understand the request, but I want to make sure that I have a complete mental model before I start digging in too deep. Can you please describe in a little more details precisely what the project and environment configuration looks like for your set-up? It sounds like there are two separate environments with separate snowflake objects attempting to share a manifest, and I'm not quite sure what this looks like in practice.

Thanks!

[zendesk-sova]

Hi @nicholasyager , Yes, we have multi-project multi-environment setup with complex database resolution logic. If I understand correctly plugin just takes fully qualified name from the manifest and uses it in models, and I'm asking if it's possible to dynamically resolve what database/schema would've been if we applied custom generate macros at the moment of execution.

The other moment with fixed prebuilt manifests, how to work with multiple targets locally as $DBT_TARGET environment variable is not synchronized with --target <target_name> and such kind of operations become extremely error prone.

[sergey-vdovin]

Hello everybody! Thanks for the contributions to [dbt-loom].

Is looks like a scenario for an alternative resolution is what [dbt-mesh] is doing, I am thinking about how that could be implemented in [dbt-loom] as well:

https://docs.getdbt.com/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref

Safeguarding production data with staging environments When working in a Development environment, cross-project refs normally resolve to the Production environment of the project. However, to protect production data, set up a Staging deployment environment within your projects. With a staging environment integrated into the project, any references from external projects during development workflows resolve to the Staging environment. This adds a layer of security between your Deployment and Production environments by limiting access to production data.

[nicholasyager]

Thanks for the link, @sergey-vdovin! Based on the docs provided, it sounds like dbt Mesh is tracking state for both staging and production environments. I have a hunch that their proprietary systems are simply tracking manifests for all environments. This hunch is based on the requirement that one has a completed staging environment run prior to attempting to use cross-project refs.

In our case, dbt-loom is very simplistic. It has no concept of an environment. Rather, it assumes that the manifests being piped into it are exactly what we want. For this feature to be realistically implemented, we'd also need to have a concept of environment external to the concept of project. This is doable, but require more forethought and design around how we want to configure and implement. Maybe it can be rolled in with #61?

In any case, I'm open to suggestions!