nicholasyager / dbt-loom

A dbt-core plugin to weave together multi-project dbt-core deployments
The Unlicense
118 stars 19 forks source link

Child project no longer compiles successfully (possibly when seeds are involved upstream) #46

Closed lepagea01 closed 6 months ago

lepagea01 commented 6 months ago

Describe the bug Upstream node injection in version 0.5.0 may have introduced a bug. Models in a child project having dependencies to public models in a parent model no longer compile. Things were running smoothly in previous versions of dtb-loom.

A typical error message resulting from compiling the child project looks as follows:

14:26:03  Running with dbt=1.7.13
14:26:04  dbt-loom: Patching ref protection methods to support dbt-loom dependencies.
14:26:04  dbt-loom: Loading manifest for `parent` from `file`
14:26:04  Registered adapter: snowflake=1.7.3
14:26:04  dbt-loom: Injecting nodes
14:26:05  Encountered an error:
Compilation Error
  'model.parent.dim_commission_type' depends on 'seed.parent.seed_commission_type' which is not in the graph!

The error message above mentions the public dim_commission_type model in the parent project that stems from the protected seed_commission_type seed in the parent project as well. Some models of the child project reference the dim_commission_type model in the parent project, but none references the seed_commission_type seed. Since we have other projects for which cross-project dependencies work fine with dbt-loom 0.5.0, the problem may be limited to scenarios where protected seeds are used upstream.

To Reproduce The overall setup is as follows:

  1. A parent project, say, parent, contains various models, with both protected and public models.
  2. The parent project compiles/builds successfully.
  3. The resulting manifest.json file is properly referenced in the dbt_loom.config.yml file at the root of a child project, say, child.
  4. Some models in the child project reference public models in the parent project, in turn based on protected seeds in the parent project.
  5. Commands such as dbt run, dbt build or dbt compile will all produce error messages similar to the one above when executed in the child project.

Expected behavior To have the child project compiling successfully.

Setup

Please, let me know if additional information could be useful.

Thanks,

nicholasyager commented 6 months ago

@lepagea01 This bug report absolutely rocks. I think this defect may be related to some new features in 0.5.0, so there should be an easy fix here.

nicholasyager commented 6 months ago

I was able to find the underlying defect here, and a fix has been implemented. I've also updated our test cases so we don't miss seeds in the future. This fix will be available in dbt-loom 0.5.1, which will be available within the hour.

lepagea01 commented 6 months ago

@lepagea01 This bug report absolutely rocks. I think this defect may be related to some new features in 0.5.0, so there should be an easy fix here.

Truth be told, dbt-loom rocks! Cross-project dependencies was already a game changer for those of us working on large projects, dbt-loom is a life-saver for those of us also wanting to develop outside of dbt Cloud.

Thanks for time you all invest in this project. And thanks for this quick turnaround. Eager to test 0.5.1.