search

nicholasyager / dbt-loom

A dbt-core plugin to weave together multi-project dbt-core deployments
https://nicholasyager.github.io/dbt-loom/
The Unlicense
126 stars 19 forks source link

Check for missing projects by parsing the `dependencies.yml` file. #12

Open nicholasyager opened 1 year ago

nicholasyager commented 1 year ago

As part of dbt Lab's new multi-project deployment functionality in dbt Cloud, they are using a new config file, called dependencies.yml to track project dependencies. We should be leveraging this file to check for missing projects during injection.

geoHeil commented 9 months ago

you mean instead of the extra loom config? Or in addition?

radium226 commented 2 months ago

Hello!

First of all, thanks a lot for this amazing project which is very (very!) helpful.

For now, I dynamically generate a config file by parsing the dependencies.yml and inject it using the DBT_LOOM_CONFIG env var.

But it's true that it could be better to use directly the dependencies.yml and either an entry in the config file to setup the S3 bucket and the pattern how to access the file.

I was thinking of something like:

dependencies:
  type: s3
  config:
    bucket_name: ${DBT_STATES_S3_BUCKET}
    object_pattern states/%{name}/manifest.json

Is that the kind of thing you had in mind?