z3z1ma / dbt-osmosis

Provides automated YAML management, a dbt server, streamlit workbench, and git-integrated dbt model output diff tools
https://z3z1ma.github.io/dbt-osmosis/
Apache License 2.0
422 stars 45 forks source link

Feature request - Support offline ingestion of `manifest.json` and `catalog.json` #57

Closed b-per closed 1 year ago

b-per commented 1 year ago

I think that the ability to use existing manifest.json and catalog.json would be a great way to run the yaml refactor "offline", without requiring access to the Warehouse.

In the case of large projects, I am actually wondering if this wouldn't speed up the yaml refactoring process significantly (instead of querying the columns of each model one by one)

I see that it could be especially useful in the case someone would want to have dbt-osmosis as a CI/CD step without wanting to store their Warehouse credentials in GitHub.

In the case of dbt Cloud, for example, the artifacts could be downloaded from the API, and then:

I could potentially give this feature a go if you think it would be valuable

z3z1ma commented 1 year ago

I think this is a fantastic idea. I am out for a week or so, so feel free to take a shot! Otherwise we will keep the issue open and I will circle back to it.