I think that the ability to use existing manifest.json and catalog.json would be a great way to run the yaml refactor "offline", without requiring access to the Warehouse.
In the case of large projects, I am actually wondering if this wouldn't speed up the yaml refactoring process significantly (instead of querying the columns of each model one by one)
I see that it could be especially useful in the case someone would want to have dbt-osmosis as a CI/CD step without wanting to store their Warehouse credentials in GitHub.
In the case of dbt Cloud, for example, the artifacts could be downloaded from the API, and then:
catalog.json would provide the list of all columns and their description for all models/sources
manifest.json would provide details of the models used in {node}
I could potentially give this feature a go if you think it would be valuable
I think this is a fantastic idea. I am out for a week or so, so feel free to take a shot! Otherwise we will keep the issue open and I will circle back to it.
I think that the ability to use existing
manifest.json
andcatalog.json
would be a great way to run the yaml refactor "offline", without requiring access to the Warehouse.In the case of large projects, I am actually wondering if this wouldn't speed up the yaml refactoring process significantly (instead of querying the columns of each model one by one)
I see that it could be especially useful in the case someone would want to have
dbt-osmosis
as a CI/CD step without wanting to store their Warehouse credentials in GitHub.In the case of dbt Cloud, for example, the artifacts could be downloaded from the API, and then:
catalog.json
would provide the list of all columns and their description for all models/sourcesmanifest.json
would provide details of the models used in{node}
I could potentially give this feature a go if you think it would be valuable