datnguye / dbterd

Generate the ERD as a code from dbt artifacts
https://dbterd.datnguyen.de/
MIT License
208 stars 30 forks source link

[FEAT] option to get artifacts from dbt Cloud #63

Closed ernestoongaro closed 9 months ago

ernestoongaro commented 10 months ago

For users of dbt Cloud, generating a diagram could be much easier if they didn’t have to download manifest/catalog files but instead query dbt Cloud for them

Describe the solution you'd like Providing an API token and an environment ID, do not require the use of dbt core to generate manifest files, instead use graphQL to get required information from metadata

https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-querying

Describe alternatives you've considered Could use the regular dbt Cloud rest API to get the latest artefact files vs discovery API but requires passing in of job IDs which is lest convenient

datnguye commented 10 months ago

Thanks @ernestoongaro for the feat suggestion. It is definitely a great one 👍

In the meantime, it will be more than welcome if you would raise a PR for it, please feel free to do so and let me know. If not, I will address it in the new year. Cheers

ernestoongaro commented 10 months ago

Thanks @datnguye. At this time it’s just an idea vs something I have the ability to implement

datnguye commented 10 months ago

Hi @ernestoongaro I just had a quick look into dbt Cloud APIs. It will be much easier to implement dbt Cloud artifacts downloading from dbt Cloud rest API (1) (passing job id, which is the less-convinient option) than the recommended option using GraphQL (2):

Feel free to let me know what you think here.

Thank you 🙌

ernestoongaro commented 9 months ago

Hi @datnguye that makes a lot of sense and perhaps I was asking for too much, or something not even fully possible. The idea behind the new Discovery API endpoints is like a "parsed" version of the artefacts, but since you are working with the files already, then the "classic" way looks perfect!

ernestoongaro commented 9 months ago

I've modified the title of the FR to reflect that I'm totally happy that it's done with the REST API - so I think we can mark this closed? You are something else @datnguye, really cool!

datnguye commented 9 months ago

Thanks a lot @ernestoongaro

I will leave this open until this weekend to fulfill the unit testing of the new feature, and probably released v1.9 there 🎉

For the GraphQL option, I think it is still valid to make it, I really like this option and would like to module it within dbterd, so if you don’t mind I will raise another FEAT issue, iterate changes there, and close this one as a result.

datnguye commented 9 months ago

It is now available in v1.9 and the related docs 🎉

datnguye commented 9 months ago

FYI @ernestoongaro dbterd v1.10.0b1 is now understanding the Discovery API

https://dbterd.datnguyen.de/1.10-beta/nav/guide/dbt-cloud/read-artifact-from-an-environment.html