[FEAT] option to get artifacts from dbt Cloud

[ernestoongaro]

For users of dbt Cloud, generating a diagram could be much easier if they didn’t have to download manifest/catalog files but instead query dbt Cloud for them

Describe the solution you'd like Providing an API token and an environment ID, do not require the use of dbt core to generate manifest files, instead use graphQL to get required information from metadata

https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-querying

Describe alternatives you've considered Could use the regular dbt Cloud rest API to get the latest artefact files vs discovery API but requires passing in of job IDs which is lest convenient

[datnguye]

Thanks @ernestoongaro for the feat suggestion. It is definitely a great one 👍

In the meantime, it will be more than welcome if you would raise a PR for it, please feel free to do so and let me know. If not, I will address it in the new year. Cheers

[ernestoongaro]

Thanks @datnguye. At this time it’s just an idea vs something I have the ability to implement

[datnguye]

Hi @ernestoongaro I just had a quick look into dbt Cloud APIs. It will be much easier to implement dbt Cloud artifacts downloading from dbt Cloud rest API (1) (passing job id, which is the less-convinient option) than the recommended option using GraphQL (2):

• With (1): it follows the way dbterd was implemented - download/generate artifact files in local and read info from there to generate ERD text. Therefore it will be quick to implement, so I will go ahead and make it avalable first, something like dbterd run --dbt-cloud with some env vars of dbt Cloud Host URL, Token and Job ID.

• With (2): This option, I might be misunderstanding here, will require to introduce the new core functions in dbterd, something to define the GraphQL query and do mapping with the output erd's attributes (we are supposed not to download files anymore). It would take sometime to make it done, so currently, I will mark this as help_wanted and will address it gradually after (1) finished.

Feel free to let me know what you think here.

Thank you 🙌

[ernestoongaro]

Hi @datnguye that makes a lot of sense and perhaps I was asking for too much, or something not even fully possible. The idea behind the new Discovery API endpoints is like a "parsed" version of the artefacts, but since you are working with the files already, then the "classic" way looks perfect!

[ernestoongaro]

I've modified the title of the FR to reflect that I'm totally happy that it's done with the REST API - so I think we can mark this closed? You are something else @datnguye, really cool!

[datnguye]

Thanks a lot @ernestoongaro

I will leave this open until this weekend to fulfill the unit testing of the new feature, and probably released v1.9 there 🎉

For the GraphQL option, I think it is still valid to make it, I really like this option and would like to module it within dbterd, so if you don’t mind I will raise another FEAT issue, iterate changes there, and close this one as a result.

[datnguye]

It is now available in v1.9 and the related docs 🎉

[datnguye]

FYI @ernestoongaro dbterd v1.10.0b1 is now understanding the Discovery API

https://dbterd.datnguyen.de/1.10-beta/nav/guide/dbt-cloud/read-artifact-from-an-environment.html