Feature request: dbt cloud runner

[adrianbr]

We want to equally support dbt core and cloud. To offer cloud support, we need to create some kind of operator for this api

https://docs.getdbt.com/dbt-cloud/api-v3#/

The operator should have the following options:

• wait for outcome toggle - we can kick off a model run but we can also wait until it's finished - or not, depending on what user wants.

[jtcohen6]

@adrianbr I recommend using v2 of the dbt Cloud admin API, as it's more standard for this type of integration.

Assuming the user has already defined a job, possibly with steps like dbt build --select source:my_source_loaded_via_dlt+ — they would then need to provide your integration with:

• account ID
• project ID
• job ID
• API key (service token) with "Job Admin" permission for the relevant account/project

After kicking off, you can either poll for job status, or get a webhook on completion: https://docs.getdbt.com/docs/deploy/webhooks

Check out a similar example in the documentation, for triggering a dbt Cloud job via GHA:

• https://docs.getdbt.com/guides/orchestration/custom-cicd-pipelines/3-dbt-cloud-job-on-merge
• https://gist.github.com/b-per/f4942acb8584638e3be363cb87769b48

[rudolfix]

@jtcohen6 thx for the hint, this looks way easier than running dbt locally (no need to install deps, refresh the project, run commands etc.)

there are a few things on top of running jobs that I'd like to try:

• is there a way to pass credentials to the job? I can see credentials_id in the Environment (https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Retrieve%20Environment) but I cannot find more info. then our users could setup them once.
• even better: is there a way to get the credentials from dbt cloud? then setup on the dlt side would not be needed at all. (we do that with Airflow for example)
• is there a way to pass additional variables when triggering job? our users configure ie. schema name to which they load data and that we typically pass to dbt packages as source schema. I cannot find that in https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Trigger%20Job%20Run

I'll OFC dig deeper into the API but maybe you just know the answers :)