[platform] DBT Cloud Transformation not working after Sync job

[abhinav-sreekumar]

Topic

No response

Relevant information

Hello Airbyte Community, I am trying to integrate DBT Cloud with Airbyte Cloud. I've configured the connection using the access URL in the format https://.us1.dbt.com/ and a service token. The connection appears to be working since I can select DBT Cloud jobs within Airbyte. However, the transformation is not running after the sync job. When I check the logs, I see the following:

2024-08-21 09:21:33 platform > ----- START POST REPLICATION OPERATIONS -----
2024-08-21 09:21:33 platform > 
2024-08-21 09:21:33 platform > Invoking webhook operation XXXXXXXXXXXXXXXX...
2024-08-21 09:21:34 platform > Invoking webhook operation "dbt cloud" using URL "https://cloud.getdbt.com/api/v2/accounts/XXXXXXXXX/jobs/XXXXXXXXX/run/"...
2024-08-21 09:21:35 platform > Webhook operation "dbt cloud" (XXXXXXXXXXXXXXXX) response: 401 {"status": {"code": 401, "is_success": false, "user_message": "Invalid token.", "developer_message": null}, "data": null}
2024-08-21 09:21:35 platform > Webhook XXXXXXXXXXX completed unsuccessfully
2024-08-21 09:21:35 platform > 
2024-08-21 09:21:35 platform > ----- END POST REPLICATION OPERATIONS -----

Interestingly, when I use Postman to make an API call to https://.us1.dbt.com/api/v2/accounts/XXXXXXXXX/jobs/XXXXXXXXX/run/ it works fine. Could someone help me figure out why the transformation isn't running as expected in Airbyte? Why is airbyte making an api to call cloud.getdbt.com when I have specifically mentioned a different access URL? Thank you!

[marcosmarxm]

Thanks @abhinav-sreekumar for reporting the problem. @airbytehq/platform-compose can someone take a look into this?

[keyihuang]

I think by default we will use https://cloud.getdbt.com unless they put their custom access URL. we can check in the db see if their custom access URL is being saved/stored. Is workspace ID can be provided?

[abhinav-sreekumar]

I'm assuming the custom access URL was being saved because I was able to pick between my different DBT cloud jobs. This is my workspace ID.43cfe2f0-ac3f-482b-83fe-55cb2ec89047. Today on trying to integrate with DBT cloud in the same way. Airbyte is throwing this error dbt Cloud denied access. Please verify your service token and access URL. I made an API call to dbt cloud with the same access url and service token and i'm getting a response on postman

[keyihuang]

I'm assuming the custom access URL was being saved because I was able to pick between my different DBT cloud jobs. This is my workspace ID.43cfe2f0-ac3f-482b-83fe-55cb2ec89047. Today on trying to integrate with DBT cloud in the same way. Airbyte is throwing this error dbt Cloud denied access. Please verify your service token and access URL. I made an API call to dbt cloud with the same access url and service token and i'm getting a response on postman

Hi, Would you please try adding your DBT integration configs in Settings -> Workspace -> Integrations? (https://cloud.airbyte.com/workspaces/43cfe2f0-ac3f-482b-83fe-55cb2ec89047/settings/dbt-cloud) using your: 1. custom access URL, 2. service token Have you done this workflow before? Or you have added it once but then removed it from Settings page?

More context: I took a look into our db. I don't see any DBT config in your workspace. You can also verify this by checking your Settings page: https://cloud.airbyte.com/workspaces/43cfe2f0-ac3f-482b-83fe-55cb2ec89047/settings/dbt-cloud.

So I would suggest:

1. Go to your connection and remove the existing DBT integration job (since it is not working now without a webhook config reference in your workspace).
2. Adding/Re-adding your DBT integration in Settings page.
3. Go back to your connection, select the new DBT integration job popped there.

Let us know if this works or any other questions, happy to help!

[abhinav-sreekumar]

I had already added the DBT cloud config and the connection was successful. On the transformation page of the connection i was able to pick between the different DBT jobs. The issue was that it was not running the jobs. On friday, I removed the Integration and while trying to add it again(going through the same flow that I did the last time) It stopped working.

Edit. Today On trying again. I was able to add the Integration and the dbt cloud job ran successfully so I'll close this issue. Thanks for the help. However the intermittent functionality and the lack of information on why these features are working or not is unfortunate.

[malikdiarra]

Hi @abhinav-sreekumar, we indeed had an issue with dbt integration that we resolved at the end of last week. I'm sorry for the poor communication.