search

cube-js / cube

📊 Cube — The Semantic Layer for Building Data Applications
https://cube.dev
Other
17.87k stars 1.77k forks source link

Help with DBT and CubeDev integration #8465

Closed kraft6 closed 1 month ago

kraft6 commented 3 months ago

Problem

My main question is when I use both DBT and Cube Dev, which one is connecting to the source data? My assumption is dbt is being used to connect to the source data through the profile settings. My profile is using oauth to connect to trino:

type: trino
user: [me](mailto:tom.kraft.cx0m@statefarm.com)
method: oauth
host: myhost
catalog: views
schema: myschema
port: 443
threads: 8
ssl: true

I am using dbt-core version 1.8.3 with the trino plugin 1.8.1 and the latest cubejs/cube image from docker.

When I fire up the playground: localhost:4000, I can see my measures and dimensions, but when I go to run I am getting the error “execution error:Unauthorized”. Do I have to have anything in the .env CubeDev file? I am expecting dbt to connect to the sources and the results passed through. Dbt is connecting to starburst views.

Below are the errors I am capturing from Docker:

2024-07-15 11:34:08 Error while querying: d59c7e11-00fb-4f47-8123-731da8fec1d0-span-1 (321ms)

2024-07-15 11:34:08 {

2024-07-15 11:34:08   "queueId": 2,

2024-07-15 11:34:08   "processingId": 2,

2024-07-15 11:34:08   "queueSize": 1,

2024-07-15 11:34:08   "queryKey": [

2024-07-15 11:34:08     "my_query",

2024-07-15 11:34:08     [],

2024-07-15 11:34:08     []

2024-07-15 11:34:08   ],

2024-07-15 11:34:08   "queuePrefix": "SQL_QUERY_STANDALONE_default",

2024-07-15 11:34:08   "timeInQueue": 0,

2024-07-15 11:34:08   "addedToQueueTime": 1721061247746

2024-07-15 11:34:08 }

2024-07-15 11:34:08 Error: execution error:Unauthorized

2024-07-15 11:34:08     at toError ([/cube/node_modules/@cubejs-backend/prestodb-driver/src/PrestoDriver.ts:115:37](mailto:/cube/node_modules/@cubejs-backend/prestodb-driver/src/PrestoDriver.ts:115:37))

2024-07-15 11:34:08     at error ([/cube/node_modules/@cubejs-backend/prestodb-driver/src/PrestoDriver.ts:166:20](mailto:/cube/node_modules/@cubejs-backend/prestodb-driver/src/PrestoDriver.ts:166:20))

2024-07-15 11:34:08     at /cube/node_modules/presto-client/lib/presto-client/index.js:235:17

2024-07-15 11:34:08     at IncomingMessage.<anonymous> (/cube/node_modules/presto-client/lib/presto-client/index.js:133:13)

2024-07-15 11:34:08     at IncomingMessage.emit (node:events:529:35)

2024-07-15 11:34:08     at endReadableNT (node:internal/streams/readable:1400:12)

2024-07-15 11:34:08     at processTicksAndRejections (node:internal/process/task_queues:82:21)

2024-07-15 11:34:08 Dropping Cache: d59c7e11-00fb-4f47-8123-731da8fec1d0-span-1

2024-07-15 11:34:08 {

2024-07-15 11:34:08   "cacheKey": [

2024-07-15 11:34:08     "my_query",

2024-07-15 11:34:08     [],

2024-07-15 11:34:08     []

2024-07-15 11:34:08   ],

2024-07-15 11:34:08   "spanId": "05ed46c9e18d50c79b8e802763074bf4",

2024-07-15 11:34:08   "primaryQuery": true

2024-07-15 11:34:08 }

2024-07-15 11:34:08 Error: execution error:Unauthorized

2024-07-15 11:34:08     at QueryQueue.parseResult ([/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:397:13](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:397:13))

2024-07-15 11:34:08     at QueryQueue.executeInQueue ([/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:369:19](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:369:19))

2024-07-15 11:34:08     at processTicksAndRejections (node:internal/process/task_queues:95:5)

2024-07-15 11:34:08     at [/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:756:17](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:756:17)

2024-07-15 11:34:08     at QueryOrchestrator.fetchQuery ([/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryOrchestrator.ts:302:20](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryOrchestrator.ts:302:20))

2024-07-15 11:34:08     at OrchestratorApi.executeQuery ([/cube/node_modules/@cubejs-backend/server-core/src/core/OrchestratorApi.ts:98:20](mailto:/cube/node_modules/@cubejs-backend/server-core/src/core/OrchestratorApi.ts:98:20))

2024-07-15 11:34:08     at [/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1523:21](mailto:/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1523:21)

2024-07-15 11:34:08     at async Promise.all (index 0)

2024-07-15 11:34:08     at ApiGateway.getSqlResponseInternal ([/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1521:31](mailto:/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1521:31))

2024-07-15 11:34:08     at [/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1699:28](mailto:/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1699:28)

2024-07-15 11:34:08     at async Promise.all (index 0)

2024-07-15 11:34:08     at ApiGateway.load ([/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1690:23](mailto:/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1690:23))

2024-07-15 11:34:08     at [/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:278:7](mailto:/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:278:7)

2024-07-15 11:34:08 Dropping Cache: d59c7e11-00fb-4f47-8123-731da8fec1d0-span-1

2024-07-15 11:34:08 {

2024-07-15 11:34:08   "cacheKey": [

2024-07-15 11:34:08     "my_query",

2024-07-15 11:34:08     [],

2024-07-15 11:34:08     []

2024-07-15 11:34:08   ],

2024-07-15 11:34:08   "spanId": "1de2fbba5de7c7fe331a9c8a2329e88d",

2024-07-15 11:34:08   "primaryQuery": true,

2024-07-15 11:34:08   "renewCycle": true

2024-07-15 11:34:08 }

2024-07-15 11:34:08 Error: execution error:Unauthorized

2024-07-15 11:34:08     at QueryQueue.parseResult ([/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:397:13](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:397:13))

2024-07-15 11:34:08     at QueryQueue.executeInQueue ([/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:369:19](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:369:19))

2024-07-15 11:34:08     at processTicksAndRejections (node:internal/process/task_queues:95:5)

2024-07-15 11:34:08     at [/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:756:17](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:756:17)

2024-07-15 11:34:08 Error while renew cycle: d59c7e11-00fb-4f47-8123-731da8fec1d0-span-1

2024-07-15 11:34:08 --

2024-07-15 11:34:08 "my_query"

2024-07-15 11:34:08 --

2024-07-15 11:34:08 {

2024-07-15 11:34:08   "query_values": []

2024-07-15 11:34:08 }

2024-07-15 11:34:08 Error: execution error:Unauthorized

2024-07-15 11:34:08     at QueryQueue.parseResult ([/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:397:13](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:397:13))

2024-07-15 11:34:08     at QueryQueue.executeInQueue ([/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:369:19](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:369:19))

2024-07-15 11:34:08     at processTicksAndRejections (node:internal/process/task_queues:95:5)

2024-07-15 11:34:08     at [/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:756:17](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:756:17)

2024-07-15 11:34:08 Error querying db: d59c7e11-00fb-4f47-8123-731da8fec1d0-span-1

2024-07-15 11:34:08 --

2024-07-15 11:34:08 "my_query"

2024-07-15 11:34:08 --

2024-07-15 11:34:08 {

2024-07-15 11:34:08   "params": []

2024-07-15 11:34:08 }

2024-07-15 11:34:08 Error: execution error:Unauthorized

2024-07-15 11:34:08     at QueryQueue.parseResult ([/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:397:13](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:397:13))

2024-07-15 11:34:08     at QueryQueue.executeInQueue ([/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:369:19](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:369:19))

2024-07-15 11:34:08     at processTicksAndRejections (node:internal/process/task_queues:95:5)

2024-07-15 11:34:08     at [/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:756:17](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:756:17)

2024-07-15 11:34:08     at QueryOrchestrator.fetchQuery ([/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryOrchestrator.ts:302:20](mailto:/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryOrchestrator.ts:302:20))

2024-07-15 11:34:08     at OrchestratorApi.executeQuery ([/cube/node_modules/@cubejs-backend/server-core/src/core/OrchestratorApi.ts:98:20](mailto:/cube/node_modules/@cubejs-backend/server-core/src/core/OrchestratorApi.ts:98:20))

2024-07-15 11:34:08     at [/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1523:21](mailto:/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1523:21)

2024-07-15 11:34:08     at async Promise.all (index 0)

2024-07-15 11:34:08     at ApiGateway.getSqlResponseInternal ([/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1521:31](mailto:/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1521:31))

2024-07-15 11:34:08     at [/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1699:28](mailto:/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1699:28)

2024-07-15 11:34:08     at async Promise.all (index 0)

2024-07-15 11:34:08     at ApiGateway.load ([/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1690:23](mailto:/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1690:23))

2024-07-15 11:34:08     at [/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:278:7](mailto:/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:278:7)

2024-07-15 11:34:08 Orchestrator error: d59c7e11-00fb-4f47-8123-731da8fec1d0-span-1 (350ms)

2024-07-15 11:34:08 --

2024-07-15 11:34:08 {

2024-07-15 11:34:08   "measures": [

2024-07-15 11:34:08     "int_gensig__claim_closed_raw.count"

2024-07-15 11:34:08   ],

2024-07-15 11:34:08   "dimensions": [

2024-07-15 11:34:08     "int_gensig__claim_closed_raw.CLAIM_LAST_CLS_YR"

2024-07-15 11:34:08   ],

2024-07-15 11:34:08   "order": {

2024-07-15 11:34:08     "int_gensig__claim_closed_raw.count": "desc"

2024-07-15 11:34:08   }

2024-07-15 11:34:08 }

2024-07-15 11:34:08 --

2024-07-15 11:34:08 {

2024-07-15 11:34:08   "securityContext": {

2024-07-15 11:34:08     "iat": 1721059945,

2024-07-15 11:34:08     "exp": 1721146345

2024-07-15 11:34:08   }

2024-07-15 11:34:08 }

2024-07-15 11:34:08 Error: execution error:Unauthorized
igorlukanin commented 3 months ago

Hi @kraft6 👋

when I use both DBT and Cube Dev, which one is connecting to the source data?

Cube is.

My assumption is dbt is being used to connect to the source data through the profile settings.

Well, I have to admit that this is an incorrect assumption. Cube does not access or use dbt profiles. However, it can read the manifest file and simplify defining the data model through that.

Do I have to have anything in the .env CubeDev file?

You mentioned that your data source is Trino. You need to configure a connection from Cube to Trino then. Please use these instructions: https://cube.dev/docs/product/configuration/data-sources/trino

You might also find this guide helpful: https://cube.dev/docs/guides/dbt

igorlukanin commented 2 months ago

Hi @kraft6 👋 Did my advice help?

kraft6 commented 2 months ago

It did. Thank you. U can claw this request.