dbt Cloud TriggerJob API response cannot be decoded

christophs91 commented 8 months ago

Expected Behavior

It should render the API response from dbt Cloud and not throw an exception.

Actual Behaviour

The flow is triggering the dbt Cloud Job but cannot render the API response and returns this error:

Request failed '200' and body 'null'

Error decoding HTTP response body: Error decoding JSON stream for type [runResponse]: Java 8 date/time type `java.time.ZonedDateTime` not supported by default: add Module "com.fasterxml.jackson.datatype:jackson-datatype-jsr310" to enable handling
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 569] (through reference chain: io.kestra.plugin.dbt.cloud.models.RunResponse$RunResponseBuilderImpl["data"]->io.kestra.plugin.dbt.cloud.models.Run$RunBuilderImpl["created_at"])
Error decoding JSON stream for type [runResponse]: Java 8 date/time type `java.time.ZonedDateTime` not supported by default: add Module "com.fasterxml.jackson.datatype:jackson-datatype-jsr310" to enable handling
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 569] (through reference chain: io.kestra.plugin.dbt.cloud.models.RunResponse$RunResponseBuilderImpl["data"]->io.kestra.plugin.dbt.cloud.models.Run$RunBuilderImpl["created_at"])
Java 8 date/time type `java.time.ZonedDateTime` not supported by default: add Module "com.fasterxml.jackson.datatype:jackson-datatype-jsr310" to enable handling
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 569] (through reference chain: io.kestra.plugin.dbt.cloud.models.RunResponse$RunResponseBuilderImpl["data"]->io.kestra.plugin.dbt.cloud.models.Run$RunBuilderImpl["created_at"])

Attached you can find the full log.

kestra-execution-20240312164932-59h8Bm6BE7aT8jbyGNOXGQ-6SERYgot9OUAysiVUVkaHD.log

Steps To Reproduce

This is my flow definition (I have removed the account number, job number and the token):

tasks:

id: "1" type: io.kestra.plugin.dbt.cloud.TriggerRun accountId: "" baseUrl: https://emea.dbt.com jobId: "" parseRunResults: false token: ***** wait: true

Environment Information

Kestra Version: 0.15.4
Plugin version: 0.15.1
Operating System (OS / Docker / Kubernetes): Ubuntu VM 22.04 / Docker version 20.10.17 / Docker Compose version v2.24.5
Java Version (If not docker):

Example flow

` tasks:

id: "1" type: io.kestra.plugin.dbt.cloud.TriggerRun accountId: "" baseUrl: https://emea.dbt.com jobId: "" parseRunResults: false token: ***** wait: true`

christophs91 commented 8 months ago

Hi @loicmathieu,

I wanted to give you a short feedback. I can confirm that the response can now be decoded. So the dbt Cloud Job Run is being triggered by kestra.

There is now a different error message after I got the response back. Should I open a new issue? I have removed some IDs.

`2024-03-19T11:57:15.693Z INFO Job status OK with response: RunResponse(data=Run(id=, triggerId=, accountId= , projectId= , jobId= , jobDefinitionId= , status=1, gitBranch=null, gitSha=null, statusMessage=null, dbtVersion=1.7.0-latest, createdAt=2024-03-19T11:57:15.559970Z, updatedAt=2024-03-19T11:57:15.559985Z, dequeuedAt=null, startedAt=null, finishedAt=null, lastCheckedAt=null, lastHeartbeatAt=null, shouldStartAt=null, ownerThreadId=null, executedByThreadId=null, deferringRunId=null, artifactsSaved=false, artifactS3Path=null, hasDocsGenerated=false, hasSourcesGenerated=false, notificationsSent=false, scribeEnabled=true, trigger=Trigger(id= , cause=Triggered by Kestra., jobDefinitionId= , gitBranch=null, gitSha=null, githubPullRequestId=null, schemaOverride=null, dbtVersionOverride=null, threadsOverride=null, targetNameOverride=null, generateDocsOverride=null, timeoutSecondsOverride=null, stepsOverride=null, createdAt=2024-03-19T11:57:15.548837Z), job=Job(id= , accountId= , projectId= , environmentId= , name=CW1 Job, dbtVersion=null, triggers=JobTriggers(githubWebhook=false, gitProviderWebhook=false, schedule=true, customBranchOnly=null), executeSteps=[], settings=JobSettings(threads=4, targetName=default), state=1, generateDocs=false, schedule=JobSchedule(cron=0 5,13,21 1,2,3,4,5, date=days_of_week, time=at_exact_hours)), environment=null, runSteps=[], duration=00:00:00, queuedDuration=00:00:00, runDuration=00:00:00, durationHumanized=0 minutes, queuedDurationHumanized=0 minutes, runDurationHumanized=0 minutes, finishedAtHumanized=0 minutes from now, statusHumanized=Queued, createdAtHumanized=0 minutes ago), status=Status(code=200, isSuccess=true, userMessage=Success!, developerMessage=)) 2024-03-19T11:57:16.280Z ERROR Request failed '401' and body '{"status": {"code": 401, "is_success": false, "user_message": "Invalid token.", "developer_message": null}, "data": null}' 2024-03-19T11:57:16.280Z ERROR Unauthorized 2024-03-19T11:57:16.280Z TRACE io.micronaut.http.client.exceptions.HttpClientResponseException: Request failed '401' and body '{"status": {"code": 401, "is_success": false, "user_message": "Invalid token.", "developer_message": null}, "data": null}' at io.kestra.plugin.dbt.cloud.AbstractDbtCloud.request(AbstractDbtCloud.java:84) at io.kestra.plugin.dbt.cloud.CheckStatus.fetchRunResponse(CheckStatus.java:192) at io.kestra.plugin.dbt.cloud.CheckStatus.lambda$run$1(CheckStatus.java:109) at io.kestra.core.utils.Rethrow.lambda$throwSupplier$2(Rethrow.java:70) at io.kestra.core.utils.Await.lambda$untilSupplier$0(Await.java:59) at io.kestra.core.utils.Await.until(Await.java:40) at io.kestra.core.utils.Await.until(Await.java:31) at io.kestra.core.utils.Await.until(Await.java:72) at io.kestra.plugin.dbt.cloud.CheckStatus.run(CheckStatus.java:107) at io.kestra.plugin.dbt.cloud.TriggerRun.run(TriggerRun.java:230) at io.kestra.plugin.dbt.cloud.TriggerRun.run(TriggerRun.java:24) at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:710) Caused by: io.micronaut.http.client.exceptions.HttpClientResponseException: Unauthorized at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.makeErrorFromRequestBody(DefaultHttpClient.java:2259) at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.forwardResponseToPromise(DefaultHttpClient.java:2210) at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2179) at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2147) at io.micronaut.http.client.netty.SimpleChannelInboundHandlerInstrumented.channelRead0(SimpleChannelInboundHandlerInstrumented.java:46) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.handler.codec.http2.AbstractHttp2StreamChannel$Http2ChannelUnsafe.doRead0(AbstractHttp2StreamChannel.java:971) at io.netty.handler.codec.http2.AbstractHttp2StreamChannel.fireChildRead(AbstractHttp2StreamChannel.java:600) at io.netty.handler.codec.http2.Http2MultiplexHandler.channelRead(Http2MultiplexHandler.java:195) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.codec.http2.Http2FrameCodec.onHttp2Frame(Http2FrameCodec.java:712) at io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onDataRead(Http2FrameCodec.java:651) at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onDataRead(Http2FrameListenerDecorator.java:36) at io.netty.handler.codec.http2.Http2EmptyDataFrameListener.onDataRead(Http2EmptyDataFrameListener.java:49) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:322) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:415) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:250) at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:188) at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63) at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393) at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at io.micronaut.http.client.netty.ResettableReadTimeoutHandler$NextInterceptor.channelRead(ResettableReadTimeoutHandler.java:92) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) at io.micronaut.http.client.netty.ResettableReadTimeoutHandler.channelRead(ResettableReadTimeoutHandler.java:64) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Suppressed: java.lang.Exception: #block terminated with an error at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:103) at reactor.core.publisher.Flux.blockFirst(Flux.java:2707) at io.micronaut.http.client.netty.DefaultHttpClient$1.exchange(DefaultHttpClient.java:571) at io.micronaut.http.client.BlockingHttpClient.exchange(BlockingHttpClient.java:77) at io.kestra.plugin.dbt.cloud.AbstractDbtCloud.request(AbstractDbtCloud.java:78) at io.kestra.plugin.dbt.cloud.CheckStatus.fetchRunResponse(CheckStatus.java:192) at io.kestra.plugin.dbt.cloud.CheckStatus.lambda$run$1(CheckStatus.java:109)

2024-03-19T11:57:16.281Z TRACE at io.kestra.core.utils.Rethrow.lambda$throwSupplier$2(Rethrow.java:70) at io.kestra.core.utils.Await.lambda$untilSupplier$0(Await.java:59) at io.kestra.core.utils.Await.until(Await.java:40) at io.kestra.core.utils.Await.until(Await.java:31) at io.kestra.core.utils.Await.until(Await.java:72) at io.kestra.plugin.dbt.cloud.CheckStatus.run(CheckStatus.java:107) at io.kestra.plugin.dbt.cloud.TriggerRun.run(TriggerRun.java:230) at io.kestra.plugin.dbt.cloud.TriggerRun.run(TriggerRun.java:24) at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:710)`

Thanks and best regards!

loicmathieu commented 8 months ago

@christophs91 yes please, open a different issue with a reproducer and the stacktrace.

kestra-io / plugin-dbt