[platform] Upgrading to v0.63.2 results in continuous failures for all sync jobs.

Helm Chart Version

0.233.2

What step the error happened?

During the Sync

Relevant information

I'm in the process of upgrading an Airbyte deployment on GKE Autopilot from v0.50.45 to v0.63.2. Following the migration guide here: https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm#migration-steps, the deployment was upgraded and services started successfully, but syncs fail to run with the following errors:

Caused by: java.lang.NullPointerException: connectionId is marked non-null but is null This seems to be similar to https://github.com/airbytehq/airbyte/issues/38345 Full error log:

java.lang.RuntimeException: Failure processing workflow task. WorkflowId=connection_manager_05d7c17e-fb62-4f06-a39d-47eb8f411b62, RunId=e5c14e41-b47b-450b-892c-eeb841d8a2cc, Attempt=924
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:404) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.wrapFailure(WorkflowWorker.java:261) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:110) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event 24 of type 'EVENT_TYPE_WORKFLOW_TASK_STARTED' during execution. {WorkflowTaskStartedEventId=24, CurrentStartedEventId=24}
at io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:373) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:297) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:260) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:249) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:231) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:165) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:133) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:98) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:413) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:320) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:261) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]
... 3 more
Caused by: java.lang.RuntimeException: WorkflowTask: failure executing SCHEDULED->WORKFLOW_TASK_STARTED, transition history is [CREATED->WORKFLOW_TASK_SCHEDULED]
at io.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:163) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:103) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.EntityStateMachineBase.handleEvent(EntityStateMachineBase.java:84) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleSingleEvent(WorkflowStateMachines.java:419) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:295) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:260) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:249) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:231) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:165) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:133) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:98) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:413) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:320) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:261) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]
... 3 more
Caused by: java.lang.NullPointerException: connectionId is marked non-null but is null
at io.airbyte.commons.temporal.scheduling.ConnectionUpdaterInput$ConnectionUpdaterInputBuilder.connectionId(ConnectionUpdaterInput.java:23) ~[io.airbyte-airbyte-commons-temporal-0.63.2.jar:?]
at io.airbyte.commons.temporal.TemporalWorkflowUtils.buildStartWorkflowInput(TemporalWorkflowUtils.java:37) ~[io.airbyte-airbyte-commons-temporal-0.63.2.jar:?]
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.runMandatoryActivityWithOutput(ConnectionManagerWorkflowImpl.java:775) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.runMandatoryActivity(ConnectionManagerWorkflowImpl.java:798) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.reportFailure(ConnectionManagerWorkflowImpl.java:395) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.reportFailure(ConnectionManagerWorkflowImpl.java:385) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.run(ConnectionManagerWorkflowImpl.java:223) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]
at ConnectionManagerWorkflowImplProxy.run$accessor$aZ2BQpyy(Unknown Source) ~[?:?]
at ConnectionManagerWorkflowImplProxy$auxiliary$cnnGarfX.call(Unknown Source) ~[?:?]
at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79) ~[io.airbyte-airbyte-micronaut-temporal-0.63.2.jar:?]
at ConnectionManagerWorkflowImplProxy.run(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:339) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:314) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:135) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
... 3 more

Caused by: io.temporal.failure.ApplicationFailure: message='baseUrl is invalid.', type='java.lang.IllegalStateException', nonRetryable=false This seems to be the same or similar issue to https://github.com/airbytehq/airbyte/issues/38854, but none of the workaround methods listed in that thread have worked so far. Full error log:

io.temporal.failure.ActivityFailure: Activity with activityType='AttemptFailureWithAttemptNumber' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=d7cba4ac-668a-368d-9cb8-62da245f4a07, identity='1@airbyte-worker-5665bc6d56-vmxg9', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED",2024-06-26T14:56:48.909720041Z
024-06-26 14:56:48 [1;31mERROR[m i.a.w.t.s.ConnectionManagerWorkflowImpl(runMandatoryActivityWithOutput):747 - [ACTIVITY-FAILURE] Connection null failed to run an activity.(AttemptNumberFailureInput).  Connection manager workflow will be restarted after a delay of PT10M.,2024-06-26T14:56:48.909684401Z
at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?]",2024-06-26T14:56:48.906697614Z
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]",2024-06-26T14:56:48.906692370Z
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]",2024-06-26T14:56:48.906687932Z
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906682791Z
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906677256Z
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906661071Z
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906656366Z
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906651396Z
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906646292Z
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906632640Z
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906585601Z
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]",2024-06-26T14:56:48.906578949Z
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]",2024-06-26T14:56:48.906554079Z
at io.airbyte.workers.temporal.scheduling.activities.ConfigFetchActivityImpl.isWorkspaceTombstone(ConfigFetchActivityImpl.java:254) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]",2024-06-26T14:56:48.906550625Z
at io.airbyte.api.client.generated.WorkspaceApi.getWorkspaceByConnectionIdWithTombstone(WorkspaceApi.kt:509) ~[io.airbyte-airbyte-api-0.63.2.jar:?]",2024-06-26T14:56:48.906546918Z
at io.airbyte.api.client.generated.WorkspaceApi.getWorkspaceByConnectionIdWithTombstoneWithHttpInfo(WorkspaceApi.kt:3140) ~[io.airbyte-airbyte-api-0.63.2.jar:?]",2024-06-26T14:56:48.906543110Z
Caused by: io.temporal.failure.ApplicationFailure: message='baseUrl is invalid.', type='java.lang.IllegalStateException', nonRetryable=false",2024-06-26T14:56:48.906539540Z
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]",2024-06-26T14:56:48.906535910Z
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]",2024-06-26T14:56:48.906530800Z
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]",2024-06-26T14:56:48.906525790Z
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]",2024-06-26T14:56:48.906520843Z
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]",2024-06-26T14:56:48.906515335Z
at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906509783Z
at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906495406Z
at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906490297Z
at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:135) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906483516Z
at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906477543Z
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:314) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906472260Z
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:339) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906467224Z
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]",2024-06-26T14:56:48.906461364Z
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]",2024-06-26T14:56:48.906458070Z
at ConnectionManagerWorkflowImplProxy.run(Unknown Source) ~[?:?]",2024-06-26T14:56:48.906454638Z
at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79) ~[io.airbyte-airbyte-micronaut-temporal-0.63.2.jar:?]",2024-06-26T14:56:48.906450709Z
at ConnectionManagerWorkflowImplProxy$auxiliary$cnnGarfX.call(Unknown Source) ~[?:?]",2024-06-26T14:56:48.906446758Z
at ConnectionManagerWorkflowImplProxy.run$accessor$aZ2BQpyy(Unknown Source) ~[?:?]",2024-06-26T14:56:48.906442942Z
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.run(ConnectionManagerWorkflowImpl.java:166) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]",2024-06-26T14:56:48.906438311Z
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.isTombstone(ConnectionManagerWorkflowImpl.java:238) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]",2024-06-26T14:56:48.906415865Z
at jdk.proxy2/jdk.proxy2.$Proxy83.isWorkspaceTombstone(Unknown Source) ~[?:?]",2024-06-26T14:56:48.906409008Z
at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:60) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906403478Z
at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:83) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906397281Z
at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:49) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.906391184Z
at java.base/java.lang.Thread.getStackTrace(Thread.java:2450) [?:?]",2024-06-26T14:56:48.906385190Z

"Caused by: io.temporal.failure.ApplicationFailure: message='Cannot invoke "java.lang.Long.longValue()" because the return value of "io.airbyte.workers.temporal.scheduling.activities.JobCreationAndStatusUpdateActivity$AttemptNumberFailureInput.getJobId()" is null', type='java.lang.NullPointerException', nonRetryable=false" This seems distinct, though as seen in the log from above it's visible in the baseUrl error stack as well.

WARN[m i.t.i.r.ReplayWorkflowTaskHandler(failureToWFTResult):279 - Workflow task processing failure. startedEventId=24, WorkflowId=connection_manager_dd9dfdc0-c6ab-4a00-91a4-52cba0d13502, RunId=6e9d4a35-67b8-413f-9b42-09a71560c195. If seen continuously the workflow might be stuck.",2024-06-26T14:56:48.912141643Z
INFO[m i.a.w.t.s.ConnectionManagerWorkflowImpl(runMandatoryActivityWithOutput):773 - Finished wait for connection null, restarting connection manager workflow",2024-06-26T14:56:48.910478014Z
WARN[m i.a.w.t.s.ConnectionManagerWorkflowImpl(runMandatoryActivityWithOutput):770 - Can't properly fail the job, the next run will clean the state in the EnsureCleanJobStateActivity",2024-06-26T14:56:48.910389040Z
INFO[m i.a.w.t.s.ConnectionManagerWorkflowImpl(runMandatoryActivityWithOutput):756 - Waiting PT10M before restarting the workflow for connection null, to prevent spamming temporal with restarts.",2024-06-26T14:56:48.910040468Z
at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?]",2024-06-26T14:56:48.910019847Z
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]",2024-06-26T14:56:48.910014685Z
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]",2024-06-26T14:56:48.910009812Z
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.910005197Z
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.910000143Z
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909994767Z
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909989662Z
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909984120Z
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909978606Z
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909972701Z
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909954511Z
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]",2024-06-26T14:56:48.909945966Z
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]",2024-06-26T14:56:48.909941493Z
at io.airbyte.workers.temporal.scheduling.activities.JobCreationAndStatusUpdateActivityImpl.attemptFailureWithAttemptNumber(JobCreationAndStatusUpdateActivityImpl.java:163) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]",2024-06-26T14:56:48.909936668Z
Caused by: io.temporal.failure.ApplicationFailure: message='Cannot invoke ""java.lang.Long.longValue()"" because the return value of ""io.airbyte.workers.temporal.scheduling.activities.JobCreationAndStatusUpdateActivity$AttemptNumberFailureInput.getJobId()"" is null', type='java.lang.NullPointerException', nonRetryable=false",2024-06-26T14:56:48.909931172Z
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]",2024-06-26T14:56:48.909925431Z
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]",2024-06-26T14:56:48.909918743Z
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]",2024-06-26T14:56:48.909893552Z
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]",2024-06-26T14:56:48.909888607Z
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]",2024-06-26T14:56:48.909883686Z
at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909878030Z
at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909872834Z
at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909867681Z
at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:135) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909862002Z
at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909857656Z
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:314) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909852567Z
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:339) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909847439Z
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]",2024-06-26T14:56:48.909841422Z
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]",2024-06-26T14:56:48.909835984Z
at ConnectionManagerWorkflowImplProxy.run(Unknown Source) ~[?:?]",2024-06-26T14:56:48.909830279Z
at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79) ~[io.airbyte-airbyte-micronaut-temporal-0.63.2.jar:?]",2024-06-26T14:56:48.909824646Z
at ConnectionManagerWorkflowImplProxy$auxiliary$cnnGarfX.call(Unknown Source) ~[?:?]",2024-06-26T14:56:48.909818795Z
at ConnectionManagerWorkflowImplProxy.run$accessor$aZ2BQpyy(Unknown Source) ~[?:?]",2024-06-26T14:56:48.909813518Z
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.run(ConnectionManagerWorkflowImpl.java:223) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]",2024-06-26T14:56:48.909807195Z
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.reportFailure(ConnectionManagerWorkflowImpl.java:385) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]",2024-06-26T14:56:48.909784650Z
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.reportFailure(ConnectionManagerWorkflowImpl.java:395) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]",2024-06-26T14:56:48.909779298Z
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.runMandatoryActivity(ConnectionManagerWorkflowImpl.java:798) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]",2024-06-26T14:56:48.909774229Z
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.runMandatoryActivityWithOutput(ConnectionManagerWorkflowImpl.java:744) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]",2024-06-26T14:56:48.909767702Z
at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.lambda$runMandatoryActivity$1(ConnectionManagerWorkflowImpl.java:799) ~[io.airbyte-airbyte-workers-0.63.2.jar:?]",2024-06-26T14:56:48.909760996Z
at jdk.proxy2/jdk.proxy2.$Proxy82.attemptFailureWithAttemptNumber(Unknown Source) ~[?:?]",2024-06-26T14:56:48.909754227Z
at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:60) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909747719Z
at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:83) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909742218Z
at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:49) ~[temporal-sdk-1.22.3.jar:?]",2024-06-26T14:56:48.909735706Z
at java.base/java.lang.Thread.getStackTrace(Thread.java:2450) [?:?]",2024-06-26T14:56:48.909728466Z

Deleting all Airbyte deployments and volumes fully and reapplying the chart seems to allow a few syncs to make progress for a few minutes, only to start failing in the same manner. I have not tried resetting the airbyte database, but that's not really an option in any case.

This issue persists on 0.63.1 and 0.60.1; unfortunately upgrading the cluster to 0.63.2 has made rolling back to 0.50.45 impossible, as various SQL queries fail due to missing columns/other backward compatibility issues.

Relevant log output

No response

airbytehq / airbyte