search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.13k stars 4.13k forks source link

Worker can't create job pod's #32461

Closed Kamaradeivanov closed 12 months ago

Kamaradeivanov commented 12 months ago

What method are you using to run Airbyte?

Kubernetes

Platform Version or Helm Chart Version

Chart 0.49.22 - App 0.50.34

What step the error happened?

During the Sync

Revelant information

I recently setup airbyte with the last helm chart (0.49.22) on a GKE v1.27.4 cluster. I just overide configuration regarding ingress for webapp.

Install is fine, web app is accessible with the URI configure in the ingress host.

Error appear when I try to create a source, I try with certified Postgres and MongoDb connectors. After the click on Set up source button, errors appear on 2 pods :

There is no pod or job e-mongodb-v2-check-xxxxxxxxxxxxxx that appear in my kubernetes namespace.

I check RBAC access but everithing seems fine : ServiceAccount airbyte-admin, Role airbyte-admin-role and RoleBindings airbyte-admin-binding exists.

Role is setting following rules : 

rules:
  - verbs:
      - get
      - list
      - watch
      - create
      - update
      - patch
      - delete
    apiGroups:
      - '*'
    resources:
      - jobs
      - pods
      - pods/log
      - pods/exec
      - pods/attach

All pods are using the airbyte-admin service account.

Relevant log output

2023-11-13 15:36:46,971 main ERROR Error processing element Then ([If: null]): CLASS_NOT_FOUND
2023-11-13 15:36:47,122 main ERROR Error processing element Else ([If: null]): CLASS_NOT_FOUND
2023-11-13 15:36:47,143 main ERROR Error processing element If ([Filters: null]): CLASS_NOT_FOUND
2023-11-13 15:36:48,135 main ERROR Console contains an invalid element or attribute "If"
2023-11-13 15:36:48,871 main INFO Loading mask data from '/seed/specs_secrets_mask.yaml

    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
        : airbyte-workers :

  Micronaut (v3.10.1)

2023-11-13 15:37:02 INFO i.m.c.e.DefaultEnvironment(<init>):167 - Established active environments: [k8s, cloud, gcp, control-plane]
2023-11-13 15:37:08 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):111 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
2023-11-13 15:37:15 INFO i.a.f.ConfigFileClient(<init>):105 - path /flags does not exist, will return default flag values
2023-11-13 15:37:23 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable METRIC_CLIENT: ''
2023-11-13 15:37:23 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable METRIC_CLIENT: ''
2023-11-13 15:37:23 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`. 
2023-11-13 15:37:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2023-11-13 15:37:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-11-13 15:37:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2023-11-13 15:37:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-11-13 15:37:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-11-13 15:37:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-11-13 15:37:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-11-13 15:37:27 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):148 - Waiting for temporal server...
2023-11-13 15:37:27 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):159 - Waiting for namespace default to be initialized in temporal...
2023-11-13 15:37:33 INFO i.t.s.WorkflowServiceStubsImpl(<init>):86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal:7233}}
2023-11-13 15:37:40 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):176 - Temporal namespace default initialized!
2023-11-13 15:37:40 INFO i.a.w.ApplicationInitializer(initializeCommonDependencies):167 - Initializing common worker dependencies.
2023-11-13 15:37:40 INFO i.a.w.ApplicationInitializer(configureTemporal):322 - Configuring Temporal....
2023-11-13 15:37:41 INFO i.a.c.t.TemporalUtils(configureTemporalNamespace):123 - Workflow execution TTL already set for namespace default. Remains unchanged as: 30 days
2023-11-13 15:37:46 INFO i.a.w.ApplicationInitializer(registerWorkerFactory):182 - Registering worker factories....
2023-11-13 15:37:47 INFO i.a.w.ApplicationInitializer(registerGetSpec):278 - Get Spec Workflow registered.
2023-11-13 15:37:47 INFO i.a.w.ApplicationInitializer(registerCheckConnection):231 - Check Connection Workflow registered.
2023-11-13 15:37:47 INFO i.a.w.ApplicationInitializer(registerDiscover):264 - Discover Workflow registered.
2023-11-13 15:37:47 INFO i.a.w.ApplicationInitializer(registerSync):291 - Registering sync workflow for task queue 'SYNC'...
2023-11-13 15:37:47 INFO i.a.w.ApplicationInitializer(registerSync):301 - Sync Workflow registered.
2023-11-13 15:37:47 INFO i.a.w.ApplicationInitializer(registerConnectionManager):247 - Connection Manager Workflow registered.
2023-11-13 15:37:47 INFO i.a.w.ApplicationInitializer(registerNotification):209 - registering new notification workflow
2023-11-13 15:37:47 INFO i.a.w.ApplicationInitializer(onApplicationEvent):149 - Starting worker factory...
2023-11-13 15:37:47 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="DISCOVER_SCHEMA", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:47 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Local Activity Poller taskQueue="DISCOVER_SCHEMA", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:47 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="DISCOVER_SCHEMA", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:47 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="NOTIFY", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Local Activity Poller taskQueue="NOTIFY", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="NOTIFY", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="CONNECTION_UPDATER", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Local Activity Poller taskQueue="CONNECTION_UPDATER", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="CONNECTION_UPDATER", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="CHECK_CONNECTION", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Local Activity Poller taskQueue="CHECK_CONNECTION", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="CHECK_CONNECTION", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="SYNC", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Local Activity Poller taskQueue="SYNC", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="SYNC", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="GET_SPEC", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Local Activity Poller taskQueue="GET_SPEC", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="GET_SPEC", namespace="default", identity=1@airbyte-worker-599c544f8d-s624d}
2023-11-13 15:37:48 INFO i.a.w.ApplicationInitializer(onApplicationEvent):152 - Application initialized.
2023-11-13 15:37:48 INFO i.m.r.Micronaut(lambda$start$2):98 - Startup completed in 54633ms. Server Running: http://airbyte-worker-599c544f8d-s624d:9000
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
Using cache monitor: TimePeriodBasedBufferMonitor(periodInSeconds: 60)
Registering AWS S3 publish helper -> S3 configuration (airbyte-dev-logs:job-logging/workspace/c7de9faa-b818-499d-81b2-d94d2789406d/0/logs.log in region null; compressed: false)
Log4j2Appender says: Cloud storage job log path: /workspace/c7de9faa-b818-499d-81b2-d94d2789406d/0/logs.log
2023-11-13 15:38:20 INFO i.a.w.t.TemporalAttemptExecution(get):126 - Cloud storage job log path: /workspace/c7de9faa-b818-499d-81b2-d94d2789406d/0/logs.log
Log4j2Appender says: Executing worker wrapper. Airbyte version: 0.50.34
2023-11-13 15:38:20 INFO i.a.w.t.TemporalAttemptExecution(get):129 - Executing worker wrapper. Airbyte version: 0.50.34
Log4j2Appender says: Attempt 0 to save workflow id for cancellation
2023-11-13 15:38:20 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):290 - Attempt 0 to save workflow id for cancellation
Collecting content into /tmp/toBePublished15359002390331363271.tmp before uploading.
Log4j2Appender says: 
2023-11-13 15:38:20 INFO i.a.c.i.LineGobbler(voidCall):149 - 
Log4j2Appender says: ----- START CHECK -----
2023-11-13 15:38:20 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
Log4j2Appender says: 
2023-11-13 15:38:20 INFO i.a.c.i.LineGobbler(voidCall):149 - 
Publishing to S3 (bucket=airbyte-dev-logs; key=job-logging/workspace/c7de9faa-b818-499d-81b2-d94d2789406d/0/logs.log/20231113153820_airbyte-worker-599c544f8d-s624d_66df8f403f444583be7549d8d22e7605):
Log4j2Appender says: Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
Log4j2Appender says: Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
Log4j2Appender says: Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
Log4j2Appender says: Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
Log4j2Appender says: Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
Log4j2Appender says: Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
Log4j2Appender says: Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
Log4j2Appender says: Attempting to start pod = e-mongodb-v2-check-c7de9faa-b818-499d-81b2-d94d2789406d-0-bidsd for airbyte/source-mongodb-v2:1.0.8 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@6bb9f226[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@63b480d3[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@67c4d3e4[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@7ff0f46[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@7ff0f46[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null
2023-11-13 15:38:20 INFO i.a.w.p.KubeProcessFactory(create):128 - Attempting to start pod = e-mongodb-v2-check-c7de9faa-b818-499d-81b2-d94d2789406d-0-bidsd for airbyte/source-mongodb-v2:1.0.8 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@6bb9f226[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@63b480d3[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@67c4d3e4[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@7ff0f46[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@7ff0f46[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null
Log4j2Appender says: e-mongodb-v2-check-c7de9faa-b818-499d-81b2-d94d2789406d-0-bidsd stdoutLocalPort = 9024
2023-11-13 15:38:20 INFO i.a.w.p.KubeProcessFactory(create):132 - e-mongodb-v2-check-c7de9faa-b818-499d-81b2-d94d2789406d-0-bidsd stdoutLocalPort = 9024
Log4j2Appender says: e-mongodb-v2-check-c7de9faa-b818-499d-81b2-d94d2789406d-0-bidsd stderrLocalPort = 9025
2023-11-13 15:38:20 INFO i.a.w.p.KubeProcessFactory(create):135 - e-mongodb-v2-check-c7de9faa-b818-499d-81b2-d94d2789406d-0-bidsd stderrLocalPort = 9025
Log4j2Appender says: Using default value for environment variable SYNC_JOB_INIT_RETRY_TIMEOUT_MINUTES: '5'
2023-11-13 15:38:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SYNC_JOB_INIT_RETRY_TIMEOUT_MINUTES: '5'
Log4j2Appender says: Creating stdout socket server...
2023-11-13 15:38:20 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):672 - Creating stdout socket server...
Log4j2Appender says: Creating stderr socket server...
2023-11-13 15:38:20 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):690 - Creating stderr socket server...
Log4j2Appender says: Creating pod e-mongodb-v2-check-c7de9faa-b818-499d-81b2-d94d2789406d-0-bidsd...
2023-11-13 15:38:20 INFO i.a.w.p.KubePodProcess(<init>):597 - Creating pod e-mongodb-v2-check-c7de9faa-b818-499d-81b2-d94d2789406d-0-bidsd...
java.net.SocketException: Socket closed
    at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:675)
    at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:748)
    at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:690)
    at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:655)
Log4j2Appender says: Unexpected: PodDefinition is null when closing in KubePodProcess. Log an error to help debugging but not blocking the process.
    at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:631)
2023-11-13 15:38:31 ERROR i.a.w.p.KubePodProcess(close):815 - Unexpected: PodDefinition is null when closing in KubePodProcess. Log an error to help debugging but not blocking the process.
    at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:588)
    at java.base/java.net.ServerSocket.accept(ServerSocket.java:546)
    at io.airbyte.workers.process.KubePodProcess.lambda$setupStdOutAndStdErrListeners$10(KubePodProcess.java:691)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1589)
java.net.SocketException: Socket closed
    at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:675)
    at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:748)
    at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:690)
    at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:655)
    at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:631)
    at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:588)
    at java.base/java.net.ServerSocket.accept(ServerSocket.java:546)
    at io.airbyte.workers.process.KubePodProcess.lambda$setupStdOutAndStdErrListeners$9(KubePodProcess.java:673)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1589)
Log4j2Appender says: Unexpected error while checking connection: 
2023-11-13 15:38:31 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):133 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: Failed to create pod for check step
    at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:188) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:143) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:133) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.34.jar:?]
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:118) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1589) ~[?:?]
  Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129) ~[kubernetes-client-api-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122) ~[kubernetes-client-api-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.utils.internal.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:51) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:310) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:93) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:1077) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:93) ~[kubernetes-client-6.5.1.jar:?]
    at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:600) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:184) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    ... 20 more
Caused by: java.io.IOException: timeout
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:543) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:566) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:350) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:707) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:93) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.utils.internal.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:51) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:310) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:93) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:1077) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:93) ~[kubernetes-client-6.5.1.jar:?]
    at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:600) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:184) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    ... 20 more
Caused by: java.net.SocketTimeoutException: timeout
    at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.kt:675) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.kt:684) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.kt:143) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.kt:96) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) ~[okhttp-4.10.0.jar:?]
    ... 3 more
2023-11-13 15:38:31 INFO i.a.c.t.HeartbeatUtils(withBackgroundHeartbeat):64 - Stopping temporal heartbeating...
2023-11-13 15:38:31 INFO i.a.c.t.HeartbeatUtils(withBackgroundHeartbeat):73 - Temporal heartbeating stopped.
2023-11-13 15:38:31 WARN i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=e702b029-9dc7-3920-8a16-88c6c5250d8b, activityType=RunWithJobOutput, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
    at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:62) ~[io.airbyte-airbyte-commons-temporal-core-0.50.34.jar:?]
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:118) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
    at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:57) ~[temporal-serviceclient-1.17.0.jar:?]
    at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:461) ~[temporal-sdk-1.17.0.jar:?]
    at io.temporal.activity.Activity.wrap(Activity.java:52) ~[temporal-sdk-1.17.0.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:139) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:133) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.34.jar:?]
    ... 14 more
Caused by: io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:135) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:133) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.34.jar:?]
    ... 14 more
Caused by: io.airbyte.workers.exception.WorkerException: Failed to create pod for check step
    at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:188) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:143) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:133) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.34.jar:?]
    ... 14 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129) ~[kubernetes-client-api-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122) ~[kubernetes-client-api-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.utils.internal.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:51) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:310) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:93) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:1077) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:93) ~[kubernetes-client-6.5.1.jar:?]
    at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:600) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:184) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:143) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:133) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.34.jar:?]
    ... 14 more
Caused by: java.io.IOException: timeout
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:543) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:566) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:350) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:707) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:93) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.utils.internal.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:51) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:310) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:93) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:1077) ~[kubernetes-client-6.5.1.jar:?]
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.createOrReplace(BaseOperation.java:93) ~[kubernetes-client-6.5.1.jar:?]
    at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:600) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:184) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:143) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.34.jar:?]
    at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:133) ~[io.airbyte-airbyte-workers-0.50.34.jar:?]
    at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.50.34.jar:?]
    ... 14 more
Caused by: java.net.SocketTimeoutException: timeout
    at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.kt:675) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.kt:684) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.kt:143) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.kt:96) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.10.0.jar:?]
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) ~[okhttp-4.10.0.jar:?]
    ... 3 more
Log4j2Appender says: 
2023-11-13 15:38:31 INFO i.a.c.i.LineGobbler(voidCall):149 - 
Log4j2Appender says: ----- END CHECK -----
2023-11-13 15:38:31 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
Log4j2Appender says: 
2023-11-13 15:38:31 INFO i.a.c.i.LineGobbler(voidCall):149 -
Collecting content into /tmp/toBePublished2845716829573760786.tmp before uploading.
Publishing to S3 (bucket=airbyte-dev-logs; key=job-logging/workspace/c7de9faa-b818-499d-81b2-d94d2789406d/0/logs.log/20231113153920_airbyte-worker-599c544f8d-s624d_66df8f403f444583be7549d8d22e7605):
Kamaradeivanov commented 12 months ago

Same behaviour with chart in version 0.45.31 and app in version 0.43.1. I try an old one just in case.

Kamaradeivanov commented 12 months ago

Problem solved !

One of my mutating webhooks took almost 10 seconds to answer, so each create call to the api-server took more than 10sec to answer, but airbyte-worker has a timeout of 10sec.

To find the problem I've checked the log of the api-server on the GKE console. Latency appear in labels : 

apiserver.latency.k8s.io/serialize-response-object: "6.336689ms"
apiserver.latency.k8s.io/mutating-webhook: "9.995078933s"
apiserver.latency.k8s.io/total: "10.007804155s"
apiserver.latency.k8s.io/response-write: "824ns"
edfincham commented 10 months ago

@Kamaradeivanov I have exactly the same issue. Can you provide some detail on how you resolved it?

Kamaradeivanov commented 10 months ago

Hi @edfincham you should check your api-server to see if there is no timeout request. If you are on a managed kubernetes ask your cloud provider, personally I was on GKE and logs are available through the console.

You can also edit all your validatingwebhookconfigurations and mutatingwebhookconfigurations to reduce the timeoutSeconds below 10 (because airbyte have harcoded the timeout to 10s)

kubectl edit mutatingwebhookconfigurations
kubectl edit validatingwebhookconfigurations
edfincham commented 10 months ago

mutatingwebhookconfigurations

Worked a treat - thank you :slightly_smiling_face: