search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.52k stars 4k forks source link

Airbyte OSS via Helm on EKS: sync jobs are failing constantly #35346

Closed alexremn closed 7 months ago

alexremn commented 7 months ago

Helm Chart Version

0.53.137

What step the error happened?

During the Sync

Revelant information

We have Airbyte OSS deployed via Helm on EKS with Karpenter. Each try of sync is ending with failure due to timeout for jobs waiting - using karpenter to dynamically spin up nodes during the pod creation.

Relevant log output

2024-02-16 10:40:38 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=4, totalCompleteFailures=4, successivePartialFailures=0, totalPartialFailures=0)
2024-02-16 10:40:38 platform > Backing off for: 4 minutes 30 seconds.
2024-02-16 10:45:09 platform > Cloud storage job log path: /workspace/23/4/logs.log
2024-02-16 10:45:09 platform > Executing worker wrapper. Airbyte version: 0.50.48
2024-02-16 10:45:09 platform > Attempt 0 to save workflow id for cancellation
2024-02-16 10:45:09 platform > 
2024-02-16 10:45:09 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:09 platform > ----- START CHECK -----
2024-02-16 10:45:09 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:09 platform > 
2024-02-16 10:45:09 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:09 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:09 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-16 10:45:09 platform > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-02-16 10:45:09 platform > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-02-16 10:45:09 platform > Attempting to start pod = source-bing-ads-check-23-4-gjono for airbyte/source-bing-ads:2.1.4 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@1c01df19[cpuRequest=1000m,cpuLimit=1000m,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@74a8b128[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@371a3e30[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@7d6df4b1[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@7d6df4b1[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@4ce2b5b6[hosts=[bingads.microsoft.com, login.microsoftonline.com, login.live.com, login.windows-ppe.net, ads.microsoft.com, api.ads.microsoft.com, clientcenter.api.bingads.microsoft.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-02-16 10:45:09 platform > source-bing-ads-check-23-4-gjono stdoutLocalPort = 9006
2024-02-16 10:45:09 platform > source-bing-ads-check-23-4-gjono stderrLocalPort = 9007
2024-02-16 10:45:09 platform > Creating stdout socket server...
2024-02-16 10:45:09 platform > Creating stderr socket server...
2024-02-16 10:45:09 platform > Creating pod source-bing-ads-check-23-4-gjono...
2024-02-16 10:45:09 platform > Waiting for init container to be ready before copying files...
2024-02-16 10:45:10 platform > Init container ready..
2024-02-16 10:45:10 platform > Copying files...
2024-02-16 10:45:10 platform > Uploading file: source_config.json
2024-02-16 10:45:10 platform > kubectl cp /tmp/bad6058a-fca5-466b-8205-d0d5cf8cf770/source_config.json airbyte/source-bing-ads-check-23-4-gjono:/config/source_config.json -c init --retries=3
2024-02-16 10:45:10 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:10 platform > kubectl cp complete, closing process
2024-02-16 10:45:10 platform > Uploading file: FINISHED_UPLOADING
2024-02-16 10:45:10 platform > kubectl cp /tmp/b1c73b9f-005c-4aaf-987c-caed0b86d480/FINISHED_UPLOADING airbyte/source-bing-ads-check-23-4-gjono:/config/FINISHED_UPLOADING -c init --retries=3
2024-02-16 10:45:10 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:10 platform > kubectl cp complete, closing process
2024-02-16 10:45:10 platform > Waiting until pod is ready...
2024-02-16 10:45:11 platform > Setting stdout...
2024-02-16 10:45:11 platform > Setting stderr...
2024-02-16 10:45:12 platform > Reading pod IP...
2024-02-16 10:45:12 platform > Pod IP: 10.10.2.170
2024-02-16 10:45:12 platform > Using null stdin output stream...
2024-02-16 10:45:12 platform > Reading messages from protocol version 0.2.0
2024-02-16 10:45:15 platform > Fetching access token ...
2024-02-16 10:45:15 platform > Check succeeded
2024-02-16 10:45:16 platform > (pod: airbyte / source-bing-ads-check-23-4-gjono) - Closed all resources for pod
2024-02-16 10:45:16 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@1bd0c87b[status=succeeded,message=<null>,additionalProperties={}]
2024-02-16 10:45:16 platform > 
2024-02-16 10:45:16 platform > ----- END CHECK -----
2024-02-16 10:45:16 platform > 
2024-02-16 10:45:16 platform > Cloud storage job log path: /workspace/23/4/logs.log
2024-02-16 10:45:16 platform > Executing worker wrapper. Airbyte version: 0.50.48
2024-02-16 10:45:16 platform > Attempt 0 to save workflow id for cancellation
2024-02-16 10:45:16 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:16 platform > 
2024-02-16 10:45:16 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:16 platform > ----- START CHECK -----
2024-02-16 10:45:16 platform > 
2024-02-16 10:45:16 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:16 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:16 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-16 10:45:16 platform > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-02-16 10:45:16 platform > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-02-16 10:45:16 platform > Attempting to start pod = destination-redshift-check-23-4-vyski for airbyte/destination-redshift:2.1.7 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@175150d5[cpuRequest=1000m,cpuLimit=1000m,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@74a8b128[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@371a3e30[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@7d6df4b1[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@7d6df4b1[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null
2024-02-16 10:45:16 platform > destination-redshift-check-23-4-vyski stdoutLocalPort = 9008
2024-02-16 10:45:16 platform > destination-redshift-check-23-4-vyski stderrLocalPort = 9009
2024-02-16 10:45:16 platform > Creating stdout socket server...
2024-02-16 10:45:16 platform > Creating stderr socket server...
2024-02-16 10:45:16 platform > Creating pod destination-redshift-check-23-4-vyski...
2024-02-16 10:45:16 platform > Waiting for init container to be ready before copying files...
2024-02-16 10:45:19 platform > Init container ready..
2024-02-16 10:45:19 platform > Copying files...
2024-02-16 10:45:19 platform > Uploading file: source_config.json
2024-02-16 10:45:19 platform > kubectl cp /tmp/19d997f7-769d-4290-946d-7451aa85b212/source_config.json airbyte/destination-redshift-check-23-4-vyski:/config/source_config.json -c init --retries=3
2024-02-16 10:45:19 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:19 platform > kubectl cp complete, closing process
2024-02-16 10:45:19 platform > Uploading file: FINISHED_UPLOADING
2024-02-16 10:45:19 platform > kubectl cp /tmp/08dad9ac-ae71-4963-93b4-befde0bfb7fe/FINISHED_UPLOADING airbyte/destination-redshift-check-23-4-vyski:/config/FINISHED_UPLOADING -c init --retries=3
2024-02-16 10:45:19 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:19 platform > kubectl cp complete, closing process
2024-02-16 10:45:19 platform > Waiting until pod is ready...
2024-02-16 10:45:20 platform > Setting stdout...
2024-02-16 10:45:20 platform > Setting stderr...
2024-02-16 10:45:21 platform > Reading pod IP...
2024-02-16 10:45:21 platform > Pod IP: 10.10.2.108
2024-02-16 10:45:21 platform > Using null stdin output stream...
2024-02-16 10:45:21 platform > Reading messages from protocol version 0.2.0
2024-02-16 10:45:21 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-02-16 10:45:21 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-02-16 10:45:21 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-02-16 10:45:21 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.i.d.r.RedshiftDestination(main):79 - starting destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.b.IntegrationCliParser(parseOptions):126 - integration args: {check=null, config=source_config.json}
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.b.IntegrationRunner(runInternal):132 - Running integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.b.IntegrationRunner(runInternal):133 - Command: CHECK
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.b.IntegrationRunner(runInternal):134 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword groups - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword group - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.d.j.c.SwitchingDestination(check):56 - Using destination type: COPY_S3
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.b.s.SshTunnel(getInstance):252 - Starting connection with method: NO_TUNNEL
2024-02-16 10:45:22 platform > 2024-02-16 10:45:22 INFO i.a.c.i.d.s.S3DestinationConfig(createS3Client):239 - Creating S3 client...
2024-02-16 10:45:24 platform > 2024-02-16 10:45:24 INFO i.a.c.i.d.s.S3BaseChecks(testIAMUserHasListObjectPermission):131 - Started testing if IAM user can call listObjects on the destination bucket
2024-02-16 10:45:24 platform > 2024-02-16 10:45:24 INFO i.a.c.i.d.s.S3BaseChecks(testIAMUserHasListObjectPermission):134 - Finished checking for listObjects permission
2024-02-16 10:45:24 platform > 2024-02-16 10:45:24 INFO c.z.h.HikariDataSource(<init>):79 - HikariPool-1 - Starting...
2024-02-16 10:45:24 platform > 2024-02-16 10:45:24 WARN c.z.h.u.DriverDataSource(<init>):68 - Registered driver with driverClassName=com.amazon.redshift.jdbc.Driver was not found, trying direct instantiation.
2024-02-16 10:45:24 platform > 2024-02-16 10:45:24 INFO c.z.h.HikariDataSource(<init>):81 - HikariPool-1 - Start completed.
2024-02-16 10:45:26 platform > 2024-02-16 10:45:26 INFO i.a.i.d.r.u.RedshiftUtil(checkSvvTableAccess):43 - checking SVV_TABLE_INFO permissions
2024-02-16 10:45:27 platform > 2024-02-16 10:45:27 INFO i.a.c.d.j.DefaultJdbcDatabase(lambda$unsafeQuery$1):132 - closing connection
2024-02-16 10:45:27 platform > 2024-02-16 10:45:27 INFO c.z.h.HikariDataSource(close):349 - HikariPool-1 - Shutdown initiated...
2024-02-16 10:45:27 platform > 2024-02-16 10:45:27 INFO c.z.h.HikariDataSource(close):351 - HikariPool-1 - Shutdown completed.
2024-02-16 10:45:27 platform > 2024-02-16 10:45:27 INFO i.a.c.i.b.IntegrationRunner(runInternal):231 - Completed integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
2024-02-16 10:45:27 platform > 2024-02-16 10:45:27 INFO i.a.i.d.r.RedshiftDestination(main):81 - completed destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2024-02-16 10:45:28 platform > (pod: airbyte / destination-redshift-check-23-4-vyski) - Closed all resources for pod
2024-02-16 10:45:28 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@7afbc2c0[status=succeeded,message=<null>,additionalProperties={}]
2024-02-16 10:45:28 platform > 
2024-02-16 10:45:28 platform > ----- END CHECK -----
2024-02-16 10:45:28 platform > 
2024-02-16 10:45:29 platform > Cloud storage job log path: /workspace/23/4/logs.log
2024-02-16 10:45:29 platform > Executing worker wrapper. Airbyte version: 0.50.48
2024-02-16 10:45:29 platform > Attempt 0 to save workflow id for cancellation
2024-02-16 10:45:29 platform > Creating orchestrator-repl-job-23-attempt-4 for attempt number: 4
2024-02-16 10:45:29 platform > Successfully deleted all running pods for the connection!
2024-02-16 10:45:29 platform > Waiting for pod to be running...
2024-02-16 10:45:31 platform > Pod airbyte/orchestrator-repl-job-23-attempt-4 is running on 10.10.2.228
2024-02-16 10:45:31 platform > Uploading file: envMap.json
2024-02-16 10:45:31 platform > kubectl cp /tmp/82c17528-8c31-4a76-8dc6-1b5d5ecd6777/envMap.json airbyte/orchestrator-repl-job-23-attempt-4:/config/envMap.json -c init --retries=3
2024-02-16 10:45:31 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:31 platform > kubectl cp complete, closing process
2024-02-16 10:45:31 platform > Uploading file: application.txt
2024-02-16 10:45:31 platform > kubectl cp /tmp/aa26e9b2-57ff-4953-aa54-b70f971f74bf/application.txt airbyte/orchestrator-repl-job-23-attempt-4:/config/application.txt -c init --retries=3
2024-02-16 10:45:31 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:31 platform > kubectl cp complete, closing process
2024-02-16 10:45:31 platform > Uploading file: jobRunConfig.json
2024-02-16 10:45:31 platform > kubectl cp /tmp/9a342626-1bb3-4719-a3fa-c8a1767fa285/jobRunConfig.json airbyte/orchestrator-repl-job-23-attempt-4:/config/jobRunConfig.json -c init --retries=3
2024-02-16 10:45:31 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:31 platform > kubectl cp complete, closing process
2024-02-16 10:45:31 platform > Uploading file: destinationLauncherConfig.json
2024-02-16 10:45:31 platform > kubectl cp /tmp/93a00515-ee41-4326-b45d-1f549d52a695/destinationLauncherConfig.json airbyte/orchestrator-repl-job-23-attempt-4:/config/destinationLauncherConfig.json -c init --retries=3
2024-02-16 10:45:31 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:31 platform > kubectl cp complete, closing process
2024-02-16 10:45:31 platform > Uploading file: sourceLauncherConfig.json
2024-02-16 10:45:31 platform > kubectl cp /tmp/cffe93b2-41ca-4264-a142-23c66e298ed2/sourceLauncherConfig.json airbyte/orchestrator-repl-job-23-attempt-4:/config/sourceLauncherConfig.json -c init --retries=3
2024-02-16 10:45:32 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:32 platform > kubectl cp complete, closing process
2024-02-16 10:45:32 platform > Uploading file: input.json
2024-02-16 10:45:32 platform > kubectl cp /tmp/b10c6203-cb45-4418-ac2e-9da8838c259f/input.json airbyte/orchestrator-repl-job-23-attempt-4:/config/input.json -c init --retries=3
2024-02-16 10:45:32 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:32 platform > kubectl cp complete, closing process
2024-02-16 10:45:32 platform > Uploading file: KUBE_POD_INFO
2024-02-16 10:45:32 platform > kubectl cp /tmp/3cd6c461-bd7f-4a19-ad42-cd4ec5d1ad34/KUBE_POD_INFO airbyte/orchestrator-repl-job-23-attempt-4:/config/KUBE_POD_INFO -c init --retries=3
2024-02-16 10:45:32 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:32 platform > kubectl cp complete, closing process
2024-02-16 10:45:32 platform > Uploading file: FINISHED_UPLOADING
2024-02-16 10:45:32 platform > kubectl cp /tmp/f5501974-4c89-405b-aaed-7287b5c3226a/FINISHED_UPLOADING airbyte/orchestrator-repl-job-23-attempt-4:/config/FINISHED_UPLOADING -c init --retries=3
2024-02-16 10:45:32 platform > Waiting for kubectl cp to complete
2024-02-16 10:45:32 platform > kubectl cp complete, closing process
2024-02-16 10:45:37 INFO i.m.r.Micronaut(lambda$start$2):98 - Startup completed in 2593ms. Server Running: http://orchestrator-repl-job-23-attempt-4:9000
2024-02-16 10:45:40 replication-orchestrator > Writing async status INITIALIZING for KubePodInfo[namespace=airbyte, name=orchestrator-repl-job-23-attempt-4, mainContainerInfo=KubeContainerInfo[image=airbyte/container-orchestrator:0.50.48, pullPolicy=IfNotPresent]]...
2024-02-16 10:45:38 INFO i.a.f.ConfigFileClient(<init>):105 - path /flags does not exist, will return default flag values
2024-02-16 10:45:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):694 - Using default value for environment variable METRIC_CLIENT: ''
2024-02-16 10:45:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):694 - Using default value for environment variable METRIC_CLIENT: ''
2024-02-16 10:45:39 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`. 
2024-02-16 10:46:22 INFO i.a.a.SegmentAnalyticsClient(close):226 - Closing Segment analytics client...
2024-02-16 10:46:22 INFO i.a.a.BlockingShutdownAnalyticsPlugin(waitForFlush):281 - Waiting for Segment analytic client to flush enqueued messages...
2024-02-16 10:46:22 INFO i.a.a.BlockingShutdownAnalyticsPlugin(waitForFlush):293 - Segment analytic client flush complete.
2024-02-16 10:46:22 INFO i.a.a.SegmentAnalyticsClient(close):230 - Segment analytics client closed.  No new events will be accepted.
2024-02-16 10:45:40 replication-orchestrator > sourceLauncherConfig is: io.airbyte.persistence.job.models.IntegrationLauncherConfig@727dcc64[jobId=23,attemptId=4,connectionId=0e706487-7b0e-44e5-98b4-ba3fa3dfea44,workspaceId=edfa088e-e32a-462c-a6f6-07e7fc2cca36,dockerImage=airbyte/source-bing-ads:2.1.4,normalizationDockerImage=<null>,supportsDbt=false,normalizationIntegrationType=<null>,protocolVersion=Version{version='0.2.0', major='0', minor='2', patch='0'},isCustomConnector=false,allowedHosts=io.airbyte.config.AllowedHosts@49e62b17[hosts=[bingads.microsoft.com, login.microsoftonline.com, login.live.com, login.windows-ppe.net, ads.microsoft.com, api.ads.microsoft.com, clientcenter.api.bingads.microsoft.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}],additionalEnvironmentVariables=<null>,additionalLabels=<null>,additionalProperties={}]
2024-02-16 10:45:40 replication-orchestrator > Attempt 0 to get the source definition for feature flag checks
2024-02-16 10:45:40 replication-orchestrator > Attempt 0 to get the source definition
2024-02-16 10:45:41 replication-orchestrator > Concurrent stream read enabled? false
2024-02-16 10:45:41 replication-orchestrator > Setting up source...
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2024-02-16 10:45:41 replication-orchestrator > Setting up destination...
2024-02-16 10:45:41 replication-orchestrator > Setting up replication worker...
2024-02-16 10:45:41 replication-orchestrator > Running replication worker...
2024-02-16 10:45:41 replication-orchestrator > start sync worker. job id: 23 attempt id: 4
2024-02-16 10:45:41 replication-orchestrator > 
2024-02-16 10:45:41 replication-orchestrator > ----- START REPLICATION -----
2024-02-16 10:45:41 replication-orchestrator > 
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:41 replication-orchestrator > Running destination...
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-02-16 10:45:41 replication-orchestrator > Attempting to start pod = destination-redshift-write-23-4-haodd for airbyte/destination-redshift:2.1.7 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@3b724f73[cpuRequest=1000m,cpuLimit=1000m,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@419ce7a0[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@65ce152b[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@53d6003d[cpuRequest=0.1,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@7af6c4e3[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null
2024-02-16 10:45:41 replication-orchestrator > Attempting to start pod = source-bing-ads-read-23-4-qmdqm for airbyte/source-bing-ads:2.1.4 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@1307c9af[cpuRequest=1000m,cpuLimit=1000m,memoryRequest=2Gi,memoryLimit=2Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@419ce7a0[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@5a363cc0[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=null, stdOut=io.airbyte.config.ResourceRequirements@48ecca8a[cpuRequest=0.2,cpuLimit=1,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@49e62b17[hosts=[bingads.microsoft.com, login.microsoftonline.com, login.live.com, login.windows-ppe.net, ads.microsoft.com, api.ads.microsoft.com, clientcenter.api.bingads.microsoft.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-02-16 10:45:41 replication-orchestrator > destination-redshift-write-23-4-haodd stdoutLocalPort = 9879
2024-02-16 10:45:41 replication-orchestrator > source-bing-ads-read-23-4-qmdqm stdoutLocalPort = 9878
2024-02-16 10:45:41 replication-orchestrator > destination-redshift-write-23-4-haodd stderrLocalPort = 9880
2024-02-16 10:45:41 replication-orchestrator > source-bing-ads-read-23-4-qmdqm stderrLocalPort = 9877
2024-02-16 10:45:41 replication-orchestrator > Using default value for environment variable SYNC_JOB_INIT_RETRY_TIMEOUT_MINUTES: '5'
2024-02-16 10:45:41 replication-orchestrator > Creating stdout socket server...
2024-02-16 10:45:41 replication-orchestrator > Creating stdout socket server...
2024-02-16 10:45:41 replication-orchestrator > Creating stderr socket server...
2024-02-16 10:45:41 replication-orchestrator > Creating stderr socket server...
2024-02-16 10:45:41 replication-orchestrator > Creating pod source-bing-ads-read-23-4-qmdqm...
2024-02-16 10:45:41 replication-orchestrator > Creating pod destination-redshift-write-23-4-haodd...
2024-02-16 10:45:42 replication-orchestrator > Waiting for init container to be ready before copying files...
2024-02-16 10:45:42 replication-orchestrator > Waiting for init container to be ready before copying files...
2024-02-16 10:46:00 replication-orchestrator > Init container ready..
2024-02-16 10:46:00 replication-orchestrator > Copying files...
2024-02-16 10:46:00 replication-orchestrator > Uploading file: destination_config.json
2024-02-16 10:46:00 replication-orchestrator > kubectl cp /tmp/f1cc64a8-180c-43a9-823f-0eb7c2808c0b/destination_config.json airbyte/destination-redshift-write-23-4-haodd:/config/destination_config.json -c init --retries=3
2024-02-16 10:46:00 replication-orchestrator > Waiting for kubectl cp to complete
2024-02-16 10:46:00 replication-orchestrator > (pod: airbyte / destination-redshift-write-23-4-haodd) - Destroying Kube process.
2024-02-16 10:46:00 replication-orchestrator > (pod: airbyte / destination-redshift-write-23-4-haodd) - Closed all resources for pod
2024-02-16 10:46:00 replication-orchestrator > (pod: airbyte / destination-redshift-write-23-4-haodd) - Destroyed Kube process.
2024-02-16 10:46:22 replication-orchestrator > Init container ready..
2024-02-16 10:46:22 replication-orchestrator > Copying files...
2024-02-16 10:46:22 replication-orchestrator > Uploading file: source_config.json
2024-02-16 10:46:22 replication-orchestrator > kubectl cp /tmp/e0491f98-2d7d-4f2e-b308-caf1d46948be/source_config.json airbyte/source-bing-ads-read-23-4-qmdqm:/config/source_config.json -c init --retries=3
2024-02-16 10:46:22 replication-orchestrator > Waiting for kubectl cp to complete
2024-02-16 10:46:22 replication-orchestrator > (pod: airbyte / source-bing-ads-read-23-4-qmdqm) - Destroying Kube process.
2024-02-16 10:46:22 replication-orchestrator > (pod: airbyte / source-bing-ads-read-23-4-qmdqm) - Closed all resources for pod
2024-02-16 10:46:22 replication-orchestrator > (pod: airbyte / source-bing-ads-read-23-4-qmdqm) - Destroyed Kube process.
2024-02-16 10:46:22 replication-orchestrator > thread status... timeout thread: false , replication thread: true
2024-02-16 10:46:22 replication-orchestrator > sync summary: {
  "status" : "failed",
  "startTime" : 1708080341140,
  "endTime" : 1708080382146,
  "totalStats" : {
    "bytesEmitted" : 0,
    "destinationStateMessagesEmitted" : 0,
    "destinationWriteEndTime" : 0,
    "destinationWriteStartTime" : 1708080341147,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "recordsEmitted" : 0,
    "replicationEndTime" : 1708080382144,
    "replicationStartTime" : 1708080341140,
    "sourceReadEndTime" : 0,
    "sourceReadStartTime" : 1708080341148,
    "sourceStateMessagesEmitted" : 0
  },
  "streamStats" : [ ],
  "performanceMetrics" : {
    "processFromSource" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "readFromSource" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "processFromDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "writeToDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "readFromDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    }
  }
}
2024-02-16 10:46:22 replication-orchestrator > failures: [ {
  "failureOrigin" : "replication",
  "internalMessage" : "io.airbyte.workers.exception.WorkerException: Failed to create pod for write step",
  "externalMessage" : "Something went wrong during replication",
  "metadata" : {
    "attemptNumber" : 4,
    "jobId" : 23
  },
  "stacktrace" : "java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Failed to create pod for write step\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.kt:196)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$0(BufferedReplicationWorker.java:152)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:228)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: io.airbyte.workers.exception.WorkerException: Failed to create pod for write step\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:197)\n\tat io.airbyte.workers.process.AirbyteIntegrationLauncher.write(AirbyteIntegrationLauncher.java:264)\n\tat io.airbyte.workers.internal.DefaultAirbyteDestination.start(DefaultAirbyteDestination.java:101)\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.kt:194)\n\t... 6 more\nCaused by: java.lang.RuntimeException: java.io.IOException: kubectl cp failed with exit code 1\n\tat io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:356)\n\tat io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:655)\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:193)\n\t... 9 more\nCaused by: java.io.IOException: kubectl cp failed with exit code 1\n\tat io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:350)\n\t... 11 more\n",
  "timestamp" : 1708080360954
}, {
  "failureOrigin" : "replication",
  "internalMessage" : "io.airbyte.workers.exception.WorkerException: Failed to create pod for read step",
  "externalMessage" : "Something went wrong during replication",
  "metadata" : {
    "attemptNumber" : 4,
    "jobId" : 23
  },
  "stacktrace" : "java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Failed to create pod for read step\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startSource(ReplicationWorkerHelper.kt:214)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$1(BufferedReplicationWorker.java:153)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:228)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\nCaused by: io.airbyte.workers.exception.WorkerException: Failed to create pod for read step\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:197)\n\tat io.airbyte.workers.process.AirbyteIntegrationLauncher.read(AirbyteIntegrationLauncher.java:226)\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.start(DefaultAirbyteSource.java:84)\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startSource(ReplicationWorkerHelper.kt:212)\n\t... 6 more\nCaused by: java.lang.RuntimeException: java.io.IOException: kubectl cp failed with exit code 1\n\tat io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:356)\n\tat io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:655)\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:193)\n\t... 9 more\nCaused by: java.io.IOException: kubectl cp failed with exit code 1\n\tat io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:350)\n\t... 11 more\n",
  "timestamp" : 1708080382140
} ]
2024-02-16 10:46:22 replication-orchestrator > Returning output...
2024-02-16 10:46:22 replication-orchestrator > 
2024-02-16 10:46:22 replication-orchestrator > ----- END REPLICATION -----
2024-02-16 10:46:22 replication-orchestrator > 
2024-02-16 10:46:22 replication-orchestrator > Writing async status SUCCEEDED for KubePodInfo[namespace=airbyte, name=orchestrator-repl-job-23-attempt-4, mainContainerInfo=KubeContainerInfo[image=airbyte/container-orchestrator:0.50.48, pullPolicy=IfNotPresent]]...
2024-02-16 10:46:23 platform > State Store reports orchestrator pod orchestrator-repl-job-23-attempt-4 succeeded
2024-02-16 10:46:23 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=5, totalCompleteFailures=5, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 13 minutes 30 seconds
2024-02-16 10:46:23 platform > Failing job: 23, reason: Job failed after too many retries for connection 0e706487-7b0e-44e5-98b4-ba3fa3dfea44
marcosmarxm commented 7 months ago

Closed as duplicated of https://github.com/airbytehq/airbyte/discussions/35301. Let's try to find the root cause first there. If it's a bug and not a problem with configuration I'll transfer the discussion to issue.