[Need an investigation ] Airbyte is not stable in k8s

[sivankumar86]

Helm Chart Version

0.50.34

What step the error happened?

During the Sync

Revelant information

deployment : AWS EKS

Intermittently sync failed and raising p1 alerts in production. It is annoying .

Failure reason: An unknown failure occurred

c810ba10_3e93_4c4c_976f_8605746e4520_job_481711_attempt_2_txt.log

Relevant log output

2023-12-17 08:20:10 INFO i.a.w.t.TemporalAttemptExecution(get):126 - Cloud storage job log path: /workspace/481711/1/logs.log
2023-12-17 08:20:10 INFO i.a.w.t.TemporalAttemptExecution(get):129 - Executing worker wrapper. Airbyte version: 0.50.34
2023-12-17 08:20:10 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):290 - Attempt 0 to save workflow id for cancellation
2023-12-17 08:20:10 INFO i.a.w.s.LauncherWorker(run):193 - Creating orchestrator-repl-job-481711-attempt-1 for attempt number: 1
2023-12-17 08:20:10 WARN i.a.w.s.LauncherWorker(killRunningPodsForConnection):285 - There are currently running pods for the connection: [orchestrator-repl-job-481661-attempt-0]. Killing these pods to enforce one execution at a time.
2023-12-17 08:20:10 INFO i.a.w.s.LauncherWorker(killRunningPodsForConnection):288 - Attempting to delete pods: [orchestrator-repl-job-481661-attempt-0]
2023-12-17 08:20:10 INFO i.a.w.s.LauncherWorker(killRunningPodsForConnection):293 - Waiting for deletion...

[marcosmarxm]

Closing the issue as it doesn't have steps to reproduce.