search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.89k stars 4.08k forks source link

[destination-postgres] Sync failed after a reset #31112

Open ibitolamayowa opened 1 year ago

ibitolamayowa commented 1 year ago

Connector Name

postgres

Connector Version

3.1.3

What step the error happened?

None

Relevant information

We had recently installed a new version of airbyte 0.50.19. We perviously had been using 0.38.3-alpha. Our Data Engineer had sone a reset which succeeded and since then all out sync started failing.

Screenshot 2023-10-05 at 20 25 42

I looked into the logs and noticed that the error it shows it io.airbyte.workers.exception.WorkerException: Timed out waiting for [300000] milliseconds for [Pod] with name:[destination-postgres-write-13-2-xspet] in namespace [airbyte].

Screenshot 2023-10-05 at 20 30 35

I noticed some of the pods are orphaned and deleted immediately the sync failed too. Is there anyway i can solve this.

cc: @sh4sh

Relevant log output

at io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$0(BufferedReplicationWorker.java:153)
        at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:208)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: io.airbyte.workers.exception.WorkerException: Timed out waiting for [300000] milliseconds for [Pod] with name:[destination-postgres-write-13-2-xspet] in namespace [airbyte].
        at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:171)
        at io.airbyte.workers.process.AirbyteIntegrationLauncher.write(AirbyteIntegrationLauncher.java:256)
        at io.airbyte.workers.internal.DefaultAirbyteDestination.start(DefaultAirbyteDestination.java:89)
        at io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.java:130)
        ... 6 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [300000] milliseconds for [Pod] with name:[destination-postgres-write-13-2-xspet] in namespace [airbyte].
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:893)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:93)
        at io.airbyte.workers.process.KubePodProcess.waitForInitPodToRun(KubePodProcess.java:353)
        at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:600)</init>
        at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:167)
        ... 9 more
,retryable=<null>,timestamp=1696520159625,additionalProperties={}]],additionalProperties={}]</null>

Contribute

evantahler commented 5 months ago

re-routing to the deployments team