search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.82k stars 4.06k forks source link

Airbyte installation on eks fails with timeout error #17096

Open US-Thakur opened 2 years ago

US-Thakur commented 2 years ago
## Environment - **Airbyte version**: Not version specific but I tried using 0.46.0, 0.41.0 - **OS Version / Instance**: EKS node running Amazon Linux 2 kernel-5.4.196-108.356.amzn2.x86_64 - **Deployment**: Kubernetes deploy env - **Source Connector and version**: (if applicable example Salesforce 0.2.3)

client.go:299: [debug] Starting delete for "airbyte-admin" ServiceAccount client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-airbyte-env" ConfigMap client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-airbyte-secrets" Secret client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-db" Deployment client.go:328: [debug] deployments.apps "airbyte-db" not found client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-db-svc" Service client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-minio" Deployment client.go:328: [debug] deployments.apps "airbyte-minio" not found client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-minio-pv-claim" PersistentVolumeClaim client.go:328: [debug] persistentvolumeclaims "airbyte-minio-pv-claim" not found client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-minio-svc" Service client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-postgresql" Secret client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-postgresql" Service client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-postgresql" StatefulSet client.go:328: [debug] statefulsets.apps "airbyte-postgresql" not found client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-postgresql-headless" Service client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-volume-db" PersistentVolumeClaim client.go:328: [debug] persistentvolumeclaims "airbyte-volume-db" not found client.go:128: [debug] creating 1 resource(s) client.go:299: [debug] Starting delete for "airbyte-airbyte-bootloader" Pod client.go:328: [debug] pods "airbyte-airbyte-bootloader" not found client.go:128: [debug] creating 1 resource(s) client.go:528: [debug] Watching for changes to Pod airbyte-airbyte-bootloader with timeout of 5m0s client.go:556: [debug] Add/Modify event for airbyte-airbyte-bootloader: ADDED client.go:615: [debug] Pod airbyte-airbyte-bootloader pending client.go:556: [debug] Add/Modify event for airbyte-airbyte-bootloader: MODIFIED client.go:615: [debug] Pod airbyte-airbyte-bootloader pending client.go:556: [debug] Add/Modify event for airbyte-airbyte-bootloader: MODIFIED client.go:615: [debug] Pod airbyte-airbyte-bootloader pending client.go:556: [debug] Add/Modify event for airbyte-airbyte-bootloader: MODIFIED client.go:617: [debug] Pod airbyte-airbyte-bootloader running client.go:556: [debug] Add/Modify event for airbyte-airbyte-bootloader: MODIFIED client.go:617: [debug] Pod airbyte-airbyte-bootloader running client.go:556: [debug] Add/Modify event for airbyte-airbyte-bootloader: MODIFIED client.go:617: [debug] Pod airbyte-airbyte-bootloader running Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition helm.go:88: [debug] failed pre-install: timed out waiting for the condition INSTALLATION FAILED main.newInstallCmd.func2 helm.sh/helm/v3/cmd/helm/install.go:127 github.com/spf13/cobra.(Command).execute github.com/spf13/cobra@v1.2.1/command.go:856 github.com/spf13/cobra.(Command).ExecuteC github.com/spf13/cobra@v1.2.1/command.go:974 github.com/spf13/cobra.(*Command).Execute github.com/spf13/cobra@v1.2.1/command.go:902 main.main helm.sh/helm/v3/cmd/helm/helm.go:87 runtime.main runtime/proc.go:225 runtime.goexit runtime/asm_amd64.s:1371

kubectl get pods -n airbyte NAME READY STATUS RESTARTS AGE airbyte-airbyte-bootloader 1/2 NotReady 0 53m airbyte-db-5965d9c677-bhgkz 2/2 Running 1 55m airbyte-minio-797fd9d6cf-vnnzg 2/2 Running 0 52m airbyte-postgresql-0 2/2 Running 0 52m


## Current Behavior
Airbyte installation times out with error mentioned above. Bootloader container exits successfully as per its log. I see this healthcheck event for the bootloader container though:

Readiness probe failed: Get "http://192.168.216.90:15021/healthz/ready": dial tcp 192.168.216.90:15021: connect: connection refused Source kubelet ip-10-202-39-2.ec2.internal Count 1 Sub-object spec.containers{istio-proxy} Last seen 2022-09-23T19:24:45+05:30



## Expected Behavior
Airbyte deployment should complete successfully
## Logs
<!--
Please upload the complete log files from the failing operation. 

We ask for complete log files because these files capture essential metadata about your environment.
The team will read the full stack trace of errors and try to find the root cause of the error. 
For a sync failure, you can download the log in the sync page, and for other errors please download the server and scheduler logs in the Settings Page -> Configuration.
You can remove this after uploading the file.
-->

## Steps to Reproduce
1. helm dependency build --debug -n airbyte 
2. helm install airbyte -n airbyte . --debug
bdashrad commented 2 years ago

I have the same behavior with bootloader during the helm install on GKE as well.

EDIT: I realized my problem only happens when running a sidecar container (in this case google cloud-sql-proxy), because the sidecar doesn't exit when the main pod finishes. Perhaps there is another approach for doing the initial migration, like an init container or a job?