Istio Sidecar Failures

[marcosmarxm]

This Github issue is synchronized with Zendesk:

Ticket ID: #1686 Priority: normal Group: Community Assistance Engineer Assignee: Nataly Merezhuk

Original ticket description:

Re-opening this for visibility:

• Is this your first time deploying Airbyte: No
• OS Version / Instance: Kubernetes on EKS
• Memory / Disk: 64GB+ / gp3 volume of the manifests’ default sizes.
• Deployment: Kubernetes
• Airbyte Version: 0.35.62-alpha
• Source name/version: N/A
• Destination name/version: N/A
• Step: Trying to add a source or destination, or update a connector.
• Description: I’ve deployed to an EKS cluster that’s using the istio service mesh, and am running into problems when adding an initial source connector. The source connector pod that’s created errs, and the pod logs show:

Using existing AIRBYTE_ENTRYPOINT: /airbyte/base.sh
Waiting on CHILD_PID 7
PARENT_PID: 1
Heartbeat to worker failed, exiting…
received ABRT

–
I tried today without the Istio sidecars, and things worked without issue.

The manifests I used to deploy istio and then airbyte are linked in a comment in the original topic: Airbyte issues with Istio sidecar

I received a DM on Slack from another community member stating she was having the same issue, but I haven’t troubleshot further since posting.

[Discourse post]

[marcosmarxm]

Comment made from Zendesk by Nataly Merezhuk on 2022-07-23 at 01:28:

Hi @bradfair, I am not very familiar with Istio sidecars, so I'll go ahead and ask for input from my team. While I do that, would you be able to provide the full logs you're getting? Is it possible to tag the other community member, so we can see what errors she is getting as well? Thank you!

[marcosmarxm]

Comment made from Zendesk by Marcos Marx on 2022-07-25 at 10:10:

Hello,
I’m the other member.

I’m running Airbyte v0.39.37-alpha on GKE.
If I deactivate istio, it works.

Here are some logs I had on my side for a BigQuery destination check test when istio is enabled.
Note that I had the same kind of error trying other checks for a mysql source.

Job logs:
connection_check_job.txt (10.0 KB)

Job pod logs by container:
main:
Started at 16:34:37, Ended at 16:34:43 (Error 1)

Using existing AIRBYTE_ENTRYPOINT: /airbyte/base.sh
Waiting on CHILD_PID 8
PARENT_PID: 1
Heartbeat to worker failed, exiting...
received ABRT

relay-stdout:
Started at 16:34:38, Ended at 16:34:40 (Error 1)

2022/07/22 14:34:39 socat[9] N reading from and writing to stdio
2022/07/22 14:34:39 socat[9] N opening connection to AF=2 10.208.3.92:9024
2022/07/22 14:34:39 socat[9] W recvmsg(): Resource temporarily unavailable
2022/07/22 14:34:39 socat[9] E connect(5, AF=2 10.208.3.92:9024, 16): Connection refused
2022/07/22 14:34:39 socat[9] N exit(1)

relay-stderr:
Started at 16:34:39, Ended at 16:34:39 (Error 1)

2022/07/22 14:34:39 socat[8] N reading from and writing to stdio
2022/07/22 14:34:39 socat[8] N opening connection to AF=2 10.208.3.92:9025
2022/07/22 14:34:39 socat[8] W recvmsg(): Resource temporarily unavailable
2022/07/22 14:34:39 socat[8] E connect(5, AF=2 10.208.3.92:9025, 16): Connection refused
2022/07/22 14:34:39 socat[8] N exit(1)

call-heartbeat-server: no logs
Started at 16:34:42, Ended at 16:34:42 (Error 7)

istio-proxy:
Started at 16:34:42, Running
istio_logs.txt (30.9 KB)

[Discourse post]

[marcosmarxm]

Comment made from Zendesk by Nataly Merezhuk on 2022-07-26 at 18:18:

Hi @mcGIh, thank you for all the info! This looks like a long-running issue that hasn't been resolved, so I've gone ahead and made a GitHub issue for it. I'll triage it to correct team, and if you don't mind let's continue the discussion there:
https://github.com/airbytehq/airbyte/issues/15043

[evantahler]

Re-tagging for team/infra

[bradfair]

I'm running into this issue right now while preparing for a demo, so I thought I'd check in and see if it's on anyone's to-do list yet.

[c-p-b]

@bradfair Can you verify if this is still an issue? We would like some additional details on our end if so

[xorl]

Seeing this myself on an IPv6 Kube cluster:

2023/06/07 20:13:44 socat[9] N reading from and writing to stdio
2023/06/07 20:13:44 socat[9] E TCP: wrong number of parameters (9 instead of 2)
2023/06/07 20:13:44 socat[9] N exit(1)

[nstankov-bg]

I am seeing this in a vanilla K3S cluster.

Works perfectly well w/o ISTIO

Activity with activityType='RunWithJobOutput' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=ecf47afd-36ac-35c9-895c-d1b18539ad6a, identity='1@airbyte-full-worker-6dc6d494cd-tx4bd', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED

It happened with an existing source.

[octavia-squidington-iii]

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

[nstankov-bg]

Bump