Closed crhuber closed 4 years ago
crhuber
commented
5 years ago I was able to work around this issue by setting this annotation on the deployment:
traffic.sidecar.istio.io/excludeOutboundPorts: "5672"However, this is not a permanent solution
rshriram
commented
5 years ago Oddly, When we run a second process of the rabbitmq consumer within the original container that had connectivity problems while the sidecar is running, we are then able to connect and consume messages.
does the consumer talk to itself by any chance? the above observation indicates that connectivity is working as intended.. it could also be that the consumer is racing with pilot [consumer calls out before pilot sends config or something strange of that sort]
crhuber
commented
5 years ago @rshriram Yes, we found it to be a race condition where the consumer is calling out before pilot has sent the config. The consumer didnt have any retry logic and didnt handle the exception so it was difficult to troubleshoot. But ultimately we found that by making the application exit out on exception and allowing the container to be recreated fixed the problem.
Is there a better approach to handling these race conditions?
howardjohn
commented
5 years ago We are working on improving the startup ordering problem -- see https://github.com/istio/istio/issues/11130
istio-policy-bot
commented
4 years ago 🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2019-07-31. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.
Created by the issue and PR lifecycle manager.
Bug description
We have a number of rabbitmq consumer pods with Istio sidecar running. These pods should consume messages from rabbitmq running as a hosted service via cloudamqp. As soon as the pod starts the consumer container never receives any messages even though the queue has messages waiting to be consumed. When we turn off sidecar injection, messages get consumed as expected. Connectivity to cloudampq seems to be broken when Istio is running.
Oddly, When we run a second process of the rabbitmq consumer within the original container that had connectivity problems while the sidecar is running, we are then able to connect and consume messages.
To troubleshoot we created a ServiceEntry resource but it did not seem to have an impact.
I confirmed traffic policy is ALLOW_ANY
Here are stats from the sidecar:
Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure [ ] Docs [ ] Installation [x ] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure
Expected behavior
Pods that connect to rabbitmq with Istio running have no connectivity issues
Steps to reproduce the bug
Version (include the output of
istioctl version --remoteandkubectl version)client version: 1.2.0 citadel version: 1.2.0 galley version: 1.2.0 ingressgateway version: 1.2.0 policy version: 1.2.0 sidecar-injector version: 1.2.0 telemetry version: 1.2.0
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-eks-c57ff8", GitCommit:"c57ff8e35590932c652433fab07988da79265d5b", GitTreeState:"clean", BuildDate:"2019-06-07T20:43:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
How was Istio installed?
Helm chart
Environment where bug was observed (cloud vendor, OS, etc)
Amazon AMI, Amazon EKS
Additionally, please consider attaching a cluster state archive by attaching the dump file to this issue.