search

istio / old_issues_repo

Deprecated issue-tracking repo, please post new issues or feature requests to istio/istio instead.
37 stars 9 forks source link

High latency at scale #390

Closed anthonymorrisjohnson closed 6 years ago

anthonymorrisjohnson commented 6 years ago

BUG

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?: Yes

Bug: Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version 0.8
kubectl version 0.10.1

Is Istio Auth enabled or not ? helm template - no auth: helm template install/kubernetes/helm/istio --name istio --namespace istio-system --set sidecarInjectorWebhook.enabled=false > $HOME/istio.yaml

What happened: the system works fine - at a low % of live traffic, but we start seeing incredibly high latency at a certain % of traffic (100ms -> 400ms plus a bunch of timeouts) the supporting pods are over-provisioned for the amount of traffic

In prometheus, we start seeing failures on telemetry and increase in latency for istio policy. Do we need to scale these out?

What you expected to happen:

How to reproduce it:

Feature Request: N

Describe the feature:

screen shot 2018-06-13 at 6 42 18 pm
anthonymorrisjohnson commented 6 years ago

ok - it looks like this was the mixer getting maxed out. I scaled it out 10x, and that seems to have solved the issue. (although scaling out telemetry or the policy pod seems to entail some high latency)