Closed apatwal-suki closed 2 weeks ago
@yanavlasov
I don't think exact balance helps here, as this is on the listener side for client connections. Each client connection will have its own independent request streams on the worker it lands at, spread across multiple connections to the backend. The more pods you have, the more independent instances of Envoy you have. I'm not sure if you can really guarantee this hard limits on a per-backend basis easily this way - I think you might have better luck with a more distributed approach, e.g. retries + outlier detection, custom load balancing that is aware of backend capacity directly, etc.
@htuch Thank you for the reply. I work with @apatwal-suki (OP).
If we use gRPC health check and mark the upstream pod as "unhealthy" by hosting gRPC health check endpoint and responding with non-200 status code once it has 2 connections, does that force envoy to route traffic to different pod on next incoming connection?
It will send to a different host (unless in panic mode), but I don't believe we do inline checks with every request, so the health check will be out of sync potentially and you might see failures here during periods of inconsistency.
Out of curiosity, if you have exactly 1 Envoy pod and 1 worker thread, with a round-robin policy, are you seeing the same behavior?
You might also want to share your /clusters
and /stats
dumps from the admin service to see what is going on.
It will send to a different host (unless in panic mode), but I don't believe we do inline checks with every request, so the health check will be out of sync potentially and you might see failures here during periods of inconsistency.
Out of curiosity, if you have exactly 1 Envoy pod and 1 worker thread, with a round-robin policy, are you seeing the same behavior?
You might also want to share your
/clusters
and/stats
dumps from the admin service to see what is going on.
We have to try that. We tried with an envoy pod that had 8 cores so 8 worker threads were spawned and we noticed same behaviour: requests started failing after a certain amount. Exact configuration we tried was 40 requests and I think 35-37 of them succeed and others fail (as if envoy rejects them - they don't land on upstream service).
cc @apatwal-suki to add more here.
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.
This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.
FWIW, this turned out to be a mistake in configuration where we didn't mark the upstream nodes as "headless" while using GKE. Once did the same, kubernetes network proxy stopped interfering and envoy load balancing worked properly. After that selecting https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/load_balancers#weighted-least-request did the trick for us.
Just in case it helps anyone else.
cc @htuch @apatwal-suki
Title: envoy not load balancing grpc connections effectively
Description:
Repro steps:
Envoy deployment:
With the above config and doing a load-test, even when I have 20 pods of
ms-dummy-asr-v2-ambient
(which should be able to support 40 concurrent requests, as in our application, we have a hard limit of 2 connections per pod), I see connection failures for a lot of requests(this is not a fixed number). On increasing the CPU or the number of envoy pods however, the number of failures does decrease. However, this does not follow any deterministic pattern. For instance, with 1 envoy pod, around 26 requests return a success; with 5 envoy pods, around 37 requests return a success.From the above observation, it does seem like something to do with load-balancing among the worker threads. As you can see from the above config, I do have exact_balance config in place. What am I missing here?