mbrancato
commented
2 years ago As a follow-up, this is mainly overflowing the upstream_rq_pending_overflow value.
foo/myapp-00006.upstream_rq_pending_overflow: 17345There are also some upstream_cx_overflow and upstream_rq_pending_overflow values > 0 on HTTP/1.1 services, but they are fairly small.
The dynamic clusters are getting default circuit breaker values.
% curl -s localhost:9000/clusters | grep myapp
foo/myapp-00006::observability_name::foo/myapp-00006
foo/myapp-00006::default_priority::max_connections::1024
foo/myapp-00006::default_priority::max_pending_requests::1024
foo/myapp-00006::default_priority::max_requests::1024
foo/myapp-00006::default_priority::max_retries::3
foo/myapp-00006::high_priority::max_connections::1024
foo/myapp-00006::high_priority::max_pending_requests::1024
foo/myapp-00006::high_priority::max_requests::1024
foo/myapp-00006::high_priority::max_retries::3
foo/myapp-00006::added_via_api::true
...And looking where new clusters appear to be created via the API, there is no circuit breaker config: https://github.com/knative-sandbox/net-kourier/blob/969ae748c33768049c72b10ae9fb0cd0235fe6ff/pkg/envoy/api/cluster.go#L39-L52
As my testing has shown, the requests from multiple clients still triggers this overflow. Otherwise, I could create some transport pool to spread the load. I do think each individual Kourier instance should be able to handle > 1024 concurrent requests to a service.
github-actions[bot]
knative-prow[bot]
knative-prow-robot
I use a custom client to send requests to Knative services with a Kourier gateway. It has a slow-start algorithm that gradually scales up concurrent requests until a configured limit is reached, or until error rates are too high. Under normal conditions, with HTTP/1.1, this can easily scale to 10k concurrent requests with just a few Kourier pods. Using HTTP/1.1 is more fragile as it could hit socket limits. However, if I switch that same config to HTTP/2, once the concurrency hits approximately 1000 (maybe slightly lower), Kourier / envoy responds with:
If I reduce the concurrency, but increase the number of clients sending requests, once the total concurrent requests goes above 1000, the same result happens. This might depend on the number of upstream service pods, I was testing with just 2.
Is there a way to tune this setting in the Kourier config? Ideally, if requests per socket connection need to be limited, it would allow spawning more connections when the limit is reached.
edit: to clarify, the upstream Service is not changing here, just how the client connects. The upstream service port is defined as a named port (
h2c).