unexpected behaviour regarding load balncer and http2MaxRequests

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description

i am experiencing unexpected behaviour regarding load balncer and http2MaxRequests . I have multiple clients and a service with replica (let's say 2 instances). I want each instance to only be able to handle one http request at a time. if the request takes some time to complete (instance is busy) the traffic should go to the other non busy instances. if all instances are busy the request should be queued.

I tried to use LEAST_REQUEST with http2MaxRequests: 1 but that caused unexpected behaviour. if one request takes some time to complete the second request returns : upstream connect error or disconnect/reset before headers. reset reason: overflow. it does not matter how many idle instances there are.

I tried using maxRequestsPerConnection: 1 , http1MaxPendingRequests: 1, maxConnections:1 and it almost worked. if one pod is busy the second request might go to the busy pod and be required or go to the idle pod. i wanted all requests to go to the idle pod.

current destination rule

apiVersion: [networking.istio.io/v1beta1](http://networking.istio.io/v1beta1)
kind: DestinationRule
metadata:
  name: billing-api-service-rule
  namespace: stage
spec:
  host: billing-api-service.stage.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
    loadBalancer:
      simple: LEAST_REQUEST
      localityLbSetting:
        enabled: false
    connectionPool:
      http:
        maxRequestsPerConnection: 1
        http1MaxPendingRequests: 1
        http2MaxRequests: 1

      tcp:
        maxConnections: 1
    outlierDetection:
      baseEjectionTime: 3m
      consecutive5xxErrors: 1
      interval: 1s
      maxEjectionPercent: 50

Version

istioctl version
client version: 1.17.6
control plane version: 1.20.2
data plane version: 1.20.2 (97 proxies)

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.8-eks-8cb36c9

Additional Information

No response

istio / istio