search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
25.04k stars 4.82k forks source link

http2_multiplexing: http stream created on existing dead connection waits until http2 ping timeout to detect connection failure #37236

Open IssaAbuKalbein opened 4 hours ago

IssaAbuKalbein commented 4 hours ago

Title: http2_multiplexing: http stream created on existing dead connection waits until http2 ping timeout to detect connection failure

Description: We're using Tunneling TCP over HTTP feature for tunneling TCP over HTTP2. Our cluster is a dynamic forward proxy with allow_coalesced_connections: true and following protocol options config:

typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options:
            connection_keepalive:
              interval: 120s
              timeout: 20s

We've observed failures on streams created on already used and idle connections. TCP flow that is using one of such streams and waiting for response headers, only fails after the timeout of the keepalive ping, which could take more than 140s. Is this the expected behavior in such case? could we apply the cluster connect_timeout on "waiting for response headers"?

Also, we've configured the tcp_proxy with max_connect_attempts: 4 and we see in access log that we've tried for four times to connect but we still fail. Does envoy use the same connection to create a stream in all attempts? why the retries don't help in this case?

Thanks!

soulxu commented 2 hours ago

cc @alyssawilk