Cluster: fallback to HTTP/1.1 Websocket in auto_config when RFC 8441 ws over HTTP/2 negotiation fails (or always permit websocket over h1)

[bmcalary-atlassian]

Description: We have the following cluster configuration to ensure that HTTP/2 is used when signaled via ALPN, while still retaining the ability to fall back to HTTP/1.1 if HTTP/2 is not in the ALPN. This simplifies our xDS in a environment with mixed backends.

      "typed_extension_protocol_options": {
        "envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
          "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
          "common_http_protocol_options": {
            "idle_timeout": "55s"
          },
          "auto_config": {
            "http_protocol_options": {
              "header_key_format": {
                "proper_case_words": {
                }
              }
            },
            "http2_protocol_options": {
              "max_concurrent_streams": 128,
              "initial_stream_window_size": 1048576,
              "allow_connect": true
            }
          }
        }
      },

You'll see that allow_connect is present, which enables support for RFC 8441 (websocket over http/2 via extended connect), which expects the backend to send SETTINGS_ENABLE_CONNECT_PROTOCOL=1.

We find that if the backend DOES support HTTP/2 but does NOT support RFC 8441 SETTINGS_ENABLE_CONNECT_PROTOCOL=1 (e.g. AWS ALB) Envoy does not gracefully fall back to using HTTP/1.1 for the websocket, and the connections fail. The only recourse is to disable HTTP/2 for the cluster entirely.

Even if allow_connect is false, Envoy fails to proxy the Websocket connection if HTTP/2 is supported in ALPN.

It appears Envoy lacks logic to infer that a Websocket connection with allow_connect: false + ALPN h2 http1.1, or with allow_connect :true + ALPN h2 http1.1 + a backend that does not signal SETTINGS_ENABLE_CONNECT_PROTOCOL=1, would succeed if HTTP/1.1 were used.

We suggest the following solutions:

• Envoy can have a behaviour change: remember if a h2 endpoint/upstream doesn't signal SETTINGS_ENABLE_CONNECT_PROTOCOL (or allow_connect is false) but ALPN includes http1.1 can from that point on send websockets over http/1.1. This has the benefit of not introducing any new configuration key.
• if there is no provision for remembering the endpoint's history of signaling SETTINGS_ENABLE_CONNECT_PROTOCOL then a setting always_use_h1_for_websocket: true which forcefully uses http/1.1 for Websocket connections (if ALPN includes http1.1 or ALPN is not sent), but permits HTTP/2 for all other requests.
• If we can remember whether SETTINGS_ENABLE_CONNECT_PROTOCOL was signaled, then an auto_config key allow_websocket_h1_fallback: true or remember_upstream_connect_protocol_setting: true which would permit Envoy to open a HTTP/1.1 connection for websocket proxying in the presence of a HTTP/2 backend that does not signal RFC 8441 SETTINGS_ENABLE_CONNECT_PROTOCOL=1 support.

[optional Relevant Links:] https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-msg-config-core-v3-http2protocoloptions https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/upgrades.html https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-msg-config-core-v3-http2protocoloptions

[KBaichoo]

cc @alyssawilk

[alyssawilk]

It's too bad the protocol negotiation doesn't include feature negotiation or at least I don't know of any mechanism or plans to add one (cc @RyanTheOptimist ) I think it's a reasonable feature but I think you'd have to add support yourselves.

[bmcalary-atlassian]

@alyssawilk do you have a option preference by the way?

I personally prefer option 2: always_use_h1_for_websocket: true since it seems more definitive and possibly easier to implement.

I worry that option 1 and 3 introduce a round-trip to learn about SETTINGS_ENABLE_CONNECT_PROTOCOL support. And would be more complex to implement. Even if they would be more intelligent ultimately.

[alyssawilk]

no preference - I'd take it up with API shepherds, but I wonder if it'd make sense to make it more generic since for H2 it's all connect requests not just websocket right?