alt-svc for Envoy

[alyssawilk]

We can do basic alt-svc for Envoy using the route response_headers_to_add but it'd be nice to be able to do alt-svc on a per listener basis rather than a route at a time, and to allow pooling across domains (alt-authority)

Given this is pretty core to using QUIC, I think it's worth putting directly in the HCM

[alyssawilk]

cc @danzh2010 @RyanTheOptimist @DavidSchinazi

[moderation]

With Firefox turning on h3 in beta and nightly - https://hacks.mozilla.org/2021/04/quic-and-http-3-support-now-in-firefox-nightly-and-beta/ - I was wondering what version does Envoy support? From my testing with clients like neqo and proxygen that can specify h3 draft version's it appears that only h3-29 works with Envoy. And https://quiche.googlesource.com/quiche/+/88c8d5903d851744410ea9840201b6507feae981/quic/core/quic_versions.cc#52 confirms that 29 is the latest draft. Would be good to add to the documentation

[RyanTheOptimist]

Makes sense. I suspect Envoy also supports the various Google QUIC versions. (there's also a flag which can be set to enable the final RFC version "h3", but that should not be enabled "in production" until the RFC is final) @danzh2010 probably knows for sure. I'd be happy to update the documentation if someone can point me in the right direction?

[DavidSchinazi]

To clarify, Firefox does not yet support h3, they're waiting for the spec to be done like everyone else. Firefox supports h3-27, h3-28, h3-29, h3-30, h3-31 and h3-32. Chrome supports h3-29, and Safari does not yet support HTTP/3. Envoy supports h3-29 today, and has disabled-by-policy support for h3. We'll enable h3 when the spec is done. From discussions at IETF, everyone has agreed to support h3-29 until enough time has passed after the spec's publication so that there is widespread support for h3.

[alyssawilk]

Yeah, I think with alt-svc we should add documentation for what versions are supported, and make sure that we can enable/disable versions.

@DavidSchinazi do we have a hand-wavy time frame for when we think we'll do H3 by default? I'm hoping by the time QUIC leaves alpha we'll have H3 on by default and not have to worry about the 1 year SLO on deprecation, but it'd be good to think about that early.

[DavidSchinazi]

I suspect we'll switch the default from h3-29 to h3,h3-29 in Q3 or Q4 this year?

[howardjohn]

One question, if the do we want to unconditionally set the alt-svc header when configured in HCM? Or only if not set?

[alyssawilk]

that could be a configuration option as well :-)

[su225]

So if I understand that correctly - in the context of the config here - https://github.com/envoyproxy/envoy/blob/3876d7c0fef12c8f47402eda841e59265372e837//configs/envoyproxy_io_proxy_http3_downstream.yaml - the TCP listener's HTTP Connection Manager would have an option like advertise_http3 and it is the responsibility of the user to configure a corresponding QUIC listener on the same port. Is that correct?

Or how about HTTP/3 advertise options. Something like this (Options from here - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Alt-Svc)

      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
+         http3_advertise_options:
+           supported_alpns: ["h3", "h3-29"]
+           max_age_seconds: 3600   # 3600 is just some number for illustration
+           persist: false
          codec_type: HTTP2
          stat_prefix: ingress_http

Or let us keep it simple in the beginning and just have

http3_advertise_options:
  enabled: true

which will send alt-svc for the same port with h3 and h3-29 (whatever we are using here)?

[alyssawilk]

I'm not an api shephard but I'd be inclined to keep the string literal in the config, so folks could have an easy time seeing what would be sent for the route config (which is done by request headers to add, so really has the literal) vs the HCM defaults.

That said one of the other things I'd like to see here is host options. When we were rolling out HTTP/3 in-house we had some domains we weren't advertising as on, some because we hadn't tuned latency for that traffic type, or for compliance reasons. so being able to advertise HTTP/3 for say all of .bar.com except eep.bar.com would be nice. This gets more fun if you're using ccTLDs so want .bar.* but not eep.bar.com or foo.bar.uk

If we allow this level of granularity, I think we might want to structure this as a general config which only has one use case today (HTTP/3) though. In-house we have 2 levels of route picking - one which is purely DNS based ("frontend service", for DNS load balancing) and one which is more like Enovy's route selection which can include path and custom headers. There are some nice htings you can do only at a frontend level like allow certain ciphers, enable/disable QUIC, etc, so I'd be inclined to design for that in the long run and QUIC as the first use of it.

[su225]

Adding a note here - In Istio (https://github.com/istio/istio/pull/33817), we use good old "add header" at HTTP route level for alt-svc as this is not there yet. Not a complex case as mentioned by @alyssawilk, just alt-svc: h3=:443; ma=86400.

[columbus9963]

@alyssawilk I'm trying to use quic enabled chrome client(107.0.5304.107) + envoy proxy(v1.23.1) to test h3 web, but it seems not to be totally ready. when chrome client sent a quic packet with version=draft-29, envoy response a version negotiation packet with supported version=1, and then the quic tls handshake failed. Any suggestions or solutions?

[alyssawilk]

hmmm, this should work with modern chrome. cc @danzh2010 @RyanTheOptimist

[RyanTheOptimist]

I believe Envoy only support QUIC v1 (RFC 9000), not draft-29. Are you choosing draft-29 explicitly? It should work with QUIC v1, as I understand it.

[moderation]

Looks like @columbus9963 opened issue https://github.com/envoyproxy/envoy/issues/24103