Enabling HTTP3 removes `h2` from TLS ALPN

[byhemechi]

Description: Enabling HTTP3 in a ClientTrafficPolicy disables the spec.tls.alpn field, and sets ALPN in such a way that it advertises h3 but not h2

Repro steps:

Create a ClientTrafficPolicy with the http3 field set

Expected result

$ openssl s_client -alpn h2 -connect [gateway ip]:443
...
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
ALPN protocol: h2
Early data was not sent
Verify return code: 0 (ok)

Actual result

$ openssl s_client -alpn h2 -connect [gateway ip]:443
...
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Environment: Gateway: docker.io/envoyproxy/gateway-dev@sha256:cb9ff748dc8ea9c54e0cd5541658bff1aa3b8d68a945d7486561e4c593579f8c Proxy: docker.io/envoyproxy/envoy@sha256:0d9cd17db2674550f06625acd5f347dee8bcdf59a812ebfd741202ce179b253b

[byhemechi]

had a quick look through and found this, is there any reasoning for it? It doesn't seem to achieve anything that prepending h3 to the default list wouldn't

https://github.com/envoyproxy/gateway/blob/26ddfb0ae1740a9fa6b1a200823c1ed02c20bfc5/internal/gatewayapi/clienttrafficpolicy.go#L490-L499

[liorokman]

According to this, I think the ALPN configuration for the regular (TCP) listener doesn't need h3 to be included at all.

The linked documentation says that since HTTP/3 reuses the same URL schema (https://), the first connection to an HTTP/3 enabled server will be over TCP using HTTP/2. The first response would include the alt-svc header, and subsequent responses can use HTTP/3.

Seems like the issue here is that ALPN is touched at all.

[byhemechi]

Yeah that lines up with my knowledge of the standard, i just assumed i knew less than i thought i did.

Ironically, the ALPN change seems to completely prevent browsers from using HTTP3 because they seem to only accept the Alt-Svc header if the existing connection is http2

At least it should be an easy fix 🤷

[arkodg]

hey @byhemechi can you confirm if the issue is resolved for you ?

[byhemechi]

ok give me a moment to build the docker image, dockerhub is still at 0856f66

[byhemechi]

OK, http2 works, but now http3 is broken, it seems to be setting the wrong port on the Alt-Svc header (10443 instead of 443)

[arkodg]

uggh, the CI is a little flaky, kicked it, hopefully it should dish out an updated image soon https://github.com/envoyproxy/gateway/actions/runs/8241052639/job/22537972442

[byhemechi]

After change:

curl --http3 https://[domain] -i
HTTP/2 200
date: Mon, 11 Mar 2024 23:39:41 GMT
content-length: 366
content-type: text/plain; charset=utf-8
alt-svc: h3=":10443"; ma=86400

Currently rolling back to make sure this is actually a regression

[byhemechi]

Yeah looks like this broke h3

Before change

curl --http3 https://[redacted]
HTTP/3 200
date: Mon, 11 Mar 2024 23:47:09 GMT
content-length: 399
content-type: text/plain; charset=utf-8
alt-svc: h3=":10443"; ma=86400

[byhemechi]

the alt-svc is the same though, wondering if i actually misdiagnosed why it wasn't upgrading

[arkodg]

yeah there are 2 issues here

1. above change broke http3 on the same listener port
2. the alt-svc header contains the internal container port (10443), not the external port which has been highlighted here https://github.com/envoyproxy/gateway/pull/2111#discussion_r1392646333

I think we need a little more work to fix http3 and call it complete

[arkodg]

cc @tanujd11 are you free to take a look ?

[liorokman]

Are you sure that the "h3" string needs to be added at all?

I took a look at how https://www.cloudflare.com/ works (redacted for readability):

$ openssl s_client -connect www.cloudflare.com:443 -alpn h3,http/1.1
SSL handshake has read 4713 bytes and written 418 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE

ALPN protocol: http/1.1

Early data was not sent
Verify return code: 0 (ok)
---
HEAD / HTTP/1.1
Host: www.cloudflare.com 
---
HTTP/1.1 200 OK
Date: Tue, 12 Mar 2024 06:04:10 GMT
Content-Type: text/html; charset=utf-8

alt-svc: h3=":443"; ma=86400

You can see that I offer h3 from the client as well as http/1.1 (using openssl from the CLI, so no HTTP/2 for me), and the server rejects h3 and accepts http/1.1. Later on, the alt-svc header is used to offer the h3 protocol.