Connection is HTTP/1, but request requires HTTP/2

SharkyRawr commented 1 month ago

Bug report:

Hi, I'm trying to use dragonflyoss/helm-charts as a transparent proxy and cache on my servers for all public docker registries.
I have set up:

proxy:
    server:
        caCert: /certs/wild.crt
        caKey: /certs/wild.key

using my own CA cert infrastructure and it appears to be validating okay.
However when dfdaemon tries to connect to registry.hub.docker.com or ghcr.io it fails with this error:

  2024-10-12T03:06:38.425239352+00:00  INFO  accepted connection from 10.88.1.34:58407
    at dragonfly-client/src/proxy/mod.rs:191
    in run

  2024-10-12T03:06:38.429826434+00:00  INFO  handle HTTPS request: Request { method: CONNECT, uri: ghcr.io:443, version: HTTP/1.1, headers: {"host": "ghcr.io:443", "user-agent": "Go-http-client/1.1"}, body: Body(Empty) }
    at dragonfly-client/src/proxy/mod.rs:412
    in https_handler
    in handler with uri: "ghcr.io:443", method: "CONNECT"

  2024-10-12T03:06:38.429910082+00:00  INFO  generate self-signed certificate by CA certificate
    at dragonfly-client/src/proxy/mod.rs:466
    in upgraded_tunnel

  2024-10-12T03:06:38.477473855+00:00  INFO  proxy HTTPS request directly to remote server for method: GET, uri: https://ghcr.io/token?scope=repository%3Adragonflyoss%2Fdragonfly2%2Fdfdaemon%3Apull&service=ghcr.io
    at dragonfly-client/src/proxy/mod.rs:568
    in upgraded_handler with uri: "https://ghcr.io/token?scope=repository%3Adragonflyoss%2Fdragonfly2%2Fdfdaemon%3Apull&service=ghcr.io", method: "GET"

  2024-10-12T03:06:38.493531640+00:00  WARN  Connection is HTTP/1, but request requires HTTP/2
    at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.9/src/client/legacy/client.rs:291
    in proxy_https

  2024-10-12T03:15:12.537080587+00:00  INFO  accepted connection from 10.88.1.34:58824
    at dragonfly-client/src/proxy/mod.rs:191
    in run

  2024-10-12T03:15:12.537367129+00:00  INFO  handle HTTPS request: Request { method: CONNECT, uri: registry-1.docker.io:443, version: HTTP/1.1, headers: {"host": "registry-1.docker.io:443", "user-agent": "Go-http-client/1.1"}, body: Body(Empty) }
    at dragonfly-client/src/proxy/mod.rs:412
    in https_handler
    in handler with uri: "registry-1.docker.io:443", method: "CONNECT"

  2024-10-12T03:15:12.537491214+00:00  INFO  generate self-signed certificate by CA certificate
    at dragonfly-client/src/proxy/mod.rs:466
    in upgraded_tunnel

  2024-10-12T03:15:12.583357036+00:00  INFO  proxy HTTPS request directly to remote server for method: HEAD, uri: https://registry-1.docker.io/v2/library/busybox/manifests/latest
    at dragonfly-client/src/proxy/mod.rs:568
    in upgraded_handler with uri: "/v2/library/busybox/manifests/latest", method: "HEAD"
    in upgraded_tunnel

  2024-10-12T03:15:13.030116463+00:00  INFO  accepted connection from 10.88.1.34:58825
    at dragonfly-client/src/proxy/mod.rs:191
    in run

  2024-10-12T03:15:13.030367948+00:00  INFO  handle HTTPS request: Request { method: CONNECT, uri: auth.docker.io:443, version: HTTP/1.1, headers: {"host": "auth.docker.io:443", "user-agent": "Go-http-client/1.1"}, body: Body(Empty) }
    at dragonfly-client/src/proxy/mod.rs:412
    in https_handler
    in handler with uri: "auth.docker.io:443", method: "CONNECT"

  2024-10-12T03:15:13.030460432+00:00  INFO  generate self-signed certificate by CA certificate
    at dragonfly-client/src/proxy/mod.rs:466
    in upgraded_tunnel

  2024-10-12T03:15:13.078166875+00:00  INFO  proxy HTTPS request directly to remote server for method: POST, uri: https://auth.docker.io/token
    at dragonfly-client/src/proxy/mod.rs:568
    in upgraded_handler with uri: "https://auth.docker.io/token", method: "POST"

  2024-10-12T03:15:13.280956381+00:00  WARN  Connection is HTTP/1, but request requires HTTP/2
    at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.9/src/client/legacy/client.rs:291
    in proxy_https
    in upgraded_handler with uri: "https://auth.docker.io/token", method: "POST"

  2024-10-12T03:15:13.281001045+00:00 ERROR  request failed: hyper_util::client::legacy::Error(UserUnsupportedVersion)
    at dragonfly-client/src/proxy/mod.rs:906
    in proxy_https
    in upgraded_handler with uri: "https://auth.docker.io/token", method: "POST"

It appears the client wants to upgrade to h2/HTTP2 but hyper-util v0.1.9 refuses at https://github.com/hyperium/hyper-util/blob/v0.1.9/src/client/legacy/client.rs#L291. I'm not sure if this is a dragonfly client or hyper-util issue as I'm not that experienced with Rust - or it's something completely different. Perhaps my approach is wrong but I would appreciate if someone could take a look at this and shed some light. :)

Expected behavior:

docker pull any image and it works.

How to reproduce it:

Set up a client with caCert and caKey, enable HTTPS_PROXY for docker daemon and try:

docker pull busybox HTTPS_PROXY="http://127.0.0.1:4001/" curl -vvv registry.hub.docker.com -L

CURL error:

* Uses proxy env variable HTTPS_PROXY == 'http://127.0.0.1:4001/'
*   Trying 127.0.0.1:4001...
* Connected to 127.0.0.1 (127.0.0.1) port 4001
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to registry.hub.docker.com:443
> CONNECT registry.hub.docker.com:443 HTTP/1.1
> Host: registry.hub.docker.com:443
> User-Agent: curl/8.7.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< Date: Sat, 12 Oct 2024 03:45:03 GMT
<
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=rcgen self signed cert
*  start date: Jan  1 00:00:00 1975 GMT
*  expire date: Jan  1 00:00:00 4096 GMT
*  subjectAltName: host "registry.hub.docker.com" matched cert's "registry.hub.docker.com"
*  issuer: CN=Dragonfly Wildcard CA
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://registry.hub.docker.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: registry.hub.docker.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: registry.hub.docker.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
* HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
* Connection #1 to host 127.0.0.1 left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

CURL request works with --http1.1 though.

Environment:

Client version: v0.1.111
OS: Linux/Ubuntu
Kernel (e.g. uname -a): 6.8.0-45-generic
Others: Pulling via Docker Desktop macOS 4.34.3 (170107), Engine: 27.2.0

gaius-qi commented 3 weeks ago

@CormickKneey

CormickKneey commented 3 weeks ago

Okay, i will check this today.

CormickKneey commented 3 weeks ago

The problem is: Connection between curl and proxy is http2, Connection between proxy and most of remote (like registry.hub.docker.com) is http1 after protocol negotiation. So the request transferred version is incorrect. We can revolve this by remove h2 support when serving the connection with the TLS stream, what do u think? @gaius-qi

gaius-qi commented 3 weeks ago

The problem is: Connection between curl and proxy is http2, Connection between proxy and most of remote (like registry.hub.docker.com) is http1 after protocol negotiation. So the request transferred version is incorrect.

We can revolve this by remove h2 support when serving the connection with the TLS stream, what do u think? @gaius-qi

LGTM，set the http1_only to true in the proxy.

CormickKneey commented 3 weeks ago

Closed by https://github.com/dragonflyoss/client/pull/796

dragonflyoss / client