Support configuring backend requests to use the same HTTP protocol as the client request

[liorokman]

Description:

The way that EG sets up HTTPRoutes by default in Envoy Proxy today is that if TLS is enabled downstream, then ALPN negotiation is enabled with the default values h2, http/1.1 . Clients that support ALPN will prefer to use HTTP/2.

In contrast, upstream connections are always HTTP/1.1 for HTTPRoutes. This can be seen in the following output (edited for brevity) :

$ curl  --resolve www.example.com:443:172.18.255.200 https://www.example.com/ -H 'Host: www.example.com' -v -k
* Added www.example.com:443:172.18.255.200 to DNS cache
* Hostname www.example.com was found in DNS cache
*   Trying 172.18.255.200:443...
* Connected to www.example.com (172.18.255.200) port 443
* ALPN: curl offers h2,http/1.1
* ALPN: server accepted h2
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.example.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.example.com]
> GET / HTTP/2
> Host: www.example.com
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/2 200 
< content-type: application/json
< date: Fri, 12 Jan 2024 19:02:37 GMT
< 
{
 "path": "/",
 "host": "www.example.com",
 "method": "GET",
 "proto": "HTTP/1.1",
}
* Connection #0 to host www.example.com left intact

The output shows that while the curl command uses HTTP/2, the echo server identifies the protocol as HTTP/1.1.

The fact that Envoy Proxy downgrades the protocol from HTTP/2 to HTTP/1.1 means that there is a potential attack surface created here for HTTP smuggling attacks, as detailed here.

When running the HTTP smuggling detection tool ( https://github.com/neex/http2smugl ) and keeping track of actual requests received in the server and the Envoy Proxy logs, there are several requests that appear in Envoy Proxy's log as rejected but are actually sent all the way to the backend server.

For example:

The following request specifies a response code of 0 and response flags DPE ("The downstream request had an HTTP protocol error"), and bytes sent and received sizes of 0.

{"start_time":"2024-01-12T18:32:48.224Z","method":"POST","x-envoy-origin-path":"/","protocol":"HTTP/2","response_code":"0","response_flags":"DPE","response_code_details":"codec_error:The_user_callback_function_failed","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"1000","x-envoy-upstream-service-time":"-","x-forwarded-for":"172.18.0.1","user-agent":"Mozilla/5.0","x-request-id":"b4977c3f-9f08-492c-bc72-e7cebfab4240",":authority":"www.example.com","upstream_host":"10.244.0.8:3000","upstream_cluster":"httproute/server/backend/rule/0","upstream_local_address":"10.244.0.10:44152","downstream_local_address":"10.244.0.10:10443","downstream_remote_address":"172.18.0.1:37424","requested_server_name":"-","route_name":"httproute/server/backend/rule/0/match/0/www_example_com"}

When running a tcpdump session on the backend server side this specific request is actually found, including a valid HTTP 200 response. This was correlated via the x-request-id header shown in the Envoy Proxy log. Here is the packet dump, edited for brevity.

No.     Time           Source                Destination           Protocol Length Info
     50 8.414138       10.244.0.8            10.244.0.10           HTTP/JSON 705    HTTP/1.1 200 OK , JSON (application/json)

Frame 50: 705 bytes on wire (5640 bits), 705 bytes captured (5640 bits)
Hypertext Transfer Protocol
    HTTP/1.1 200 OK\r\n
        [Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n]
            [HTTP/1.1 200 OK\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Response Version: HTTP/1.1
        Status Code: 200
        [Status Code Description: OK]
        Response Phrase: OK
    Content-Type: application/json\r\n
    X-Content-Type-Options: nosniff\r\n
    Date: Fri, 12 Jan 2024 18:32:49 GMT\r\n
    Content-Length: 478\r\n
        [Content length: 478]
    Connection: close\r\n
    \r\n
    [HTTP response 1/1]
    File Data: 478 bytes
JavaScript Object Notation: application/json
    Object
        Member: path
            [Path with value: /path:/]
            [Member with value: path:/]
            String value: /
            Key: path
            [Path: /path]
        Member: host
            [Path with value: /host:www.example.com]
            [Member with value: host:www.example.com]
            String value: www.example.com
            Key: host
            [Path: /host]
        Member: method
            [Path with value: /method:POST]
            [Member with value: method:POST]
            String value: POST
            Key: method
            [Path: /method]
        Member: proto
            [Path with value: /proto:HTTP/1.1]
            [Member with value: proto:HTTP/1.1]
            String value: HTTP/1.1
            Key: proto
            [Path: /proto]
        Member: headers
            Object
                Member: X-Request-Id
                    Array
                        [Path with value: /headers/X-Request-Id/[]:b4977c3f-9f08-492c-bc72-e7cebfab4240]
                        [Member with value: []:b4977c3f-9f08-492c-bc72-e7cebfab4240]
                        String value: b4977c3f-9f08-492c-bc72-e7cebfab4240
                    Key: X-Request-Id
                    [Path: /headers/X-Request-Id]
            Key: headers
            [Path: /headers]

Since the Envoy Proxy log shows that there was a protocol error and no bytes sent or received, the request should not have appeared at all in the packet capture on the backend side.

I propose that it be made possible to prevent this category of issues by configuring Envoy Gateway to use the same HTTP protocol in upstream requests as received from downstream.

[optional Relevant Links:]

https://book.hacktricks.xyz/pentesting-web/http-request-smuggling/request-smuggling-in-http-2-downgrades

[arkodg]

@liorokman can't this be solved by pinning alpnProtocols https://gateway.envoyproxy.io/latest/api/extension_types/#alpnprotocol to http/1.1 ? https://gateway.envoyproxy.io/latest/api/extension_types/#tlssettings

[liorokman]

Soon Envoy Gateway will support BackendTLSPolicy ( #2247 ), at which point the protocol between the Envoy Proxy and the backend will most likely be negotiated with ALPN. At that point it makes sense to me to have the ability to prevent protocol upgrades/downgrades in the context of a specific request.

Pinning HTTP/1.1 on the client side is a solution that only makes sense if the upstream isn't TLS enabled. If it's possible to negotiate the HTTP protocol with upstream, I think it's better to allow HTTP/2 when the client supports it so that the benefits of HTTP/2 can be realised.

[arkodg]

it sounds like for the case when downstream tls supports h2 and http/1.1 and upstream tls also supports h2 and http/1.1, we want to make sure that protocol downgrades are not supported. So instead of a field like USE_DOWNSTREAM_PROTOCOL (which disables protocol upgrades) what we'd really want is for envoy to disable downgrades by default or add a knob to support that, cc @ggreenway in case you have any thoughts here

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[liorokman]

The useClientProtocol added in #2433 solves this issue.