search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
25.03k stars 4.82k forks source link

CONNECT-UDP [QUIC] Stream reset: reset reason: protocol error, response details: http3.invalid_header_field #37157

Open rishabh78 opened 1 day ago

rishabh78 commented 1 day ago

If you are reporting any crash or any potential security issue, do not open an issue in this repo. Please report the issue via emailing envoy-security@googlegroups.com where the issue will be triaged appropriately.

Title: One line description I have client which sends CONNECT-UDP traffic through an Envoy proxy, which forwards it to an upstream UDP server.

Description:

What issue is being seen? I am seeing following error on envoy stream reset: reset reason: protocol error, response details: http3.invalid_header_field

Envoy configuration 

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: UDP
        address: 127.0.0.1
        port_value: 10001
    udp_listener_config:
      quic_options: {}
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.quic
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
          downstream_tls_context:
            common_tls_context:
              tls_certificates:
              - certificate_chain:
                  filename: certs/servercert.pem
                private_key:
                  filename: certs/serverkey.pem
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP3
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  connect_matcher:
                    {}
                route:
                  cluster: cluster_0
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          http3_protocol_options:
            allow_extended_connect: true
          upgrade_configs:
          - upgrade_type: CONNECT-UDP
  clusters:
  - name: cluster_0
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http3_protocol_options:
            allow_extended_connect: true
    load_assignment:
      cluster_name: cluster_0
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10002
    transport_socket:
      name: envoy.transport_sockets.quic
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicUpstreamTransport
        upstream_tls_context:
          sni: localhost

UDP server 

# udp_server.py
import socket

def start_udp_server():
    server_address = ('127.0.0.1', 10002)
    buffer_size = 1024

    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as server_socket:
        server_socket.bind(server_address)
        print(f"UDP server is listening on {server_address[0]}:{server_address[1]}")

        while True:
            data, client_address = server_socket.recvfrom(buffer_size)
            print(f"Received {data} from {client_address}")
            response = b"Response from UDP server"
            server_socket.sendto(response, client_address)

if __name__ == "__main__":
    start_udp_server()

Client 

# connect_udp_client.py
import asyncio
from aioquic.asyncio import connect
from aioquic.quic.configuration import QuicConfiguration
from aioquic.h3.connection import H3_ALPN, H3Connection
from aioquic.h3.events import HeadersReceived, DataReceived, H3Event
from aioquic.asyncio.protocol import QuicConnectionProtocol

class CustomQuicClient(QuicConnectionProtocol):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.h3_connection = H3Connection(self._quic)
        self.events = []

    def quic_event_received(self, event):
        for http_event in self.h3_connection.handle_event(event):
            self.events.append(http_event)

async def main():
    # Configure QUIC client
    configuration = QuicConfiguration(is_client=True)
    configuration.alpn_protocols = H3_ALPN
    configuration.verify_mode = False  # Disable verification for local testing
    configuration.server_name = "localhost"  # Set the appropriate SNI

    async with connect("127.0.0.1", 10001, configuration=configuration, create_protocol=CustomQuicClient) as protocol:
        quic = protocol._quic
        h3 = protocol.h3_connection

        # Create a new stream
        stream_id = quic.get_next_available_stream_id()

        # Send CONNECT-UDP request headers
        target_host = "127.0.0.1"
        target_port = 10002
        h3.send_headers(
            stream_id=stream_id,
            headers=[
                (b":method", b"CONNECT-UDP"),
                (b":authority", f"{target_host}:{target_port}".encode()),
            ],
        )

        # Send some data over the UDP connection
        message = b"Hello from CONNECT-UDP client"
        h3.send_data(stream_id, message, end_stream=False)

        # Handle incoming events
        await asyncio.sleep(1)
        for event in protocol.events:
            if isinstance(event, HeadersReceived):
                print(f"Received headers: {event.headers}")
            elif isinstance(event, DataReceived):
                print(f"Received data: {event.data.decode()}")

if __name__ == "__main__":
    asyncio.run(main())

Envoy logs 

[2024-11-14 15:14:07.360][2386822][debug][http] [source/common/http/conn_manager_impl.cc:385] [Tags: "ConnectionId":"15930477111956885085"] new stream
[2024-11-14 15:14:07.360][2386822][debug][http] [source/common/http/conn_manager_impl.cc:1894] [Tags: "ConnectionId":"15930477111956885085","StreamId":"18077200307027120781"] stream reset: reset reason: protocol error, response details: http3.invalid_header_field
[2024-11-14 15:14:08.256][2386793][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-14 15:14:13.259][2386793][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-14 15:14:18.264][2386793][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-14 15:14:18.454][2386817][debug][http] [source/common/http/conn_manager_impl.cc:385] [Tags: "ConnectionId":"11738211879808972429"] new stream
[2024-11-14 15:14:18.454][2386817][debug][http] [source/common/http/conn_manager_impl.cc:1894] [Tags: "ConnectionId":"11738211879808972429","StreamId":"5847963984936782337"] stream reset: reset reason: protocol error, response details: http3.invalid_header_field
rishabh78 commented 23 hours ago

@DavidSchinazi

yanavlasov commented 23 hours ago

Adding @danzh2010 for any insights

danzh2010 commented 22 hours ago

The request has some invalid HTTP header. Can you share the requests you are sending?

rishabh78 commented 22 hours ago

''' h3 = protocol.h3_connection

    # Create a new stream
    stream_id = quic.get_next_available_stream_id()

    # Send CONNECT-UDP request headers
    target_host = "127.0.0.1"
    target_port = 10002
    h3.send_headers(
        stream_id=stream_id,
        headers=[
            (b":method", b"CONNECT-UDP"),
            (b":authority", f"{target_host}:{target_port}".encode()),
        ],
    )

'''

danzh2010 commented 21 hours ago

''' h3 = protocol.h3_connection

    # Create a new stream
    stream_id = quic.get_next_available_stream_id()

    # Send CONNECT-UDP request headers
    target_host = "127.0.0.1"
    target_port = 10002
    h3.send_headers(
        stream_id=stream_id,
        headers=[
            (b":method", b"CONNECT-UDP"),
            (b":authority", f"{target_host}:{target_port}".encode()),
        ],
    )

'''

@DavidSchinazi Is this how CONNECT-UDP header should be like?

rishabh78 commented 20 hours ago

I am using this clinet 

# connect_udp_client.py
import asyncio
from aioquic.asyncio import connect
from aioquic.quic.configuration import QuicConfiguration
from aioquic.h3.connection import H3_ALPN, H3Connection
from aioquic.h3.events import HeadersReceived, DataReceived, H3Event
from aioquic.asyncio.protocol import QuicConnectionProtocol

class CustomQuicClient(QuicConnectionProtocol):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.h3_connection = H3Connection(self._quic)
        self.events = []

    def quic_event_received(self, event):
        for http_event in self.h3_connection.handle_event(event):
            self.events.append(http_event)

async def main():
    # Configure QUIC client
    configuration = QuicConfiguration(is_client=True)
    configuration.alpn_protocols = H3_ALPN
    configuration.verify_mode = False  # Disable verification for local testing
    configuration.server_name = "localhost"  # Set the appropriate SNI

    async with connect("127.0.0.1", 10001, configuration=configuration, create_protocol=CustomQuicClient) as protocol:
        quic = protocol._quic
        h3 = protocol.h3_connection

        # Create a new stream
        stream_id = quic.get_next_available_stream_id()

        # Send CONNECT-UDP request headers
        target_host = "127.0.0.1"
        target_port = 10002
        h3.send_headers(
            stream_id=stream_id,
            headers=[
                (b":method", b"CONNECT-UDP"),
                (b":authority", f"{target_host}:{target_port}".encode()),
            ],
        )

        # Send some data over the UDP connection
        message = b"Hello from CONNECT-UDP client"
        h3.send_data(stream_id, message, end_stream=False)

        # Handle incoming events
        await asyncio.sleep(1)
        for event in protocol.events:
            if isinstance(event, HeadersReceived):
                print(f"Received headers: {event.headers}")
            elif isinstance(event, DataReceived):
                print(f"Received data: {event.data.decode()}")

if __name__ == "__main__":
    asyncio.run(main())

Please feel free to suggest any other library / option of testing this envoy config @DavidSchinazi @danzh2010 . Can I warp CONNECT-UDP with http3 curl ?

rishabh78 commented 15 hours ago

@danzh2010 @DavidSchinazi I tried using masque_client too 

./bazel-bin/quiche/masque_client --disable_certificate_verification 127.0.0.1:10001  127.0.0.1:10002
E1116 01:03:07.662913  453146 masque_client.cc:128] Failed to connect. Error: QUIC_NETWORK_IDLE_TIMEOUT
E1116 01:03:07.663169  453146 masque_client_tools.cc:116] Failed to prepare MasqueEncapsulatedClient for 127.0.0.1:10002

envoy error logs 

[2024-11-16 01:03:03.655][453085][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_server_handshaker.cc:982] No hostname indicated in SNI
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:393] [Tags: "ConnectionId":"14877056478440268800"] new stream
[2024-11-16 01:03:03.662][453085][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:165] [Tags: "ConnectionId":"14877056478440268800","StreamId":"0"] Received headers: { :method=CONNECT, :protocol=connect-udp, :scheme=https, :authority=127.0.0.1:10001, :path=/.well-known/masque/udp//0/, }.
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:1183] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] request headers complete (end_stream=false):
':method', 'GET'
':scheme', 'https'
':authority', '127.0.0.1:10001'
':path', '/.well-known/masque/udp//0/'
'upgrade', 'connect-udp'
'connection', 'upgrade'

[2024-11-16 01:03:03.662][453085][warning][misc] [source/common/http/header_utility.cc:383] CONNECT-UDP request with a malformed URI template in the path /.well-known/masque/udp//0/
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/filter_manager.cc:1084] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] Sending local reply with details invalid_path
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:1878] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] encoding headers via codec (end_stream=false):
':status', '404'
'content-length', '37'
'content-type', 'text/plain'
'date', 'Sat, 16 Nov 2024 01:03:03 GMT'
'server', 'envoy'

[2024-11-16 01:03:03.662][453085][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:55] [Tags: "ConnectionId":"14877056478440268800","StreamId":"0"] encodeHeaders (end_stream=false) ':status', '404'
'content-length', '37'
'content-type', 'text/plain'
'date', 'Sat, 16 Nov 2024 01:03:03 GMT'
'server', 'envoy'
.
[2024-11-16 01:03:03.662][453085][debug][quic_stream] [source/common/quic/envoy_quic_stream.cc:14] [Tags: "ConnectionId":"14877056478440268800","StreamId":"0"] encodeData (end_stream=true) of 37 bytes.
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:1993] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] Codec completed encoding stream.
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:257] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] doEndStream() resetting stream
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:1950] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] stream reset: reset reason: local reset, response details: -
[2024-11-16 01:03:04.972][453078][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-16 01:03:07.662][453085][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:325] [Tags: "ConnectionId":"14877056478440268800","StreamId":"0"] received STOP_SENDING with reset code=6
[2024-11-16 01:03:07.662][453085][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:352] [Tags: "ConnectionId":"14877056478440268800","StreamId":"0"] received RESET_STREAM with reset code=6
[2024-11-16 01:03:09.973][453078][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-16 01:03:14.974][453078][debug][main] [source/server/server.cc