How to return real error message for a "two hop" proxy?

[imlk0]

Title: How to return real error message for a "two hop" proxy?

Description: I have an envoy instance which is an "HTTP2 CONNECT over TLS". But for some reason I implemented it as a two hop (with the help of internal_listener).

But I found that when the second hop fails (e.g. TLS handshake fails), my client only gets "upstream connect error or disconnect/reset before headers. reset reason: connection termination" which is a somewhat ambiguous reason.

Here is a streamlined configuration (envoy.yaml) to reproduce the problem:

bootstrap_extensions:
- name: envoy.bootstrap.internal_listener
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.bootstrap.internal_listener.v3.InternalListener

static_resources:

  listeners:
  - name: ingress0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10001
    socket_options:

    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress0_entry
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              log_format:
                text_format: "[%START_TIME%] ingress(0): \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE%(%RESPONSE_CODE_DETAILS%) %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n"
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: ingress0_wrap_in_h2_tls_upstream

          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
              suppress_envoy_headers: true

  - name: ingress0_wrap_in_h2_tls
    internal_listener: {}
    filter_chains:
    - filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: ingress0_wrap_in_h2_tls
          cluster: "ingress0_upstream"
          tunneling_config:
            hostname: "just.a.example.com"

  clusters:
  - name: ingress0_wrap_in_h2_tls_upstream
    load_assignment:
      cluster_name: ingress0_wrap_in_h2_tls_upstream
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              envoy_internal_address:
                server_listener_name: ingress0_wrap_in_h2_tls
    transport_socket:
      name: envoy.transport_sockets.internal_upstream
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.internal_upstream.v3.InternalUpstreamTransport
        transport_socket:
          name: envoy.transport_sockets.raw_buffer
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer

  - name: ingress0_upstream
    type: LOGICAL_DNS
    dns_lookup_family: V4_ONLY
    load_assignment:
      cluster_name: ingress0_upstream
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: www.github.com # For demonstration purposes, pick an upstream that would cause the TLS handshake to fail.
                port_value: 80
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:

Launch envoy with

envoy-static -l debug -c envoy.yaml

And then, send a request with curl

curl 127.0.0.1:10001

The HTTP response is

upstream connect error or disconnect/reset before headers. reset reason: connection termination

And the real reason for the error can only be found in the envoy log.

[2024-09-06 18:36:48.802][1315388][debug][connection] [source/common/tls/ssl_socket.cc:280] [Tags: "ConnectionId":"3"] remote address:20.205.243.166:80,TLS_error:|268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:TLS_error_end
[2024-09-06 18:36:48.802][1315388][debug][connection] [source/common/network/connection_impl.cc:281] [Tags: "ConnectionId":"3"] closing socket: 0
[2024-09-06 18:36:48.802][1315388][debug][connection] [source/common/tls/ssl_socket.cc:280] [Tags: "ConnectionId":"3"] remote address:20.205.243.166:80,TLS_error:|268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:TLS_error_end:TLS_error_end
[2024-09-06 18:36:48.802][1315388][debug][client] [source/common/http/codec_client.cc:107] [Tags: "ConnectionId":"3"] disconnect. resetting 0 pending requests
[2024-09-06 18:36:48.802][1315388][debug][pool] [source/common/conn_pool/conn_pool_base.cc:495] [Tags: "ConnectionId":"3"] client disconnected, failure reason: TLS_error:|268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:TLS_error_end:TLS_error_end

Showing that a TLS error occurred on the second hop.

I'd like to know if there is any way (either by tweaking the envoy config file, or by writing some code for envoy) to return the real error message in the HTTP response?

[phlax]

cc @kyessenov @adisuissa

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.