Dynamically select upstream cluster in TCP Proxy and Proxy Protocol

[guybrushthre]

I'm forwarding traffic to Envoy via Proxy Protocol and would like to configure Envoy to tunnel the traffic upstream using HTTP/2 Connect, similar to the encapsulate example provided here.

The goal is to use the original destination address (from the Proxy Protocol header) to determine the upstream Envoy proxy, while the original destination port is used to route the traffic to the appropriate destination service at the other end of the tunnel. My configuration looks something like this:

static_resources:
  listeners:
    - name: egress_listener
      address:
        socket_address:
          address: 0.0.0.0  
          port_value: 8080  
      listener_filters:
        - name: envoy.filters.listener.proxy_protocol
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol
      filter_chains:
        - filters:
            - name: envoy.filters.network.tcp_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                stat_prefix: egress_tcp_non_mtls
                cluster: other_envoy_proxy_cluster
                tunneling_config:
                  hostname: "127.0.0.1:%DOWNSTREAM_LOCAL_PORT%"
                  headers_to_add:
                  - header:
                      key: original_dst_port
                      value: "%DOWNSTREAM_LOCAL_PORT%"               

  clusters:
    - name: other_envoy_proxy_cluster  
      connect_timeout: 5s   
      # This ensures HTTP/2 CONNECT is used for establishing the tunnel.
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options: {}
      type: STATIC
      load_assignment:
        cluster_name: other_envoy_proxy_cluster
        endpoints:
        - lb_endpoints:
          - endpoint:
              address:
                socket_address:
                  address: "%DOWNSTREAM_LOCAL_ADDRESS_WITHOUT_PORT%"  # This is not supported
                  port_value: 10000

However, it seems that variable substitution (like %DOWNSTREAM_LOCAL_ADDRESS_WITHOUT_PORT%) is not supported for the SocketAddress, which makes this configuration invalid.

I am aware of the option to use the ORIGINAL_DST cluster type to automatically use the original destination IP and port, but in my case, I need to override the port (because the tunnel port and the final service destination port are different).

Is there any workaround to either:

1. Use variable substitution for the SocketAddress, or
2. Use the ORIGINAL_DST cluster type but still allow overriding the port for tunneling purposes?

[tyxia]

Maybe you can use envoy_v3_api_field_config.core.v3.SocketAddress.resolver_name?

add @zuercher who is probably the expert here and code owner of tcp_proxy

[guybrushthre]

Where can I find more information on custom resolvers? From what I understand, setting this up would require implementing a separate service that exposes either a gRPC-based resolution API or DNS-over-HTTPS (DoH). However, passing the original destination information to this service feels like a lot of complexity, whereas I was hoping to handle this in-process with something like a Lua script or another extension mechanism.

[pxpnetworks]

Hi, have a look if
Network Filter (network_filters/set_filter_state) in combination with cluster type ORIGINAL_DST and upstream port override (originaldstlbconfig) can work for you..

I haven't tested this but theoretically maybe something like this might work here: set_filter_state network filter config:

on_new_connection:
- object_key: envoy.network.transport_socket.original_dst_address
  format_string:
    text_format_source:
      inline_string: "%DOWNSTREAM_LOCAL_ADDRESS%"

and in the ORIGINAL_DST cluster: original_dst_lb_config: { upstream_port_override: 10000 }