Can't use internal redirect with websocket connection upgrade requests

[amortensen]

Under certain circumstances I would like to redirect a websocket connection upgrade request from one private location to another, in which the initial connection upgrade request is internally redirected after receiving a 302 from the service host fielding the initial request. It appears Envoy should support this, but does not due to a specific condition in the internal redirect handling code.

The flow below is what I would like to have working:

+--------+ ---upgrade: websocket---> +-------+ ---upgrade: websocket---> +-------+
| client |                           | envoy |                           | srv_A |
+--------+ <------101 Upgraded-----  +-------+ <-----302, Loc: srv_B---- +-------+
                                      ^     |
                                      |     |
                                      |  upgrade: websocket
                          |     |
                                     101    |
                                      |     v
                                     +-------+
                                     | srv_B |
                                     +-------+

It looks from the Envoy docs like this should be possible using internal_redirect_policy and upgrade_configs for the route configs. To test this, I modified the docker-compose.yml from the envoy websocket example. I added a cluster containing a simple node server that just redirects to the websocat cluster (see config below), and added the internal redirect policy and upgrade config. However, Envoy fails to perform the internal redirect, and the internal 302 is passed to the downstream client. I have separately verified that I can do internal redirects when the upgrade_configs block is not included in the config.

After adding some additional trace messages to the code, it appears the Filter::setupRedirect method will not perform the internal redirect if the downstream request is not complete (downstream_end_stream_ == false). The downstream request will of course not be complete because this is an upgrade request.

I don't see a way around this using currently available configuration options. It looks to me like my use case could be supported with a slightly more refined conditional in Filter::setupRedirect based on the presence of upgrade_configs (or a new config flag enabling redirect support in combination with upgrades), but I'm not sufficiently experienced with the code at this point to be sure.

Config:

# envoy-ws.yaml
admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9001

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_ws_to_ws
          route_config:
            name: local_route
            virtual_hosts:
            - name: app
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/websocat"
                route:
                  cluster: service_ws
                  internal_redirect_policy:
                    max_internal_redirects: 5
                  upgrade_configs:
                    - upgrade_type: websocket
              - match:
                  prefix: "/"
                route:
                  cluster: redir
                  internal_redirect_policy:
                    max_internal_redirects: 5
                  upgrade_configs:
                    - upgrade_type: websocket
          http_filters:
          - name: envoy.filters.http.router

  clusters:
  - name: redir
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: redir
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: redir
                port_value: 80
  - name: service_ws
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: service_ws
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: service-ws
                port_value: 80

# docker-compose.yaml
version: "3.7"
services:

  proxy-ws:
    image: envoy:parrot
    ports:
      - "9001:9001"
      - "10000:10000"
    volumes:
      - ./envoy-ws.yaml:/etc/envoy.yaml

  redir:
    build:
      context: .
      dockerfile: Dockerfile.redir-server
    hostname: redir
    environment:
      PORT: 80
      TARGET_HOST: service-ws

  service-ws:
    image: solsson/websocat
    hostname: service-ws
    command: -E ws-listen:0.0.0.0:80 literalreply:'[ws] HELO'

Logs: Full trace logs in this gist: envoy_ws_upgrade_internal_redirect_failure_logs. Key excerpts:

proxy-ws_1    | [2022-01-14 04:56:20.317][22][debug][http] [source/common/http/conn_manager_impl.cc:283] [C2] new stream
proxy-ws_1    | [2022-01-14 04:56:20.318][22][debug][http] [source/common/http/conn_manager_impl.cc:873] [C2][S17685677658886189605] request headers complete (end_stream=false):
proxy-ws_1    | ':authority', 'localhost:10000'
proxy-ws_1    | ':path', '/'
proxy-ws_1    | ':method', 'GET'
proxy-ws_1    | 'upgrade', 'websocket'
proxy-ws_1    | 'connection', 'Upgrade'
proxy-ws_1    | 'sec-websocket-key', '1cFsmiWI3V09wQQ8g+wgRQ=='
proxy-ws_1    | 'origin', 'http://localhost/'
proxy-ws_1    | 'sec-websocket-version', '13'
proxy-ws_1    |
proxy-ws_1    | [2022-01-14 04:56:20.318][22][debug][router] [source/common/router/router.cc:486] [C2][S17685677658886189605] cluster 'redir' match for URL '/'
...
proxy-ws_1    | [2022-01-14 04:56:20.320][22][trace][http] [source/common/http/http1/codec_impl.cc:869] [C2] message complete
proxy-ws_1    | [2022-01-14 04:56:20.320][22][trace][http] [source/common/http/http1/codec_impl.cc:877] [C2] Pausing parser due to upgrade.
proxy-ws_1    | [2022-01-14 04:56:20.321][22][trace][http] [source/common/http/http1/codec_impl.cc:655] [C2] parsed 181 bytes
proxy-ws_1    | [2022-01-14 04:56:20.321][22][trace][http] [source/common/http/http1/codec_impl.cc:582] [C2] direct-dispatched 0 bytes
proxy-ws_1    | [2022-01-14 04:56:20.321][22][trace][http] [source/common/http/http1/codec_impl.cc:1175] [C2] body size=0
...
proxy-ws_1    | [2022-01-14 04:56:20.325][22][trace][http] [source/common/http/http1/codec_impl.cc:1321] [C3] status_code 302
proxy-ws_1    | [2022-01-14 04:56:20.325][22][debug][router] [source/common/router/router.cc:1322] [C2][S17685677658886189605] upstream headers complete: end_stream=false
proxy-ws_1    | [2022-01-14 04:56:20.325][22][debug][router] [source/common/router/router.cc:1564] [C2][S17685677658886189605] attempting internal redirect
proxy-ws_1    | [2022-01-14 04:56:20.325][22][debug][router] [source/common/router/router.cc:1589] [C2][S17685677658886189605] Internal redirect failed
...
proxy-ws_1    | [2022-01-14 04:56:20.325][22][debug][http] [source/common/http/conn_manager_impl.cc:1472] [C2][S17685677658886189605] encoding headers via codec (end_stream=false):
proxy-ws_1    | ':status', '302'
proxy-ws_1    | 'location', 'http://service-ws/websocat'
proxy-ws_1    | 'date', 'Fri, 14 Jan 2022 04:56:20 GMT'
proxy-ws_1    | 'server', 'envoy'
proxy-ws_1    | 'connection', 'close'
proxy-ws_1    |

[jcferretti]

If you only need to redirect websocket connections, I believe you can workaround the issue by setting the websocket upgrade as the default for all routes, in the default HttpConnectionManager, eg:

      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                codec_type: AUTO
                stat_prefix: egress_http
                upgrade_configs:
                  - upgrade_type: websocket

If you have any non-websocket routes, you can explicitly disable the upgrade on them. If you also need to redirect non-websocket this won't work I think.

[acteek]

Hello @jcferretti

Your suggestion doesn't work too unfortunately

[jcferretti]

My company uses that workaround I described above in our production deployments. It works, with the caveats I mentioned.

[acteek]

@jcferretti Hello I try to test my setup here https://github.com/acteek/envoy-tests/blob/main/config/redirect.yaml and it doesn't work. I would be very happy if you could check the config and point out a mistake

[jcferretti]

Sorry I think I was wrong. I checked our configs and we don't do what I suggested above. I am not sure the following excerpt would help or not, but this is something we are actually doing:

         {
          "match": {
           "prefix": "/acl/"
          },
          "route": {
           "cluster": "8be5d925f228d2a8d0c4619402f36e03993f6357",
           "upgrade_configs": [
            {
             "upgrade_type": "websocket",
             "enabled": true
            }
           ]
          }
         },
         {
          "match": {
           "prefix": "/acl"
          },
          "redirect": {
           "path_redirect": "/acl/"
          }

[sfc-gh-mochen]

i am also looking to perform websocket redirects and this is not supported