Envoy httpConnectionManager.http_filters does not respects the execution order (WASM + RBAC)

[achetronic]

Title: http_filters is not respecting the order between WASM and RBAC

Description:

What issue is being seen? Describe what should be happening instead of the bug, for example: Envoy should not crash, the expected value isn't returned, etc.

I have created a WASM plugin to override (or create) some custom header. This plugin is publicly accessible just in case you want to inspect all the work, or replicate something

The configuration inside http_filters, is done as follow

http_filters:
    - name: envoy.filters.http.wasm
      typed_config:
        "@type": type.googleapis.com/udpa.type.v1.TypedStruct
        type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
        value:
          config:
            configuration:
              "@type": type.googleapis.com/google.protobuf.StringValue
              value: |
                {
                  "trusted_networks" : [
                    "35.0.0.0/8",
                    "10.0.0.0/8",
                    "34.0.0.0/8"
                  ],
                  "injected_header_name": "x-real-client-ip",
                  "overwrite_header_on_exists": true
                }
            vm_config:
              runtime: "envoy.wasm.runtime.v8"
              code:
                local:
                  filename: "./dist/main.wasm"

    # RBAC filter is executed just after the plugin as the plugin is in charge of defining the
    # header that will be used by RBAC
    - name: envoy.filters.http.rbac
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC
        rules:
          action: ALLOW
          policies:
            "allow-specific-origins":
              permissions:
                - any: true
              principals:
                - or_ids:
                    ids:
                      - remote_ip:
                          address_prefix: "88.0.0.0"
                          prefix_len: 8
                      - remote_ip:
                          address_prefix: "69.0.0.0"
                          prefix_len: 8

    - name: envoy.filters.http.custom_debug
      typed_config:
        '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
        # Ref: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/lua_filter.html#dynamic-metadata-object-api
        # Ref: https://stackoverflow.com/a/75218669
        default_source_code:
          inline_string: |
            function envoy_on_request(request_handle)

              request_handle:logCritical("EnvoyFilter 'envoy.filters.http.custom_debug': direct response")

              local headers = request_handle:headers()
              header_xff = headers:get("x-forwarded-for")
              header_xrci = headers:get("x-real-client-ip")

              request_handle:respond({[":status"] = "200", ["x-forwarded-for"] = header_xff, ["x-real-client-ip"] = header_xrci }, "Direct response")
            end

    - name: envoy.filters.http.router
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

The mission is to configure Envoy to trust a custom header as follows to use RBAC based on a custom header which has been modified by the WASM plugin, respecting the previous order:

filter_chains:
  - filters:
      - name: envoy.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager

          # IMPORTANT:
          # If working with CIDRs in AuthorizationPolicy resources is a must, the following is needed.
          # Ask envoy to get the real IP from a custom header:
          original_ip_detection_extensions:
            - name: envoy.extensions.http.original_ip_detection.custom_header
              typed_config:
                "@type": "type.googleapis.com/envoy.extensions.http.original_ip_detection.custom_header.v3.CustomHeaderConfig"
                allow_extension_to_set_address_as_trusted: true
                header_name: "x-real-client-ip"

The problem is when I pass x-fordarded-for AND x-real-client-ip, RBAC is using the custom one as expected, but when I don't pass custom header, and only x-fordarded-for, my plugin is creating the custom header with the expected value, but RBAC filter is ignoring it and using values from x-fordarded-for to evaluate RBAC, even when my plugin is performed first in the filter chain, or at least, it should 🥲

Repro steps:

Include sample requests, environment, etc. All data and inputs required to reproduce the bug.

git clone git@github.com:achetronic/tnep.git
make run

# A curl that should work, but does not work cause RBAC is ignoring the header injected by WASM plugin
curl --verbose "http://localhost:19000/" -H "x-forwarded-for: 96.58.65.253,93.53.63.253,34.104.204.24,35.195.15.145,88.24.134.87,34.107.201.26,35.191.17.143"

# A curl that should work, but does not work. The plugin is overriding the header, but RBAC only obey to this passed header
curl --verbose "http://localhost:19000/" -H "x-forwarded-for: 96.58.65.253,93.53.63.253,34.104.204.24,35.195.15.145,88.24.134.87,34.107.201.26,35.191.17.143" -H "x-real-client-ip: 79.69.69.69

# A curl that works as the custom header is passed with right values for RBAC
curl --verbose "http://localhost:19000/" -H "x-forwarded-for: 96.58.65.253,93.53.63.253,34.104.204.24,35.195.15.145,88.24.134.87,34.107.201.26,35.191.17.143" -H "x-real-client-ip: 88.69.69.69

The whole config here:

# https://github.com/achetronic/tnep/blob/main/docs/samples/envoy-config.yaml
static_resources:
  listeners:
    - name: main
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 18000
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                codec_type: auto
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: web_service
                http_filters:
                  - &wasmFilterSpec
                    name: envoy.filters.http.wasm
                    typed_config:
                      "@type": type.googleapis.com/udpa.type.v1.TypedStruct
                      type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
                      value:
                        config:
                          configuration:
                            "@type": type.googleapis.com/google.protobuf.StringValue
                            value: |
                              {
                                "trusted_networks" : [
                                  "35.0.0.0/8",
                                  "10.0.0.0/8",
                                  "34.0.0.0/8"
                                ],
                                "injected_header_name": "x-real-client-ip",
                                "overwrite_header_on_exists": true
                              }
                          vm_config:
                            runtime: "envoy.wasm.runtime.v8"
                            code:
                              local:
                                filename: "./dist/main.wasm"
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

    - name: debug
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 19000
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager

                # IMPORTANT:
                # If working with CIDRs in AuthorizationPolicy resources is a must, the following is needed.
                # Ask envoy to get the real IP from a custom header:
                original_ip_detection_extensions:
                  - name: envoy.extensions.http.original_ip_detection.custom_header
                    typed_config:
                      "@type": "type.googleapis.com/envoy.extensions.http.original_ip_detection.custom_header.v3.CustomHeaderConfig"
                      allow_extension_to_set_address_as_trusted: true
                      header_name: "x-real-client-ip"

                stat_prefix: ingress_http
                codec_type: auto
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: web_service
                http_filters:
                  - *wasmFilterSpec

                  # RBAC filter is executed just after the plugin as the plugin is in charge of defining the
                  # header that will be used by RBAC
                  - name: envoy.filters.http.rbac
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC
                      rules:
                        action: ALLOW
                        policies:
                          "allow-specific-origins":
                            permissions:
                              - any: true
                            principals:
                              - or_ids:
                                  ids:
                                    - remote_ip:
                                        address_prefix: "88.0.0.0"
                                        prefix_len: 8
                                    - remote_ip:
                                        address_prefix: "69.0.0.0"
                                        prefix_len: 8

                  - name: envoy.filters.http.custom_debug
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
                      # Ref: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/lua_filter.html#dynamic-metadata-object-api
                      # Ref: https://stackoverflow.com/a/75218669
                      default_source_code:
                        inline_string: |
                          function envoy_on_request(request_handle)

                            request_handle:logCritical("EnvoyFilter 'envoy.filters.http.custom_debug': direct response")

                            local headers = request_handle:headers()
                            header_xff = headers:get("x-forwarded-for")
                            header_xrci = headers:get("x-real-client-ip")

                            request_handle:respond({[":status"] = "200", ["x-forwarded-for"] = header_xff, ["x-real-client-ip"] = header_xrci }, "Direct response")
                          end

                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

    - name: staticreply
      address:
        socket_address:
          address: 127.0.0.1
          port_value: 8099
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                original_ip_detection_extensions:
                  - name: envoy.extensions.http.original_ip_detection.custom_header
                    typed_config:
                      "@type": "type.googleapis.com/envoy.extensions.http.original_ip_detection.custom_header.v3.CustomHeaderConfig"
                      allow_extension_to_set_address_as_trusted: true
                      header_name: "x-real-client-ip"
                stat_prefix: ingress_http
                codec_type: auto
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/"
                          direct_response:
                            status: 200
                            body:
                              inline_string: "example body\n"
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
    - name: web_service
      connect_timeout: 0.25s
      type: STATIC
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: mock_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 8099

admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

[nezdolik]

cc @yaelharel @yanavlasov (rbac) @mpwarres (wasm)

[achetronic]

Hi @nezdolik @mpwarres @yaelharel @yanavlasov any update on this? :)

[kyessenov]

The order is respected. The issue is that "original IP detection" logic is executed only once, early in decodeHeaders before either of the filters runs. So changing the header has no impact on what RBAC reads as the original IP. You need to see a custom filter state or metadata to match instead in the RBAC.

[achetronic]

Hi @kyessenov thank you for your explanation on this topic :)

Then there is no way to check a CIDR coming from a header in RBAC? I mean, as a workaround I did this but this only checks specific IPs, of course, and I would like to check entire CIDRs in RBAC

Is there a kind of plugin (instead of WASM or something) to be able to change this? a .so or something that I can do?