Incorrect route order when delegateAction selector selects multiple routetables

Gloo Edge Product

Enterprise

Gloo Edge Version

1.16.5

Kubernetes Version

v1.24.0

Describe the bug

Having a route in VS selecting more than 1 routetable in it's delegateAction selector results in incorrect route order in resulting configuration where a route from top of a routetable is put at the end of the envoy configuration.

Expected Behavior

I expect the order of the routes within a routetable to be consistent even if there are different routetables selected by the VS delegation selector.

Steps to reproduce the bug

kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - delegateAction:
        selector:
          labels:
            route: "true"
          namespaces:
          - '*'
EOF
kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: RouteTable
metadata:
  labels:
    route: "true"
  name: httpbin
  namespace: httpbin
spec:
  routes:
  - matchers:
    - headers:
      - name: city-id
        value: "99999"
      prefix: /
    routeAction:
      single:
        upstream:
          name: httpbin-httpbin-8000
          namespace: gloo-system
  - directResponseAction:
      body: DIRECT
      status: 200
    matchers:
    - headers:
      prefix: /get
EOF

See that the routing is now correct and the route with prefix: / and city-id header matcher doesn't return direct response, because the route is above the DirectResponse route.

curl $(glooctl proxy url)/get -H "city-id: 99999"
----------
glooctl binary version (1.15.2) differs from server components (v1.16.17) by at least a minor version.
Consider running:
glooctl upgrade --release=v1.16.17
----------

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "City-Id": "99999", 
    "Host": "172.18.9.1", 
    "User-Agent": "curl/7.81.0", 
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  }, 
  "origin": "10.109.0.27", 
  "url": "http://172.18.9.1/get"
}
curl $(glooctl proxy url)/get -H "city-id: 999991"
----------
glooctl binary version (1.15.2) differs from server components (v1.16.17) by at least a minor version.
Consider running:
glooctl upgrade --release=v1.16.17
----------

DIRECT

Apply additional routetable that will be selected by the VS matcher

kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: RouteTable
metadata:
  labels:
    route: "true"
  name: httpbin-2
  namespace: httpbin
spec:
  routes:
  - matchers:
    - headers:
      - name: city-id
        value: "10000"
      prefix: /anything/test
    routeAction:
      single:
        upstream:
          name: httpbin-httpbin-8000
          namespace: gloo-system
  - directResponseAction:
      body: DIRECT
      status: 200
    matchers:
    - headers:
      prefix: /anything/rider
EOF

Now the routing is not correct and also the request with city-id set to 99999 returns direct response

curl $(glooctl proxy url)/get -H "city-id: 99999"
----------
glooctl binary version (1.15.2) differs from server components (v1.16.17) by at least a minor version.
Consider running:
glooctl upgrade --release=v1.16.17
----------

DIRECT

Additional Environment Detail

No response

Additional Context

It's possible to workaround it by using different value of the selector label for the specific routetable. For example in below example I'd set VS:

    routes:
    - delegateAction:
        selector:
          labels:
            route: "true"
          namespaces:
          - '*'
    - delegateAction:
        selector:
          labels:
            route: true-wildcard
          namespaces:
          - '*'

and would change the value of route label in the httpbin routetable to true-wildcard

┆Issue is synchronized with this Asana task by Unito

Reproducer project: https://github.com/DuncanDoyle/ge-9830

When you look at the Envoy config dumps, you can see that when the second RouteTable is deployed and selected by the VirtualService, the order of the routes defined in the original RouteTable changes.

One RouteTable:

"dynamic_route_configs": [
  {
    "version_info": "721925422575290896",
    "route_config": {
    "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
    "name": "listener-::-8080-routes",
    "virtual_hosts": [
      {
      "name": "gloo-system_api-example-com",
      "domains": [
        "api.example.com"
      ],
      "routes": [
        {
        "match": {
          "prefix": "/",
          "headers": [
          {
            "name": "city-id",
            "exact_match": "99999"
          }
          ]
        },
        "route": {
          "cluster": "httpbin-httpbin-8000_gloo-system"
        },
        "name": "gloo-system_api-example-com-route-0-matcher-0"
        },
        {
        "match": {
          "prefix": "/get"
        },
        "direct_response": {
          "status": 200,
          "body": {
          "inline_string": "DIRECT1"
          }
        },
        "name": "gloo-system_api-example-com-route-1-matcher-0"
        }
      ],
      "typed_per_filter_config": {
        "envoy.filters.http.ext_authz": {
        "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute",
        "disabled": true
        }
      }
      }
    ]
    },
    "last_updated": "2024-09-10T13:45:20.260Z"
  }
]

Two RouteTables:

"dynamic_route_configs": [
  {
    "version_info": "14667859535387287171",
    "route_config": {
    "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
    "name": "listener-::-8080-routes",
    "virtual_hosts": [
      {
      "name": "gloo-system_api-example-com",
      "domains": [
        "api.example.com"
      ],
      "routes": [
        {
        "match": {
          "prefix": "/get"
        },
        "direct_response": {
          "status": 200,
          "body": {
          "inline_string": "DIRECT1"
          }
        },
        "name": "gloo-system_api-example-com-route-0-matcher-0"
        },
        {
        "match": {
          "prefix": "/anything/test",
          "headers": [
          {
            "name": "city-id",
            "exact_match": "10000"
          }
          ]
        },
        "route": {
          "cluster": "httpbin-httpbin-8000_gloo-system"
        },
        "name": "gloo-system_api-example-com-route-1-matcher-0"
        },
        {
        "match": {
          "prefix": "/anything/rider"
        },
        "direct_response": {
          "status": 200,
          "body": {
          "inline_string": "DIRECT2"
          }
        },
        "name": "gloo-system_api-example-com-route-2-matcher-0"
        },
        {
        "match": {
          "prefix": "/",
          "headers": [
          {
            "name": "city-id",
            "exact_match": "99999"
          }
          ]
        },
        "route": {
          "cluster": "httpbin-httpbin-8000_gloo-system"
        },
        "name": "gloo-system_api-example-com-route-3-matcher-0"
        }
      ],
      "typed_per_filter_config": {
        "envoy.filters.http.ext_authz": {
        "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute",
        "disabled": true
        }
      }
      }
    ]
    },
    "last_updated": "2024-09-10T13:50:01.266Z"
  }
]

solo-io / gloo