Ratelimiting doesn't work with both clientSelectors (i.e. headers and sourceCIDR) enabled

[yash-nisar]

Description:

What issue is being seen? Describe what should be happening instead of the bug, for example: Envoy should not crash, the expected value isn't returned, etc. Ratelimiting doesn't work with both clientSelectors (i.e. headers and sourceCIDR) enabled. I would expect it to work if the headers match and the request is from the IP in the sourceCIDR range.

Repro steps:

Include sample requests, environment, etc. All data and inputs required to reproduce the bug. Followed all steps from quickstart. This is my ratelimit config:

rateLimit:
global:
rules:
- clientSelectors:
- headers:
- name: x-user-id
value: one
sourceCIDR:
type: Exact
value: 0.0.0.0/0
limit:
requests: 3
unit: Hour
type: Global

Note: If there are privacy concerns, sanitize the data prior to sharing.

Environment:

Include the environment like gateway version, envoy version and so on.

Logs:

Include the access logs and the Envoy logs.

These are the envoy-ratelimit logs, note this line descriptor does not match any limit, no limits applied.

time="2024-09-27T00:56:04Z" level=debug msg="CertProvider: waiting for runtime update"
time="2024-09-27T00:56:04Z" level=info msg="Starting xDS client connection for rate limit configurations"
time="2024-09-27T00:56:04Z" level=info msg="Dialing xDS Management Server: 'envoy-gateway:18001'"
time="2024-09-27T00:56:04Z" level=warning msg="connecting to redis on redis.redis-system.svc.cluster.local:6379 with pool size 10"
time="2024-09-27T00:56:04Z" level=debug msg="Implicit pipelining enabled: false"
time="2024-09-27T00:56:04Z" level=warning msg="Listening for debug on '0.0.0.0:6070'"
time="2024-09-27T00:56:04Z" level=warning msg="Listening for gRPC on '0.0.0.0:8081'"
time="2024-09-27T00:56:04Z" level=debug msg="Waiting for config update event"
time="2024-09-27T00:56:04Z" level=warning msg="Listening for HTTP on '0.0.0.0:8080'"
time="2024-09-27T00:56:04Z" level=info msg="Connection to xDS Management Server is successful"
time="2024-09-27T00:56:04Z" level=debug msg="loading domain: default/eg/http"
time="2024-09-27T00:56:04Z" level=debug msg="loading descriptor: key=default/eg/http.httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example_httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example"
time="2024-09-27T00:56:04Z" level=debug msg="Creating stats for key: 'default/eg/http.httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example_httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example.masked_remote_address_0.0.0.0/0'"
time="2024-09-27T00:56:04Z" level=debug msg="loading descriptor: key=default/eg/http.httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example_httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example.masked_remote_address_0.0.0.0/0 ratelimit={requests_per_unit=3, unit=HOUR, unlimited=false, shadow_mode=false}"
time="2024-09-27T00:56:04Z" level=debug msg="Setting config retrieved from config provider"
time="2024-09-27T00:56:04Z" level=info msg="Successfully loaded new configuration"
time="2024-09-27T00:56:04Z" level=debug msg="Waiting for config update event"
time="2024-09-27T00:56:39Z" level=debug msg="got descriptor: (httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example=httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example),(rule-0-match-0=rule-0-match-0),(masked_remote_address=0.0.0.0/0)"
time="2024-09-27T00:56:39Z" level=debug msg="starting get limit lookup"
time="2024-09-27T00:56:39Z" level=debug msg="looking up key: httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example_httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example"
time="2024-09-27T00:56:39Z" level=debug msg="iterating to next level"
time="2024-09-27T00:56:39Z" level=debug msg="looking up key: rule-0-match-0_rule-0-match-0"
time="2024-09-27T00:56:39Z" level=debug msg="looking up key: rule-0-match-0"
time="2024-09-27T00:56:39Z" level=debug msg="descriptor does not match any limit, no limits applied"
time="2024-09-27T00:56:39Z" level=debug msg="starting cache lookup"
time="2024-09-27T00:56:39Z" level=debug msg="returning normal response"

I do see that the config is reflected in xds-ir (in gateway logs):

2024-09-27T00:54:27.139Z        INFO    gateway-api     runner/runner.go:186    {
        "accessLog": {
                "text": [
                        {
                                "path": "/dev/stdout"
                        }
                ]
        },
        "http": [
                {
                        "name": "default/eg/http",
                        "address": "0.0.0.0",
                        "port": 10080,
                        "metadata": {
                                "kind": "Gateway",
                                "name": "eg",
                                "namespace": "default",
                                "sectionName": "http"
                        },
                        "hostnames": [
                                "*"
                        ],
                        "routes": [
                                {
                                        "name": "httproute/default/backend/rule/0/match/0/www_example_com",
                                        "hostname": "www.example.com",
                                        "isHTTP2": false,
                                        "pathMatch": {
                                                "name": "",
                                                "prefix": "/",
                                                "distinct": false
                                        },
                                        "destination": {
                                                "name": "httproute/default/backend/rule/0",
                                                "settings": [
                                                        {
                                                                "weight": 1,
                                                                "protocol": "HTTP",
                                                                "endpoints": [
                                                                        {
                                                                                "host": "10.244.0.10",
                                                                                "port": 3000
                                                                        },
                                                                        {
                                                                                "host": "10.244.0.10",
                                                                                "port": 3000
                                                                        }
                                                                ],
                                                                "addressType": "IP"
                                                        }
                                                ]
                                        },
                                        "metadata": {
                                                "kind": "HTTPRoute",
                                                "name": "backend",
                                                "namespace": "default"
                                        }
                                },
                                {
                                        "name": "httproute/default/http-ratelimit/rule/0/match/0/ratelimit_example",
                                        "hostname": "ratelimit.example",
                                        "isHTTP2": false,
                                        "pathMatch": {
                                                "name": "",
                                                "prefix": "/",
                                                "distinct": false
                                        },
                                        "destination": {
                                                "name": "httproute/default/http-ratelimit/rule/0",
                                                "settings": [
                                                        {
                                                                "weight": 1,
                                                                "protocol": "HTTP",
                                                                "endpoints": [
                                                                        {
                                                                                "host": "10.244.0.10",
                                                                                "port": 3000
                                                                        },
                                                                        {
                                                                                "host": "10.244.0.10",
                                                                                "port": 3000
                                                                        }
                                                                ],
                                                                "addressType": "IP"
                                                        }
                                                ]
                                        },
                                        "traffic": {
                                                "rateLimit": {
                                                        "global": {
                                                                "rules": [
                                                                        {
                                                                                "headerMatches": [
                                                                                        {
                                                                                                "name": "x-user-id",
                                                                                                "exact": "one",
                                                                                                "distinct": false
                                                                                        }
                                                                                ],
                                                                                "cidrMatch": {
                                                                                        "cidr": "0.0.0.0/0",
                                                                                        "ip": "0.0.0.0",
                                                                                        "maskLen": 0,
                                                                                        "isIPv6": false,
                                                                                        "distinct": false
                                                                                },
                                                                                "limit": {
                                                                                        "requests": 3,
                                                                                        "unit": "Hour"
                                                                                }
                                                                        }
                                                                ]
                                                        }
                                                }
                                        },
                                        "metadata": {
                                                "kind": "HTTPRoute",
                                                "name": "http-ratelimit",
                                                "namespace": "default"
                                        }
                                }
                        ],
                        "isHTTP2": false,
                        "path": {
                                "mergeSlashes": true,
                                "escapedSlashesAction": "UnescapeAndRedirect"
                        }
                }
        ]
}       {"runner": "gateway-api", "xds-ir": "default/eg"}

Thanks !

[shawnh2]

Looks weird, can you share the output of egctl config rl -n envoy-gateway-system ?