search

dream11 / kong-scalable-rate-limiter

Kong plugin for Rate Limiting at high throughputs.
https://dream11.github.io/kong-scalable-rate-limiter/
MIT License
37 stars 10 forks source link

Spike in New connections metric after using this scalable plugin #22

Closed pawandhiman10 closed 5 months ago

pawandhiman10 commented 5 months ago

Hello, we recently started using this plugin and we are currently using both the default rate-limiting and scalable-rate-limiter plugins. kong version: 3.4.0 We are using elasticache redis cluster: 7.0.7

In logs we can see errors below:

1aa1eecb5bdc 2024/01/25 07:22:10 [error] 1291#0: *54481 [kong] handler.lua:144 [rate-limiting] failed to get usage: MOVED 13138 10.0.2.81:6379, client: xxxxx, server: kong, request: "POST /api/1 HTTP/1.1", host: "xxxx"
1aa1eecb5bdc 2024/01/25 07:22:10 [error] 1291#0: *51873 [kong] handler.lua:144 [rate-limiting] failed to get usage: MOVED 11814 10.0.2.81:6379, client: xxxxx, server: kong, request: "GET /api/2.json HTTP/1.1", host: "xxxx"
1aa1eecb5bdc 2024/01/25 07:22:10 [error] 1291#0: *460 [kong] handler.lua:144 [rate-limiting] failed to get usage: MOVED 11232 10.0.2.81:6379, client: xxxxx, server: kong, request: "POST /api/3.json HTTP/1.1", host: "xxxx"

Q: Not sure about the above error, this seems to be coming on routes where are using default rate-limiting plugin. Can we not use both rate-limiting and scalable-rate-limiter together?

And the connection errors like below:

430c016e534e 2024/01/24 10:14:07 [error] 1292#0: *50358 [kong] init.lua:97 [rate-limiting] failed to connect to Redis: too many waiting connect operations, client: xxxxx, server: kong, request: "POST /api/ HTTP/1.1", host: "xxxx"

Q: Number of new connections is around 32K. And with the rate-limiting plugin, this number was around 180 max. Should it be this high? Can we not pool to use the existing connections?

Please help.

chirag-manwani commented 5 months ago

The rate-limiting plugin by kong only works with a non clustered Redis, so that is probably the reason why you're getting the above logs.

Try the rate limiting plugin once with a non clustered redis.

pawandhiman10 commented 5 months ago

Seems legit. Will try changing the default plugin and observe. Thank you.