search

python / psf-salt

PSF infrastructure configuration
MIT License
111 stars 57 forks source link

Debug nginx rate limiting for bugs role #368

Closed ewdurbin closed 1 month ago

ewdurbin commented 2 months ago

Ostensibly we are rate limiting access to our roundup instances in the bugs role.

But it appears that our limits are either not working, or are mistuned.

We should debug why rate limits aren't keeping us from seeing over-load outages like the one on July 7.

JacobCoffee commented 1 month ago

It appears rate limiting is working...but its the load-balancers that are rate limited. Logs indicate that lb-a and lb-b are being limited via their internal IPs

2024/07/25 19:55:07 [error] 2527603#2527603: *40848 limiting requests, excess: 5.110 by zone "limit-cpython", client: 10.132.141.7, server: bugs.python.org, request: "GET /issue16136 HTTP/1.1", host: "bugs.python.org"
2024/07/25 19:55:08 [error] 2527603#2527603: *40855 limiting requests, excess: 5.650 by zone "limit-cpython", client: 10.132.68.69, server: bugs.python.org, request: "GET /issue46282 HTTP/1.1", host: "bugs.python.org"