search

hawtio / hawtio-online

Hawtio on Kubernetes/OpenShift
Apache License 2.0
23 stars 25 forks source link

Hawtio-Online installs are displaying an nginx "429 Too Many Requests" error #272

Open phantomjinx opened 10 months ago

phantomjinx commented 10 months ago

image

Consider how to modify the rate limiting of the web server.

phantomjinx commented 10 months ago

Logs from app:

2023/12/20 06:09:35 [error] 23#23: *58720 limiting connections by zone "limitconnbyaddr", client: 172.17.45.21, server: localhost, request: "GET /maste
r/api/v1/namespaces/openshift-operators/pods?watch=true HTTP/1.1", host: "hawtio-online-hawtio.hawtio-cluster-XXXXX"

172.17.45.21 - - [20/Dec/2023:06:09:35 +0000] "GET /master/api/v1/namespaces/openshift-operators/pods?watch=true HTTP/1.1" 429 564 "-" "Mozilla/5.0 (X1
1; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "10.244.0.5"

2023/12/20 06:09:35 [error] 23#23: *58721 limiting connections by zone "limitconnbyaddr", client: 172.17.11.234, server: localhost, request: "GET /mast
er/api/v1/namespaces/xxxxx-cert-store/pods?watch=true HTTP/1.1", host: "hawtio-online-hawtio.hawtio-cluster-XXXXX"

172.17.11.234 - - [20/Dec/2023:06:09:35 +0000] "GET /master/api/v1/namespaces/xxxxx-cert-store/pods?watch=true HTTP/1.1" 429 564 "-" "Mozilla/5.0 (X11; L
inux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "10.244.0.4"

2023/12/20 06:09:35 [error] 23#23: *58722 limiting connections by zone "limitconnbyaddr", client: 172.17.11.234, server: localhost, request: "GET /mast
er/api/v1/namespaces/hawtio/pods?watch=true HTTP/1.1", host: "hawtio-online-hawtio.hawtio-cluster-XXXXX"

172.17.11.234 - - [20/Dec/2023:06:09:35 +0000] "GET /master/api/v1/namespaces/hawtio/pods?watch=true HTTP/1.1" 429 564 "-" "Mozilla/5.0 (X11; Linux x86
_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
phantomjinx commented 10 months ago

Trying to implement limitations, either through max connections by IP or rate limiting causes parts of the application to fail in different ways. For example, rate limiting to 40 connections / second seems like a good solution (even inserting only in the /online location) yet opening up the app in 5 different tabs kills the 5th tab.

Implementation of any conn/rate limiting is going to require a much more sophisticated approach if it is indeed necessary.

@tadayosi glad for your input on this.

phantomjinx commented 10 months ago

Created https://github.com/hawtio/hawtio-online/pull/273 in the interim to workaround the too many requests errors.

phantomjinx commented 10 months ago

Pushing to subsequent milestone.