hbz / lobid

Linking Open Bibliographic Data
https://lobid.org/
Eclipse Public License 2.0
15 stars 4 forks source link

Proxy timeout as pay load on one server in the HA-Cluster is high #428

Closed dr0i closed 3 years ago

dr0i commented 3 years ago

Reported by mail M.A., using lobid.org/organisations/:

Fehlerhaftes Gateway!

Der Proxy-Server erhielt eine fehlerhafte Antwort eines übergeordneten Servers oder Proxies.

The proxy server could not handle the request GET /organisations/BVB.json.

Reason: Error reading from remote server

Sofern Sie dies für eine Fehlfunktion des Servers halten, informieren Sie bitte den Webmaster hierüber. Error 502

As we have had a nagios report of high load on one of the server at the same time this seems to be the cause of the Status code 502. Question is, if it's possible to configure the HA-Cluster to also switch server if one of them has a too high load.

dr0i commented 3 years ago

Following https://stackoverflow.com/questions/169453/bad-gateway-502-error-with-apache-mod-proxy-and-tomcat it seems not possible with the simple apache hot spare cluster we have.

But setting the timeout at BalanceMember level andfailontimeout=on at balancer level seems to do the trick:

<LocationMatch ^/organisations/?.* >
ProxyPass balancer://lobid-organisations-cluster nocanon failontimeout=on

I have set the BalancerMember to timeout=7 for organisations. I am not sure if this is really working, though - I used stress to provoke a high load (of 10 (while having only 8 CPUs)), but the server responds very quickly none-the-less...

I added that as an answer to the SO question.