Closed gaurav closed 1 year ago
After some intesive load testing on our servers in a test enviroment we have the following upcoming changes:
Next steps: In ITRB enviroment, we have redis cluster setup as our storage. The python driver in this webserver for redis-cluster is not currently async. We would want to refactor that to use async redis-cluster library. This will help remove the blocking. The changes above would reduce the 50x error but having async would be a big lift too.
It looks like has been fixed now (as confirmed by @cbizon's UptimeRobot at https://stats.uptimerobot.com/g9MlwHqXOy). Closing.
On Translator Slack, people have reported NodeNorm requests failing with 503 Service Unavailable or 504 Gateway Timeout. This may be caused by Redis issues (where requests take too long or time out), NodeNorm issues or Nginx issues on ITRB.
I think we should set up a stress-test system for NodeNorm which makes a large number of simultaneous queries to see if we can replicate this issue (probably on RENCI or on NodeNorm ITRB Test instead of NodeNorm ITRB Prod), and then look at the logs to see if we can figure out why this is happening.