search

kobotoolbox / kobocat

Our (backend) server for providing blank forms to Collect and Enketo and for receiving and storing submissions.
BSD 2-Clause "Simplified" License
117 stars 127 forks source link

Fix KoBoCAT issue when the app is not responding anymore even under low load #834

Closed noliveleger closed 2 years ago

noliveleger commented 2 years ago

Description

Fix 502/504 errors that KoBoCAT starts to return after running few hours/days even if the load is very low.

Additional info

502/504 can still happens under high load, obviously. This PR addresses a particular case: uWSGI locks and it is not able to respawn new workers (thus not able to serve any requests anymore). It happens because of deadlocks in Python. See https://github.com/unbit/uwsgi/issues/1599#issuecomment-336584282 for more details.

Moreover, to avoid respawning many workers very often, the cheaper algorithm has been replace with busyness (instead spare). It does respond better under high load.

Related issues

Fixes kobotoolbox/kpi#3566 Related to kobotoolbox/kpi#3883

bufke commented 2 years ago

The CI failure is due to the branch name being an invalid docker tag.