search

yourcelf / intertwinkles

A collection of web apps for group decision making and collaboration
BSD 2-Clause "Simplified" License
45 stars 6 forks source link

Monitor and automatically restart dead services #46

Open nightsh opened 10 years ago

nightsh commented 10 years ago

For example, right now the server throws a Nginx error, but the machine itself seems to be working (I received the daily notification, I guess it's a cron job).

A system monitoring components that are listening for connections would be nice. Of course, there are some privileges issues here, it must have some authority over processes on the server (at least the ones owned by the Intertwinkles user).

Disclaimer: though I'm not fully aware of how the whole platform operates, I tried to suggest things based on my very recent observations – server not serving over HTTP, yet other processes still moving in the backend.

(BTW, please restart the dead services now, so we can access the web frontend again)

yourcelf commented 10 years ago

Apologies, as with most outages, this was due to multiple failures. The InterTwinkles process crashed; the supervisor daemon that is supposed to restart it failed to; I was off-grid for a few days and missed the monitoring server's notices; and the backup sysadmin who was watching the server failed to act on the monitoring server's notices.

We're re-evaluating the strategy for checking this in order to prevent future outages.