CPANTesters Monitoring

[preaction]

Rather than having random people e-mail us when things are broken, we should have an automated monitor in-place. This means setting up another machine in another data center and installing/configuring a monitoring system like Nagios, Zabbix, Icinga, or Shinken.

Known issues to monitor immediately:

• Machine is alive
• Website is alive
• Disks are not full
  • /media/backend has at least 50G free
• Backup is running
  • /media/backend/cpantesters/dbx has backup files from this week
• Reports release is running
  • MySQL tables cpanstats.release_data and cpanstats.release_summary have MAX(id) that is not too far out of sync with MAX(id) from cpanstats.cpanstats

[preaction]

I've installed Icinga2 on a new machine, monitor.preaction.me. I've started a config file for cpantesters and set up a single host-alive check. I will continue with the rest of these checks soon.

[preaction]

Icinga2 is really difficult to manage, so we're going to try Graphite+Grafana for now to see if it works easier.