Monitoring Epic

[acozine]

Develop requirements for monitoring.

Research product options. Our ideal product:

• is affordable and/or open-source
• is mature / has a track record
• monitors linux and macos and windows operating systems
• monitors for hardware issues (bad disks, power systems, network cards, etc.)
• monitors on-prem and cloud resources
• sends alerts via slack, email, SMS
• sends timely alerts that reflect the progression of the problem (no alert fatigue)
• is sensitive enough to provide good warnings
• also sends alerts/notifications of recoveries
• we can automate installation of all elements of the system (server, agent if relevant, other elements)
• we can automate additions/deletions/updates to the configuration (what gets monitored, at what level, etc.)
• all configuration is under source control
• incorporates role-based access controls, so everyone in the library gets appropriate views and alerts
• uses SSO in some form, preferably Shibboleth, for the GUI
• provides a good GUI for user views
• provides a CLI for maintenance of the system itself - restarting, etc.

Develop list of services, etc., to monitor.

[acozine]

Current front-runner for product is Icinga. It's based on Nagios, so it's familiar. It's open-source. Not sure if it can monitor cloud resources.

Other options:

• CheckMK. can monitor cloud resources with a paid account. We could do a free trial to test it out.
• Centreon only runs on RHEL.
• Prometheus is cloud-native.
• Zabbix uses Java, we have no local expertise there.

[acozine]

Related documents and spreadsheets on Google Drive:

• RDSS monitoring goals
• Sensu needs spreadsheet
• Monitoring ecosystem spreadsheet
• Sensu notes document
• Notes from Monitoring meeting - unknown date

[acozine]

Related to the following post-incident reviews:

• abid 2023-08-03
• approvals cert 2023-08-03
• Papers of Princeton 2023-08-01
• approvals 2023-06-30