Built-In Redundancy Monitoring for GoAlert Instances

[mastercactapus]

What problem would you like to solve? Please describe: There is a need for a feature to facilitate the monitoring of a GoAlert production instance employing distinct installations of GoAlert. While certain elements of this process are partially implemented, they are not documented, complex, and possibly outdated.

Describe the solution you'd like: A new admin page for Remote Monitor to streamline the configuration and operation of these monitoring services. Features required:

• Auto-creation of a new monitor service with a webhook, pointing back to the main instance.
• Auto-sync or manual sync options for updating the monitor environments from the primary GoAlert instance.
• Enable the syncing of on-call users from the primary instance to the monitor installations. Consider a solution for user contact method validation since notifications may come from a new number.

Describe alternatives you've considered: The current procedure involves a command, 'monitor', and the 'remotemonitor' package, to create and manage added complexity redundancy measures.

Additional context: This feature request caters primarily to the admins responsible for a GoAlert installation rather than the application's end users.

[mastercactapus]

A good experience will rely heavily on #3007

[mastercactapus]

Current state reference info

Current check sequence


Current Check Deployment Example


---

Worth noting that the current operation requires remote monitor to be deployed with a config file with various required settings, it's own twilio number, and be publicly-routable

[mastercactapus]

For next steps:

MVP: minimal API/db additions to allow create alert -> heartbeat using webhook on an EP

+1: auto sync on-call users +1: auto config from production/main instance

+2: "sentinel mode" -- slimmer UI, warning banner, make it obvious an instance is for monitoring another (with link) +3: first time setup /w sentinel

Separate supporting feature ideas:

• admin-only resources (i.e., don't allow regular users to change, or possibly see, a monitoring/synced service)
• app config to set the default theme color (help differentiate separate envs)
• https://github.com/target/goalert/issues/3024

---

Implementation thoughts/notes:

• have sentinels do as much work as possible (PULL config rather than push for on-call)
• synthetic check: can I create an alert? did I get a notification? -> POST heartbeat(s)