openstreetmap / operations

OSMF Operations Working Group issue tracking
https://operations.osmfoundation.org/
98 stars 13 forks source link

Use StatusCake Heartbeat / Push Monitoring in backup scripts #955

Open Firefishy opened 1 year ago

Firefishy commented 1 year ago

The OSM backup scripts should have better status monitoring.

An option is for the backup script on completion to call a StatusCake Push Monitoring endpoint with a "heartbeat". Each backup will have its own StatusCake Push Monitoring setup / endpoint.

The StatusCake Push Monitoring would alert if a heartbeat hasn't been called in a to-be-defined interval.

tomhughes commented 1 year ago

We can probably get the filestat exporter to monitor the backup directory and use prometheus to alert if things are too old.

Firefishy commented 8 months ago

Added initial test heartbeat code in https://github.com/openstreetmap/opentofu-statuscake/blob/main/heartbeat.tf Updates to backup script is pending, needs decision on what I do with the heartbeat ping URL, likely is a secret.