opencrvs / opencrvs-core

A global solution to civil registration
https://www.opencrvs.org
Other
81 stars 62 forks source link

Track backups #3929

Open euanmillar opened 1 year ago

euanmillar commented 1 year ago

Not deployed with the same swarm

Ansible script for the backup server Deploy scripts for the backup server

The service would act as a free version of Pingdom, it calls the gateway health check endpoint, hosted on the backup server and will email the sys admin if anything is down. Email them if it comes up again.

Create some test files on another server which you can SSH into. The files should be named like this:

$ROOT_PATH/backups/minio/${$BACKUP_DATE} - directory exists $ROOT_PATH/backups/metabase/${$BACKUP_DATE} - directory exists $ROOT_PATH/backups/influxdb/${$BACKUP_DATE} - directory exists $ROOT_PATH/backups/mongo/hearth-dev-${$BACKUP_DATE}.gz - file exists $ROOT_PATH/backups/mongo/user-mgnt-${$BACKUP_DATE}.gz - file exists $ROOT_PATH/backups/mongo/openhim-dev-${$BACKUP_DATE}.gz - file exists $ROOT_PATH/backups/mongo/application-config-${$BACKUP_DATE}.gz - file exists $ROOT_PATH/backups/mongo/metrics-${$BACKUP_DATE}.gz - file exists $ROOT_PATH/backups/mongo/webhooks-${$BACKUP_DATE}.gz - file exists $ROOT_PATH/backups/mongo/performance-${$BACKUP_DATE}.gz - file exists

Create a shell script in the scheduler microservice which will SSH into the server and check that the files exist, then prepare a JSON payload like this:

[{ "file": "influx", "exists": true, "date": "2022-08-11" }]

Set up a database in the healthcheck service Create a route to receive the JSON payload and store in the database Create UI to query the database and show the existence of the files historically to the user

euanmillar commented 10 months ago

Blocked until the backup server is set up