rcpch / rcpch-audit-engine

Epilepsy12 Audit Platform
https://e12.rcpch.ac.uk/
GNU Affero General Public License v3.0
5 stars 5 forks source link

Disaster recovery database backup #834

Open mbarton opened 7 months ago

mbarton commented 7 months ago

As we're the "system of record" for audit data we should both keep backups of the database (point in time rollback, VM backup etc) as well as having a disaster recovery backup.

I'm not sure what that should be at the moment: database dump in Azure Blob Storage in a different region seems like a decent starter. We can lose the separate region if there's no other Azure region we can store data in, although we may want to store in another cloud provider then.

We're not really expecting Azure to lose a region or lose our data, it's more likely something goes wrong with our account or we accidentally delete everything. But the mechanism is still the same.

pacharanero commented 7 months ago

Fully agree @mbarton - I had sketched out a similar proposal a while back in #663 For a previous, non-RCPCH project I had a versioned AWS S3 bucket which I simply pushed new versions of the DB dump to. The script ran on cron on a daily basis (on the same VPS that the webapp ran on), but we could equally use a lambda or other serverless infra.

Here's what I had working for several years, in case it's of help/accelerates things:

#!/usr/bin/env bash

# get the backup over HTTPS with password
curl -v -X POST \
  -F 'master_pwd=<PASSWORD> \
  -F 'backup_format=zip' \
  -o /home/odoo/odoo-backups/odoo-back-up.zip \
  https://odoo.example.com/web/database/backup

# send dump to aws
/usr/local/bin/aws s3 cp /home/odoo/odoo-backups/odoo-back-up.zip s3://uniquely-named-versioned-backup-s3-bucket

# delete local sql dump
rm /home/odoo/odoo-backups/odoo-back-up.zip

# email notification of log
curl -s --user 'api:key-APIKEY-FOR-EMAIL-NOTIFICATION' \
    https://api.mailgun.net/v3/USERNAME/messages \
    -F from='Admin Notifier <notification@example.com>' \
    -F to=adminemail@example.com \
    -F subject='Odoo Backup Notifier & Log' \
    -F text="Odoo SQL Backup completed and saved to AWS. Check AWS bucket at https://s3.console.aws.amazon.com/s3/buckets/uniquely-named-versioned-backup-s3-bucket/?region=eu-west-2 to view status"
mbarton commented 7 months ago

We would only do this after #832 as that will involve changing databases back to hosted postgres