This PR configures nightly backups on the production instance.
Basically, there's a cron job that runs the same code we've used for the last year or so when moving databases from one host to another. It then pushes a large, gzipped tarball to bakop.com which you can access via sftp. Backups are stored in a time-stamped directory beneath the hostname from which they're from and are named after the database, for example:
Right now, the backups are to another host in the Dallas datacenter; I originally tried pushing them to a RIMU host in London but ran into some capacity constraints.
The current implementation can backup the production database in about 5 minutes and runs outside of business hours (Malawi business hours to Bellingham business hours, see the comment).
In addition, when the enable_backups.yml playbook is executed, it attempts a full backup. This way if you make a change that breaks something, you'll immediately get a notification.
An idea for future work would be to make the backup script send an email on failure, but that doesn't need to be in place for the migration.
This is in place on prd now, so I can make sure cron is running the job correctly overnight. So there's no rush but I'd love to know what you think.
This PR configures nightly backups on the production instance.
Basically, there's a cron job that runs the same code we've used for the last year or so when moving databases from one host to another. It then pushes a large, gzipped tarball to
bakop.comwhich you can access via sftp. Backups are stored in a time-stamped directory beneath the hostname from which they're from and are named after the database, for example:/home/openboxes/dbprd.pih-emr.org/2023-05-17T23-39-12Z/openboxes.tgzRight now, the backups are to another host in the Dallas datacenter; I originally tried pushing them to a RIMU host in London but ran into some capacity constraints.
The current implementation can backup the production database in about 5 minutes and runs outside of business hours (Malawi business hours to Bellingham business hours, see the comment).
In addition, when the
enable_backups.ymlplaybook is executed, it attempts a full backup. This way if you make a change that breaks something, you'll immediately get a notification.An idea for future work would be to make the backup script send an email on failure, but that doesn't need to be in place for the migration.
This is in place on prd now, so I can make sure cron is running the job correctly overnight. So there's no rush but I'd love to know what you think.