Closed mgeerdsen closed 4 years ago
@mgeerdsen we want to be reassured and have seen your plans for restoring before we are using AWS Goobi in production
datastores involved
ressources to be included in backup are bold
For EFS it appears that AWS Backup can be a good solution. Easy to set up, allowing automatic migration to cold storage, restore into the same FS as well as restore as new FS. A regular command line backup tool running as some kind of scheduled job, could also do the job of backing up the files to S3.
RDS could be backed up using RDS snapshots (increasing the current amount of snapshots saved). A different or even additional way would be a kind of scheduled job doing regular SQL dumps to S3 for example.
S3 is a bit harder, the highest risk here is probably accidental deletion by a user. This could be caught by enabling versioning and automatically dropping old versions after a defined time.
The above measures protect, to a certain limit, against accidental loss of data due to some kind of user or script error for example. For better protection against infrastructure problems, replication of backups into a different region would have to be considered.
EFS it appears that AWS Backup can be a good solution 👍 RDS could be backed up using RDS snapshots 👍 Versioned S3 👍
Great - all sounds good.
That all makes sense - do we need to make each of those mechanisms into a ticket to be worked on?
EFS and RDS are saved daily... restore to a new resource for testing purposes is currently running
restoring RDS/EFS to new resources has been tested and content checked successfully
Following components might be considered:
RDS DB containing workflow status, harvested processes etc.
EFS containing METS files
S3 bucket containing configuration
S3 bucket containing images etc.
List the data stores and whether they are rebuildable from other data
For DB dumps we should keep a couple of weeks and yes we should replicate into another region (though cost and time are factors we need to know roughly).
Wellcome would expect us to be able to bring up a staging environment replicate backups into that to demonstrate continuity of service
Wellcome usually keep images for 15 days before deleting them from Goobi so it'd be good to keep the images in the backup just as long or a little longer