Backup of cloud system

[mgeerdsen]

Following components might be considered:

• RDS DB containing workflow status, harvested processes etc.

• EFS containing METS files

• S3 bucket containing configuration

• S3 bucket containing images etc.

• List the data stores and whether they are rebuildable from other data

• For DB dumps we should keep a couple of weeks and yes we should replicate into another region (though cost and time are factors we need to know roughly).

• Wellcome would expect us to be able to bring up a staging environment replicate backups into that to demonstrate continuity of service

• Wellcome usually keep images for 15 days before deleting them from Goobi so it'd be good to keep the images in the backup just as long or a little longer

[pollecuttn]

@mgeerdsen we want to be reassured and have seen your plans for restoring before we are using AWS Goobi in production

[mgeerdsen]

datastores involved

• RDS databases:
  • goobi
  • information stored: account information, workflow status and history
  • other sources: none
  • harvester:
  • stored information: information about already harvested processes, repository URLs, past runs
  • other sources: process list could potentially be re-created from storage server or presentation system (DB backup a lot easier)
  • itm
  • information stored: status and historical data of ITM jobs
  • other sources: partially from Goobi
• EFS:
  • stored information: METS files and potentially additional files of Goobi processes, working storage
  • other sources: many METS files can be re-created from archive storage, but only of finished/exported processes
• S3:
  • wellcomedigitalworkflow-workflow-configuration
  • stored information: configuration files for applications, Goobi rulesets etc.
  • other sources: none
  • wellcomedigitalworkflow-workflow-data
  • stored information: images etc. of Goobi processes (mainly unfinished/WIP processes)
  • other sources: at least for older unfinished processes probably none (old scanned images e.g.)
  • wellcomecollection-workflow-infra
  • stored information: terraform state and vars, lambdas archivematica infra files?
  • other sources: partially none or re-creatable from scratch, lambda from github, terraform variables might need backup
  • wellcomecollection-workflow-export-bagit / workflow-export-bagit-stage
  • stored information: bags for ingests (working storage)
  • other sources: should be re-creatable from Goobi until successfully ingested
  • workflow-harvesting-results
  • stored information: harvested MARC files (successfully imported as well as problematic)
  • other sources: re-downloadable from IA, only source for processes that have not been successfully imported
  • workflow-upload
  • stored information: images for upload to Goobi (working storage)
  • other sources: scanning PCs?

ressources to be included in backup are bold

[mgeerdsen]

For EFS it appears that AWS Backup can be a good solution. Easy to set up, allowing automatic migration to cold storage, restore into the same FS as well as restore as new FS. A regular command line backup tool running as some kind of scheduled job, could also do the job of backing up the files to S3.

RDS could be backed up using RDS snapshots (increasing the current amount of snapshots saved). A different or even additional way would be a kind of scheduled job doing regular SQL dumps to S3 for example.

S3 is a bit harder, the highest risk here is probably accidental deletion by a user. This could be caught by enabling versioning and automatically dropping old versions after a defined time.

The above measures protect, to a certain limit, against accidental loss of data due to some kind of user or script error for example. For better protection against infrastructure problems, replication of backups into a different region would have to be considered.

[kenoir]

EFS it appears that AWS Backup can be a good solution 👍 RDS could be backed up using RDS snapshots 👍 Versioned S3 👍

Great - all sounds good.

[kenoir]

That all makes sense - do we need to make each of those mechanisms into a ticket to be worked on?

[mgeerdsen]

EFS and RDS are saved daily... restore to a new resource for testing purposes is currently running

[mgeerdsen]

restoring RDS/EFS to new resources has been tested and content checked successfully