Backup selected S3 buckets using AWS Backup

[carrb-moj]

Service name

Manage Intelligence (IMS)

Service environment

• [X] Dev / Development
• [X] Staging
• [X] Prod / Production
• [ ] Other

Impact on the service

Provide real impact description on the service mentioned. It can include any potential blockers for the product team.

The IMS project is delivering a replacement to the legacy Mercury system with a new IMS system hosted in Cloud Platform. As part of the platform we need to store images and attachments uploaded by users in S3 when they submit intelligence reports. There is a requirement to backup these S3 objects and to do that, we'd like to use AWS Backup.

Problem description

We need to be able to backup the objects in specific S3 buckets as these objects must be recoverable should they somehow get deleted.

Contact person

Brian Carr, brian.carr@digital.justice.gov.uk

[tom-j-smith]

Link to branch Current changes made to s3 module:

• Added the ability to take backups using AWS Backups, enabled by setting enable_backup to true, and can use backup_schedule, and backup_retention_days to configure. When enabled it creates a backup vault (matching the resource name), IAM role with relevant policies attached required for backup and restore of s3 bucket, the backup plan, and the resource selection.
• Fixed an issue where private was being passed as default for ALCs but ACLs were not enabled in buckets. ACLs are now enabled by default by aws_s3_bucket_ownership_controls being set to BucketOwnerPreferred. (To restore a bucket from a backup ACLs are required to be enabled)
• Implemented a variable disable_acl (bool), to disable ACLs.
• Added handling of versioning as a local variable instead of a variable as Backups require versioning to be enabled in the bucket
• aws_s3_bucket_server_side_encryption_configuration is now in its resource block as the option in aws_s3_bucket is deprecated.

[tom-j-smith]

Still needs to be discussed with members of CP:

• Are AWS Service default keys fine, or should each vault have its own unique CMK?
• Are ACLs being set to BucketOwnerPreferred by default fine? and should we allow the option to disable ACLs?
• Who manages restores and the process for doing them?

[tom-j-smith]

Spoke to Brian Carr and agreed that he can use the S3 module on a branch that can enable backups with some stipulations. The solution is not "production ready" as the restore solution still needs to be worked on - follow on story created to look into this: #6039 Currently, if a restore is required then it will require manual assistance from someone on CP.