Make player directories persistent outside of Docker

[nswarner]

• [x] Create docker volume(s) to store player directories
• [x] Create an S3 backup procedure for the docker volume
• [x] Create and test a docker volume restoration from S3 procedure
• [x] Create IaC for the S3 backup bucket for the docker volume
• [x] Update the Ansible Playbook for EC2 configuration to automatically create volume(s)
• [x] Update Ansible Playbook for EC2 configuration to restore volumes from S3 bucket

Outstanding questions

• [x] One volume per mud, or just one volume with multiple directories?
• [x] Mount the volume as the player directory

[nswarner]

One volume per mud, easier to segregate, backup, and restore. Volume mounted as /{MUD}/player, works as expected.

[nswarner]

S3 backup procedure thoughts;

• Simple approach: Take volume snapshots of the EC2 instance on intervals.
• Granular but complex approach: Create an S3 bucket, add S3 privileges to the EC2 instance profile, create a cronjob to archive the /usr/lib/docker/volumes/ directory, push the archive to S3. Update the Ansible playbook to create the cronjob.

Contrast:

• Snapshot: Flat $0.40/month
• Snapshot: Includes secrets / unwanted directories
• Snapshot: Includes logs
• Snapshot: Still requires some kind of cronjob to create snapshots, which likely requires additional AWS infrastructure or privileges in IAM for the EC2 instance profile
• S3: Free ( < $0.01 )
• S3: Granular choice of directories to backup
• S3: Easily known pattern to follow
• S3: Can include setup as a part of Ansible Playbook

While the EC2 snapshot approach would be more holistic, it lacks the granular ability to avoid including secrets and requires just as much setup configuration as the S3 approach. The cost of the EC2 snapshot approach is also more than a 40 times cost increase over the S3 approach.

[nswarner]

For the S3 approach:

• [x] Add a new S3 bucket for storing the /usr/lib/docker/volumes/ backups
• [x] Update the IAM Policy for S3 Instance Profile to allow S3 write to the S3 bucket
• [x] Identify an appropriate naming-date schema for the archive names
• [x] Create a cronjob to tar -czvf the /usr/lib/docker/volumes directory, upload to S3 bucket
• [x] Verify the cronjob functions as expected pushing the archive to S3
• [x] Update the Ansible Role for EC2 to create the cronjob
• [x] Create an Ansible Role with tasks to create backups and restore a backup
• [x] Ensure that the restore has a rollback feature if a bad restoration occurs
• [x] Test the restore functionality

[nswarner]

date +%Y-%m-%d-docker-volume-backup -- 2021-04-14-docker-volume-backup for the naming-date schema

[nswarner]

Looks good.