Backup Jenkins

[jacderida]

Hi Stephen,

I've edited this post to instead advise using the qa environment rather than staging. This will actually be a better test, since the macOS slave has already been provisioned for staging. We need to run that same provisioning on prod, so this will be a good test to make sure that's working.

This implements a backup for Jenkins. The entire Jenkins home directory is put in a tar.gz and uploaded to S3 at safe-jenkins-backups.

Instructions

• Spin up the environment with make env-jenkins-qa-aws
• SSH to the Bastion
• Edit ansible/roles/jenkins-master/tasks/main.yml and change the hour of the cron job for backup task
• Provision the environment with make provision-jenkins-qa-aws-initial
• Wait for this to complete and all the machines to be rebooted
• Run make provision-rust_slave-macos-high_sierra-x86_64-qa-aws
• Log in to Jenkins
• See the new documentation in the 'Configure Jenkins' section of the README for setting up the SSH service
• Wait for the cron job to kick off
• Refresh your browser to see Jenkins go offline
• Refresh your browser to see Jenkins come back online
• Wait for 2 - 5 minutes for the macOS slave to fully reconnect

Important note: the macOS slave may go offline again after it comes online, but if you leave it and look at the logs it should reconnect again after a while.

Acceptance Criteria

• The macOS slave should provision without any errors
• The macOS slave should be provisioned with a qa configuration for WireGuard (run ifconfig to see it running)
• The macOS slave should have a cron job under the qamaidsafe user that runs a ping against the qa WireGuard network (use crontab -e to see what's on the crontab - though you must be the qamaidsafe user)
• Jenkins should come back online after the backup
• A backup with a timestamp should be created in safe-jenkins-backups
• The macOS slave should be reconnected to the environment permanently (after possibly being disconnected once at the start)
• The backups in the S3 bucket should not be publicly available as the Jenkins directory can potentially contain secrets

Cheers,

Chris

[S-Coyle]

• [x] The macOS slave should provision without any errors
• [x] The macOS slave should be provisioned with a qa configuration for WireGuard (run ifconfig to see it running)
• [x] The macOS slave should have a cron job under the qamaidsafe user that runs a ping against the qa WireGuard network (use crontab -e to see what's on the crontab - though you must be the qamaidsafe user)
• [x] Jenkins should come back online after the backup
• [x] A backup with a timestamp should be created in safe-jenkins-backups
• [x] The macOS slave should be reconnected to the environment permanently (after possibly being disconnected once at the start)
• [x] The backups in the S3 bucket should not be publicly available as the Jenkins directory can potentially contain secrets