Critical backups should be kept "off cluster" (Echo/S3)

[alanbchristie]

At the moment DB backups utilise additional storage provided in the cluster by NFS but to be extremely robust against disk failures we should keep them under separate control, certainly outside the cluster.

We used to SSH/Rsync data to a remote server, we should revisit this and make sure crucial backups are located in "off cluster". These would include the following production cluster namespace Databases: -

• im-infra
• v2-production-stack
• staging-stack

There are DBs on the developer cluster might include: -

• im-infra
• im-infra-test

[mwinokan]

@alanbchristie @tdudgeon please prioritise this within the swim-lane based on the panic-level