Move backup script to run from backup servers

[drn05r]

On our ongoing project to remove funcgtionality from auth2, we should change the way backups work so that the backup servers rsync onto them and them tarball up a snapshot each day. This would reduce our reliance on auth2 and it would allow backups to run in parallel rather than in series.

Doing this previous was not easily do-able as the backup server lack CPU performance and only had 100Mbps network cards but this is no longer an issue.

We should restrict the keys on each server to only be able to rsync from the IP address of the backup server by specifying this in the authorized_keys file. We should have a different keypair for each backup server.

[TimStallard]

41 and #48 are two other things that we could include while working on backups.

Probably want to config manage the backup scripts somehow - could just deploy them via ansible, and include in that repo?

I also notice the backup log on auth2 is huge (close to a GB), so we should add logrotate config for that on the backup servers when done.

I suppose we could even look at generating the config automatically from netbox, rather than hardcoding everything, and maybe rsync entire rootfses with specific exclusions now that we have the space, would make restores far less painful.

[TimStallard]

ZFS + snapshots might also be worth considering - would let us store a load more historical copies of data, without needing the extra disk space for multiple copies.

[TimStallard]

One other thing while I'm thinking about backups, the passive checks should move over to monitor2.

[TimStallard]

I've started working on a python backup script in sown/backup. Got most of the basics going, creating zfs filesystems, rsyncing, snapshotting, talking to icinga for passive checks.

Still needs some thought for MySQL backups etc. I was pondering a directory on each server (eg /etc/backup-scripts/) that we run with run-parts before the backup (can be done easily enough via paramiko, we already have an SSH connection for reading the excludes file). Would make it easy to do #41 and #48 as well.

Would need to adapt MySQL playbook to deploy a backup script (basically just the existing mysqlbackup script with a specific path).

[drn05r]

This makes sense. Have one for mysql, one for postgres and one for dpkg and then maybe further ones if we have other applications it is useful to dump out before backing up. These backup scripts can then be deployed using an Ansible playbook.

It is probably that what backup scripts run on what server is determined in the config on the backup server itself rather than having to have different sets of backup scripts deployed on each server. Then we only need one Ansible script to do this deployment.

[TimStallard]

It is probably that what backup scripts run on what server is determined in the config on the backup server itself rather than having to have different sets of backup scripts deployed on each server. Then we only need one Ansible script to do this deployment.

Hmm - I was thinking putting the scripts on each server and deploying as part of that service playbook (much like we do with mysqlbackup now) would let us keep the state of what to run on servers themselves, rather than duplicating between backup2/3.

That would make it much easier to write one-off backup scripts specific to a server as well (eg just a little script on one server to dump the switch config for #48), and it would be automagically included in the standard backups and monitoring.

[TimStallard]

This is all done and deployed on backup2 and backup3 now. I've disabled the old cron job on auth2 and the old passive monitors.