automated database backups

[michaelkirk]

need to capture them regularly

[michaelkirk]

OK - getting close here. There are automated backups in staging, the code is here: https://github.com/jackpine/biketag-ops/blob/master/services/biketag-db-backup.service

Go systemd!

[michaelkirk]

still going strong on staging, however, forgot to reload the new backup-job on production after deploying the new systemd script.

Debugged and fixed thus:

# Here we can see the job has been starting once a day, so the timer seems to be working, it's just that the job is failing.
core@biketag-production ~ $ journalctl -u biketag-db-backup                                                                                                   
-- Logs begin at Wed 2015-06-10 14:38:00 UTC, end at Sat 2015-06-13 00:23:06 UTC. --
Jun 11 00:00:03 biketag-production systemd[1]: Started BikeTag DB Backup.
Jun 11 00:00:03 biketag-production systemd[1]: Starting BikeTag DB Backup...
Jun 11 00:00:03 biketag-production systemd[1]: biketag-db-backup.service: main process exited, code=exited, status=203/EXEC
Jun 11 00:00:03 biketag-production systemd[1]: Unit biketag-db-backup.service entered failed state.
Jun 11 00:00:03 biketag-production systemd[1]: biketag-db-backup.service failed.
Jun 12 00:00:00 biketag-production systemd[1]: Started BikeTag DB Backup.
Jun 12 00:00:00 biketag-production systemd[1]: Starting BikeTag DB Backup...
Jun 12 00:00:00 biketag-production systemd[1]: biketag-db-backup.service: main process exited, code=exited, status=203/EXEC
Jun 12 00:00:00 biketag-production systemd[1]: Unit biketag-db-backup.service entered failed state.
Jun 12 00:00:00 biketag-production systemd[1]: biketag-db-backup.service failed.
Jun 13 00:00:04 biketag-production systemd[1]: Started BikeTag DB Backup.
Jun 13 00:00:04 biketag-production systemd[1]: Starting BikeTag DB Backup...
Jun 13 00:00:04 biketag-production systemd[1]: biketag-db-backup.service: main process exited, code=exited, status=203/EXEC
Jun 13 00:00:04 biketag-production systemd[1]: Unit biketag-db-backup.service entered failed state.
Jun 13 00:00:04 biketag-production systemd[1]: biketag-db-backup.service failed.

# Here we can see the job fails even when started manually (with a helpful error message)
core@biketag-production ~ $ sudo systemctl start biketag-db-backup                                                                                            
Warning: biketag-db-backup.service changed on disk. Run 'systemctl daemon-reload' to reload units.

# I previously modified the service file to reflect the new location of the bin/archive-db script
# I didn't realize this required reloading the unit.
core@biketag-production ~ $ cat /etc/systemd/system/biketag-db-backup.service 
[Unit]
Description=BikeTag DB Backup

[Service]
TimeoutStartSec=0
ExecStart=/home/core/biketag-ops/bin/archive-db 

# Here's the fix
core@biketag-production ~ $ sudo systemctl daemon-reload                                                                                                      

# Verify I can run it manually, no errors and saw the archive appear in s3
core@biketag-production ~ $ sudo systemctl start biketag-db-backup  

So, I'll check again tomorrow, but I expect this to work.

[michaelkirk]

finally verified that this works.