tod /crusader backup filesystem full

[jdimatteo]

Related to this, we should complete https://github.com/BradnerLab/SystemAdmin/issues/3 .

(Note that I mistakenly created this as a pipeline issue when I meant to create it as a SystemAdmin issue.)

[jdimatteo]

Clearing some space now, by removing the backups we disabled last December:

root@tod:/crusader/backup/rdiff-backup# rm -rf ifs root@tod:/crusader/backup/rdiff-backup# rm -rf mnt

(Still running)

This should allow tonight's backup to proceed normally.

[jdimatteo]

After we get #3 done, we can do something like the following to remove files/diffs older than 1 month:

rdiff-backup --remove-older-than 1M --force /backups

[jdimatteo]

root@tod:/crusader# df -h .
Filesystem                   Size  Used Avail Use% Mounted on
172.24.209.62:/volume1/data   17T   15T  2.0T  89% /crusader
root@tod:/crusader# 

[jdimatteo]

I changed tod:/root/backup-scripts/backup-grail.sh

from

rdiff-backup --backup-mode --exclude-other-filesystems /grail/ /crusader/backup/rdiff-backup/grail

to

rdiff-backup --backup-mode --exclude-other-filesystems --include /grail/bam --include /grail/projects --include /grail/TONY --exclude /grail /grail/ /crusader/backup/rdiff-backup/grail

This will prevent directories like /grail/raw from being backed up in the future. When we run the rdiff-backup --remove-older-than thing we can delete /crusader/backup/rdff-backup/grail/raw

[jdimatteo]

The grail and root rdiff-backups keep failing during regressing.

Trying to correct based off suggestions at http://www.nongnu.org/rdiff-backup/FAQ.html#regress_failure :

root@tod:/crusader/backup/rdiff-backup/grail/rdiff-backup-data# mv current_mirror.2014-10-21T03:00:08-04:00.data ~/gunk/

root@tod:/crusader/backup/rdiff-backup/root_directory/rdiff-backup-data# mv current_mirror.2014-10-23T03:00:10-04:00.data ~/gunk/

There were no mirror_metadata files with a later date, so I didn't remove those like the faq suggested.

[jdimatteo]

The last time I fixed the grail backup it failed to complete because TOD was restarted in the middle of the backup. It seems like it needs to finish a full backup before it can recover itself. I just did the suggested fix again and hopefully the backup will finish. It might take a couple days to finish since it has to do extra work to recover from the prior interrupted backup.

root@tod:/crusader/backup/rdiff-backup/grail/rdiff-backup-data# mv current_mirror.2014-10-23T03\:00\:10-04\:00.data ~/gunk/current_mirror.2014-10-23T03\:00\:10-04\:00.data-2 

[jdimatteo]

Backups back to normal as of January 13th using the /mnt/bradner/data for grail.