Implement and test borg for backing up model inputs

psrc / queue

Queue: the platform for managing long-running simulation models.

http://psrc.github.io/queue

2 stars 0 forks source link

Closed billyc closed 8 years ago

billyc commented 8 years ago

Need to finalize network location, run scripts, testing...

billyc commented 8 years ago

Did some tests with borg on actual model run OUTPUTS. Using the TransFu runs as tests:

Each run folder is originally around 110Gb
First borged backup is around 55Gb - getting around 50% compression of the data itself
Second borged backup is... around 53Gb.

So, deduplication is: not helping very much when saving outputs. Bummer.

billyc commented 8 years ago

More testing results - excluding matrix files mf--.emx in all databanks:

First archive size drops to 25Gb
Second TransFu 2010 run, which was 175Gb on disk, archive size was 22Gb. Big drop!

Most of the remaining archive size is in the .h5 matrices. Do we need those @Ennazus @bricegnichols ?

billyc commented 8 years ago

Closing because the functionality "works" -- separate issue needed to track soundcast-specific backups.