search

JsBergbau / Verschiedenes

Sammlung verschiedener Dinge
3 stars 0 forks source link

gzip --rsyncable and resulting backup file reduction with restic #2

Open JsBergbau opened 3 years ago

JsBergbau commented 3 years ago

Method: 55 MySQL dumps of a productive moodle systems each almost 3 GB in size, total 150.8 GB, every 24 hours created via mysqldump --defaults-file=mysqldumpCred -h localhost -C -Q -e --create-options --single-transaction --all-databases > dump.txt

Adding these directly to a restic repository results in 41.1 GB repo

repository d38300ef opened successfully, password is correct
created new cache in /root/.cache/restic
no parent snapshot found, will read all files

Files:          54 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added to the repo: 41.075 GiB

processed 54 files, 150.792 GiB in 13:19
snapshot c1562c80 saved

Zipping these files individually via gzip results in 26.6 GB files and when added to the repo also 26.6 GB

repository b4495dba opened successfully, password is correct
created new cache in /root/.cache/restic
no parent snapshot found, will read all files

Files:          54 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added to the repo: 26.582 GiB

processed 54 files, 26.598 GiB in 2:34
snapshot 8a6e719c saved

Last try is each MySQL dump zipped via --rsyncable option. This gives a slightly larger total filesize of 27.3 GB but leads to only 8.57 GB in the repo.

repository bb714050 opened successfully, password is correct
created new cache in /root/.cache/restic
no parent snapshot found, will read all files

Files:          54 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added to the repo: 8.570 GiB

processed 54 files, 27.286 GiB in 1:53
snapshot 93a487b8 saved

It makes no difference for deduplication if you use cat mysqldump.txt | gzip --rsyncable > viaPipe.txt.gz or gzip --rsyncable mysqldump.txt So gzip won't sort the file before compressing it or stuff like that. 

ls -l
total 1076392
-rw------- 1 root root 551112323 Aug  6 13:48 mysqldump.txt.gz
-rw-r--r-- 1 root root 551112307 Aug  6 17:58 viaPipe.txt.gz
ls -lh
total 1.1G
-rw------- 1 root root 526M Aug  6 13:48 mysqldump55.txt.gz
-rw-r--r-- 1 root root 526M Aug  6 17:58 viaPipe.txt.gz
repository 0e9bed5e opened successfully, password is correct
created new cache in /root/.cache/restic
no parent snapshot found, will read all files

Files:           2 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added to the repo: 526.281 MiB

processed 2 files, 1.027 GiB in 0:03
snapshot d7d07bb8 saved