[snapshots] implement pigz for snapshot archiving

threefoldtech / grid_deployment

Deploy a full Grid backend with docker-compose and snapshots

Apache License 2.0

2 stars 0 forks source link

[snapshots] implement pigz for snapshot archiving #57

Closed coesensbert closed 3 months ago

coesensbert commented 4 months ago

compress a dir tar --use-compress-program="pigz -k " -cf dir1.tar.gz dir1

uncompress pigz -d dir1.tar.gz OR unpigz dir1.tar.gz OR tar -I pigz -xf /mnt/sd/current/backup/bigbackup_web.tar.gz -C /tmp -> this will need adjusting in all backend stack install scripts AND the 'sync processor from 0' scripts before compressing the snapshots with pigz

coesensbert commented 4 months ago

after testing ended up with

compression: tar --use-compress-program="pigz -k --best --recursive | pv " -cf "/storage/rsync-public/tfchain-mainnet-$(date '+%Y-%m-%d').tar.gz" * | pv

decompression: tar -I pigz -xf tfchain-mainnet-best.tar.gz -C extract/ | pv

testing a stack deploy from snapshot to be sure no issues popped up. Will then implement for all nets and apply

coesensbert commented 3 months ago

Big difference in compression time, but archives are about 30% bigger. In total both creating and extracting saves a lot of time so let's go on with it.

coesensbert commented 3 months ago

implemented on all nets