search

NAUbackup / VmBackup

XenServer simple backup script
228 stars 61 forks source link

Feature request: zstd compression #95

Open arthurfreycobraag opened 5 years ago

arthurfreycobraag commented 5 years ago

Would it be possible to add the zstd compression method? I'm using it with xcp-ng 8 and have found that the backups are smaller and quicker compared to gzip or similar compression methods.

See: https://xcp-ng.org/blog/2018/12/19/zstd-compression-for-xcp-ng/ Or: https://github.com/xcp-ng/xcp/issues/66

NAUbackup commented 5 years ago

The only issue there is that it's not compatible with the standard Citrix distribution and would require a separate utility to be made available. I can try to look into how much of a maintenance overhead this might be. I do agree that gzip is not a great utility: zstd, pigz and LZ4 are all better!

BogdanRudas commented 5 years ago

XCP-ng doing the compression within dom0 with zstd, so once you are exporting from XCP-ng you can enable compression and there will be zstd. It was made to decrease network traffic between XOA and XCP-ng hosts. Surely it is possible to export uncompressed data (which is default) and apply any compressor within backup scripts, but if you need capable networking for this or run scripts in dom0 (which is prohibited by Citrix). And once you have capable networking you can have NTFS compressions for Windows-based file servers and lz4 on ZFS or offline de-duplication for XFS.

NAUbackup commented 5 years ago

We do the backup uncompressed (since gzip is so inefficient) and let the host on which the NFS mount resides do all the compression work, which takes a lot of the CPU load off XS/CH. The traffic and backup time will in some cases smaller with in-line compression. It's best to test both to see what works out better in your individual environment.

arthurfreycobraag commented 5 years ago

In our case, zstd compression was the best thing that could happen. The CPUs are always around 5-20%, so no real strain on them. With gzip we completed backups in a timeframe of around 8h. zstd is now almost half that time, 4.5h While pigz was a compromise of around 6h.

As @BogdanRudas said, you'll need capable network to run full backups quickly. But we can only use 2x1Gbps for now. So zstd it is.

One possibility would be to just parse the text to vm-export. Then have a line in the help that says "zstd compression is only supported on xcp-ng >7.6"