Add option to compress VHD files

[rudolfmayer]

I have a machine where I only export the primary HDD on a regular basis, thus make no full export of the machine. This works nicely with the VDI export option (thanks for that!).

As these VHD images take a quite a lot of space, adding an option to compress them would be great (e.g. my disk image is around 20GB uncompressed, but only ~4 GB compressed. As I keep them in a rotation for a longer period, that adds up to quite some space quickly.) Therefore, I now manually run a gzip of VHD files in my backup folder, but if there is an option to integrate this in NAU/VmBackup, that would be highly appreciated.

[NAUbackup]

There is already such an option and it is documented in the README.md file: there is a "compress=true" option present that you can put on the command line (the default is false). We prefer compressing after-the-fact as it puts less stress on the server. Let us know if that works for you. We have a slightly different in-house version that we use which is much harder to maintain, hence we don't release it to the public as such so as to not cause issues that we are not comfortable with supporting. :-)

[rudolfmayer]

I am already using this option, but it only applies to vm exports in XVA format, not to vhd format disk exports.

The option compress=true documentation also says (https://github.com/NAUbackup/VmBackup/blob/master/VmBackup.py#L1159):

print '  [compress=True|False] - only for vm-export functions automatic compression (default: False)'

A usage for the XVA exports is e.g. here: https://github.com/NAUbackup/VmBackup/blob/master/VmBackup.py#L365

# vm-export vm-snapshot
        cmd = '%s/xe vm-export uuid=%s' % (xe_path, snap_vm_uuid)
        if compress:
            full_path_backup_file = os.path.join(full_backup_dir, vm_name + '.xva.gz')
            cmd = '%s filename="%s" compress=true' % (cmd, full_path_backup_file)

I am not sure whether there is support from the XenSever CLI to already compress a disk image; that would obviously preferred, as it reduces I/O. If that is not possible, I see that in https://github.com/NAUbackup/VmBackup/blob/master/VmBackup.py#L616 you have a post-processing/cleanup method, that does e.g. a gzip compression when files are >60 GB

def final_cleanup( tmp_full_path_backup_file, tmp_backup_file_size, tmp_full_backup_dir, tmp_vm_backup_dir, tmp_vm_max_backups):
    # mark this a successful backup, note: this will 'touch' a file named 'success'
    # if backup size is greater than 60G, then nfs server side compression occurs
    if tmp_backup_file_size > 60:
        log('*** LARGE FILE > 60G: %s : %sG' % (tmp_full_path_backup_file, tmp_backup_file_size))
        # forced compression via background gzip (requires nfs server side script)
        open('%s/success_compress' % tmp_full_backup_dir, 'w').close()
        log('*** success_compress: %s : %sG' % (tmp_full_path_backup_file, tmp_backup_file_size))

That could be a good place to do the gzip compression not only on large file size, but also parameter/argument controlled. I figure that it is performance-wise the same as if I manually do a gzip afterwards (which I currently do), but an integrated solution with the backup script would be still nice.

[NAUbackup]

Hi, Rudolf: As you know, vdi-export doesn't contain a native compress feature. You'd have to do something like: xe vdi-export format=vhd uuid=UUID-of-VDI filename= | gzip -c > $EXPORT_DIR/name.vhd.gz Again, the overhead on the XenServer to do on-the-fly compression is large, so compressing it after using the CPU of the machine hosting the NFS share is more efficient. The compression option add quite a bit of extra time to the export process. We don't really want to create a background process, either, as this would interfere with other VMs getting exported and slow things down even more.

It's something we can certainly consider; it's also something that could be requested of the XenServer team to incorporate as a feature request: http://bugs.xenserver.org/ is where that would need to be submitted.

[NAUbackup]

Am working with a Citrite looking into possible ways to improve on-the-fly compression. Will report back if something comes of this.