Live Daily Backup Of VM's

[jtsoftex]

I guess we someone else raised this issue here and I saw there are few lines here on it but I can't see a proper solution.

Can we add in UI something like take a daily backup of VM's in backend and let them there for say 7 days and then do like rollover.

Also for now is there any command/script for it so that we can take live backup for full VM without stopping it..

Apperaciate your early response on it.

Thanks

[retspen]

Live backup have some problem need suspend VM and then copy your hdd image.

[AlexH-HankIT]

What about LVM Snapshots for Live Backup?

[retspen]

Libvirt doesn't support LVM Snapshots

[AlexH-HankIT]

Libvirt doesn't have to support it. You can always create an LVM snapshot, if you installed lvm correctly. Can webvirtmgr execute a bash command?

[retspen]

WebVirtMgr can manage VM's trough libvirt API and can't execute a bash command.

[retspen]

#!/bin/bash

DAY=$(date +"%w")
# Get a list of the virtual machines that are running on the server.
vmlist=`/usr/bin/virsh list --all | grep running | awk {'print $2'}`

# For each virtual machine on the virtual host server, get the list of disk images that are in use.
for vm in $vmlist
        do
        # Suspend the virtual server
        /usr/bin/virsh suspend $vm
        #echo "$vm suspended at" `date`

        # For each disk, copy it to the temporary rsnapshot directory
        #echo "Backing up $vm.img from $vm"
        hda=`virsh domblklist $vm | grep vda | awk {' print $2'}`
        /usr/bin/rsync --inplace $hda /mnt/backup/images/$DAY-$vm.img

        # Resume the virtual server
        /usr/bin/virsh resume $vm
        #echo "$vm resumed at" `date`
done

[MACscr]

Why do guests have to be suspended in order to be backed up? I mean, daily backups of kvm instances is a pretty common feature for vm panels. Servers typically run 24/7 and having to suspend or shutdown to backup doesnt sound to great to me.

[retspen]

No problem you can copy a image without suspend and shutdown but when you coping the image when I write data to your database maybe something can be lost.

[jtsoftex]

restpen, thanks for sharing script!!

Also, if we want to take like 3 days rollover backup then we can add this line the end, may be it will help some.

"find /path/to/backup/*.img -mtime +3 -exec rm {} \;" where +3 means it will delete files those are 3 days old. You can change that value according to you.

Also, restpen, can we add this in UI with variables ?

[retspen]

No, we can't. Libvirt can't have access to console.

[lub]

Maybe there is a solution with external Snapshots (at least for qcow2)?

Create an external Snapshot -> copy the original image -> delete the Snapshot

[AlexH-HankIT]

I tried that a while ago. But it seems like libvirt cant remove an external snapshot while the vm is running. http://wiki.libvirt.org/page/I_created_an_external_snapshot,_but_libvirt_won%27t_let_me_delete_or_revert_to_it

LVM Snapshots are perfect for backup, but since libvirt doesnt support them you have to do it without gui. I wrote a script to backup vms on logical volumes with filesystems (ext4 for example). It is executed via cronjob every friday. I can provide the script if anybody is interested.

[brylie]

Thanks @retspen :-) How can this script be made to write to a log file? I.e. it will be running as a cron job, and it would be nice to see a log of success/failure messages.

[brylie]

I am having difficulty getting the script to work. It stops at "Backing up ..." and never displays the "... resumed at" statement.

Here is my bash file:

#!/bin/bash

backup_directory='/var/backups/images/'

DAY=$(date +"%Y-%m-%d")
# Get a list of the virtual machines that are running on the server.
vmlist=`/usr/bin/virsh list --all | grep running | awk {'print $2'}`

# For each virtual machine on the virtual host server, get the list of disk images that are in use.
for vm in $vmlist
        do
        # Suspend the virtual server
        /usr/bin/virsh suspend $vm
        echo "$vm suspended at" `date`

        # For each disk, copy it to the temporary rsnapshot directory
        echo "Backing up $vm.img from $vm"
        hda=`virsh domblklist $vm | grep vda | awk {' print $2'}`
        /usr/bin/rsync --inplace $hda $backup_directory$DAY-$vm.img

        # Resume the virtual server
        /usr/bin/virsh resume $vm
        echo "$vm resumed at" `date`
done

[brylie]

I added the --progress flag to rsync, and see that the backup just takes a while. Some of our images are >100Gb.

Is there a way to do this with less downtime?

[AlexH-HankIT]

I wouldn't suspend the vm at all. You should always use LVM for your virtual machines. It's snapshot capabilities also allows a live backup of your vms.

If you run the script via cron the output will automatically send via mail to root@domain. You can also redirect it into a log file. Just put "> /var/log/logfile" behind the line in your /etc/crontab.

[brylie]

I just want to make sure that the image is not corrupted. Would the following script be advisable @MrCrankHank?

#!/bin/bash

backup_directory='/var/backups/images/'

DAY=$(date +"%Y-%m-%d")

# Get a list of the virtual machines that are running on the server.
vmlist=`/usr/bin/virsh list --all | grep running | awk {'print $2'}`

# For each virtual machine on the virtual host server, get the list of disk images that are in use.
for vm in $vmlist
        do
        # For each disk, copy it to the temporary rsnapshot directory
        hda=`virsh domblklist $vm | grep vda | awk {' print $2'}`
        /usr/bin/rsync --inplace $hda $backup_directory$DAY-$vm.img

        # Resume the virtual server
        /usr/bin/virsh resume $vm
done

[brylie]

You should always use LVM for your virtual machines. It's snapshot capabilities also allows a live backup of your vms.

@MrCrankHank does this mean that we need to select LVM when installing the guest operating system? Does this also mean that the host OS should have LVM enabled?

[AlexH-HankIT]

The script should work. but i prefer a live backup of my vms, because some vms (Windows) have problems with suspending. If your vms are fine with suspending and you don't care about the downtime, then go ahead and use the script. But if there problems i strongly recommend to configure lvm on the storage where your vms are. What storage do you use currently?

[brylie]

I generally choose the LVM option when installing guest OSs, but cannot be certain of VMs installed by other admins.

Can LVM be enabled after the initial installation? How are you performing the live backups?

[AlexH-HankIT]

You don't need lvm inside the guest. It's important that the guests filesystem lives on a lvm volume. On my host i have a Volume Group called VG_data01. In this volume group i have a logical volume for every guest harddrive. If i want to create a backup of one vm, i snapshot the logical volume of that guest an copy the file system with dd (then gzip it of course...). Of course this can be easily scripted.

As far as i know you can add a Volume Group under "Storages" in Webvirtmgr. If you do so, libvirt should take care of creating the logical volumes for your vms (don't know for sure so, i'm using another product).

One downside of this setup is, that you can't use thin provisioning.