Error consolidating block device: block copy still active

[MaxiReglisse]

Consolidation fails with a Windows 2016 Server, i do not understand why... Backups are performed every hour, and consolidation once a day.

Below the logs:

[INF] libVirt version '1.3.1' support is experimental
[DEB] /usr/bin/qemu-img version '2.5.0' is supported
[DEB] QEMU/KVM version '2.5.0' is supported
[DEB] Domains NOTRUNNING to backup: 
[DEB] Domains RUNNING to backup:  Magnus
[DEB] Snapshot for domain 'Magnus' requested
[DEB] Using timestamp '20180720-131701'
[DEB] Snapshotting block devices for 'Magnus' using suffix 'bimg-20180720-131701'
Formatting '/var/lib/libvirt/images/Magnus.bimg-20180720-131701', fmt=qcow2 size=42949672960 backing_file=/var/lib/libvirt/images/Magnus.bimg-20180720-123807 backing_fmt=qcow2 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
[VER] Snapshot for block devices of 'Magnus' successful
[DEB] Getting all backing files for: '/var/lib/libvirt/images/Magnus.bimg-20180720-131701'
[VER] Copy backing file '/var/lib/libvirt/images/Magnus.bimg-20180720-123807' to '/var/lib/libvirt/backups/Magnus/Magnus.bimg-20180720-123807'
[VER] Copy backing file '/var/lib/libvirt/images/Magnus.bimg-20180720-121701' to '/var/lib/libvirt/backups/Magnus/Magnus.bimg-20180720-121701'
[VER] Copy backing file '/var/lib/libvirt/images/Magnus.bimg-20180720-111701' to '/var/lib/libvirt/backups/Magnus/Magnus.bimg-20180720-111701'
[VER] Copy backing file '/var/lib/libvirt/images/Magnus.bimg-20180720-101701' to '/var/lib/libvirt/backups/Magnus/Magnus.bimg-20180720-101701'
[VER] Copy backing file '/var/lib/libvirt/images/Magnus.qcow2' to '/var/lib/libvirt/backups/Magnus/Magnus.qcow2'
[INF] libVirt version '1.3.1' support is experimental
[DEB] /usr/bin/qemu-img version '2.5.0' is supported
[DEB] QEMU/KVM version '2.5.0' is supported
[DEB] Domains NOTRUNNING to backup: 
[DEB] Domains RUNNING to backup:  Magnus
[DEB] Consolidation of block devices for 'Magnus' requested
[DEB] Block devices to be consolidated:
    /var/lib/libvirt/images/Magnus.bimg-20180720-131701
[DEB] Consolidation method: blockcommit
[DEB] Consolidation of '0' block device: '/var/lib/libvirt/images/Magnus.bimg-20180720-131701' for 'Magnus'
[DEB] Parent block device: '/var/lib/libvirt/images/Magnus.bimg-20180720-123807'
[ERR] Error consolidating block device '/var/lib/libvirt/images/Magnus.bimg-20180720-131701' for 'Magnus':
 error: block copy still active: disk 'sda' already in active block job

What's the matter ?

I also use fi-backup with other Ubuntu VM, and it works very well. Would the problem be specific to Windows VM?

Can you help me, please ?

[AJRepo]

The problem would not be specific to Windows VM vs Ubuntu VM.

Virsh is saying that there's an unfinished blockcopy operation.

First I'd look to see if this is a continuing issue.

Try running the following command to get the list of blocks (might need to run all commands below with sudo)

virsh domblklist Magnus

Then look for active block operations for that file:

e.g.

virsh blockjob Magnus /var/lib/libvirt/images/Magnus.bimg-20180720-131701 --info

If nothing then next look at the backing files.

qemu-img info XXXXX

where XXXXX are the backing files (e.g. /var/lib/libvirt/images/Magnus.bimg-20180720-131701, /var/lib/libvirt/images/Magnus.bimg-20180720-123807, ...)

Should say "corrupt: false" Should say "backing file: for all except the last child in the backing chain"

[MaxiReglisse]

everything seems normal to me...

# virsh domblklist Magnus
Target     Source
------------------------------------------------
hda        /var/lib/libvirt/images/Magnus.bimg-20180720-163324
hdb        -

and

# virsh blockjob Magnus /var/lib/libvirt/images/Magnus.bimg-20180720-163324 --info
Active Block Commit: [100 %]

i followed the chain as you can see on pastebin.com but, idem, everything seems normal to me...

could the following order help us?

find /var/lib/libvirt/images/ -name Magnus.* | xargs -i -t qemu-img info {}

thanks for your help.

[AJRepo]

Wow. That is a very long chain. I noticed that two in the chain are owned by root.

-rw------- 1 root root 197632 Jul 19 16:17 /var/lib/libvirt/images/Magnus.bimg-20180719-161701 -rw------- 1 root root 6143541248 Jul 20 09:54 /var/lib/libvirt/images/Magnus.bimg-20180714-061701 So you might have a permissions issue and can try chowning them to have the same ownership as the others.

But as I look at your paste I see you might have two chains pointing to the same backing file.

[root@kvm815 ~]# qemu-img info /var/lib/libvirt/images/Magnus.bimg-20180720-101701 image: /var/lib/libvirt/images/Magnus.bimg-20180720-101701 file format: qcow2 virtual size: 40G (42949672960 bytes) disk size: 744M cluster_size: 65536 backing file: /var/lib/libvirt/images/Magnus.qcow2

and

[root@kvm815 ~]# qemu-img info /var/lib/libvirt/images/Magnus.bimg-20180711-174111 image: /var/lib/libvirt/images/Magnus.bimg-20180711-174111 file format: qcow2 virtual size: 40G (42949672960 bytes) disk size: 1.2G cluster_size: 65536 backing file: /var/lib/libvirt/images/Magnus.qcow2

Would need to look for possibly an old consolidation still running and/or follow the backing chain from the orig point.

Would probably recommend (only after making a good backup) either

A) using virsh commands to consolidate stepwise over the backing chain.

or

B) creating a new virtual qcow2 disk for your Windows VM to use, using the windows backup/restore to backup the old system and then restore to the new one you just created. Then move all those old backing files to archive elsewhere.

[MaxiReglisse]

Thanks Davide ! i chose the solution B : backup and restoration with the Windows tools, a great experience for me ! ;) the new VM is called magnus (with a tiny m). I successfully attempted this morning a consolidation operation. just before, the images directory contained the following files:

[root@kvm815 ~]# ls -ltr /var/lib/libvirt/images/magnus.*
-rw------- 1 libvirt-qemu kvm 42956488704 juil. 25 07:16 /var/lib/libvirt/images/magnus.qcow2
-rw------- 1 libvirt-qemu kvm    35061760 juil. 25 08:16 /var/lib/libvirt/images/magnus.bimg-20180725-071702
-rw------- 1 libvirt-qemu kvm    34013184 juil. 25 09:16 /var/lib/libvirt/images/magnus.bimg-20180725-081702
-rw------- 1 libvirt-qemu kvm    32899072 juil. 25 09:48 /var/lib/libvirt/images/magnus.bimg-20180725-091702

I suppose that the image of 7:16 was made by consolidation, while those of 8h16 and 9h16 were done by normal backups. when at 9:48, it corresponds to the time of my connection to the server. I used the following command for consolidation, which sends the consolidation traces to the log libvirt of the VM:

 [root@kvm815 ~]# LANG=en_US.UTF-8 && /usr/local/bin/fi-backup.sh -c -b /var/lib/libvirt/backups/magnus -d magnus >> /var/log/libvirt/qemu/magnus.log 2>&1

which resulted in the following result:

[root@kvm815 ~]# ls -ltr /var/lib/libvirt/images/magnus.*
-rw------- 1 libvirt-qemu kvm 42956488704 Jul 25 09:50 /var/lib/libvirt/images/magnus.qcow2

what is the expected result! Thanks again.

Ernest.

[AJRepo]

You are welcome, but I am not Davide. This is me: https://github.com/AJRepo

Yes with consolidation you should get back to just one file. Those other files (e.g. 35061760 juil. 25 08:16 /var/lib/libvirt/images/magnus.bimg-20180725-071702 ) aren't self-contained backups per se, they are the record of stuff done between snapshot times, so that you can roll back your system to that state at that time (e.g. to the snapshot time). In order to restore to that state, you need ALL of the files from the origin backing file to the last qcow2 file in the chain.

The good thing about lots of snapshots is that you can "roll back" to any of the states. The bad thing about lots of snapshots is that consolidation gets slower and slower because in order to consolidate it requires a running through ALL of the chain. This isn't unique to KVM - its also true of vmware, etc.

The good news is that large backing chains don't slow down startup of KVM/qemu VMs like some other VM products which shall remain nameless.

So unless you need all those snapshots, I would consolidate more frequently. You can still backup the full file. Also note the default consolidation method is blockpull which can create a smaller file as it compresses unused disk space in the final file but is much slower than blockcommit. If you run blockpull once it will create the smaller file and then you can switch to blockcommit for future consolidation for faster results and a file that's not much larger than the first one created with the first blockpull consolidation.