unable to execute QEMU command 'transaction'

[MaxiReglisse]

when the VM has multiple disks, fi-backup fails with a "permission denied" error:

[DEB] Snapshotting block devices for 'ubuntuserver_clone' using suffix 'bimg-20190803-111757'
Formatting '/var/lib/libvirt/images/ubuntuserver_clone.bimg-20190803-111757', fmt=qcow2 size=17179869184 backing_file=/var/lib/libvirt/images/ubuntuserver_clone.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
[ERR] Snapshot for 'ubuntuserver_clone' failed! Exit code: 0
'error: internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied'

the second disk was added with virsh command:

virsh attach-disk ubuntuserver_clone /var/lib/libvirt/images/ubuntuserverclone-disk1-5G.qcow2 vdb --cache none

this happens regardless of the version of qemu, 2.5 or 2.11.

Thanks in advance for you help.

[AJRepo]

Is this on a machine running apparmor?

[MaxiReglisse]

yes it is... here is the apparmor status:

root@kvm540:~/virt-sysprep # apparmor_status
apparmor module is loaded.
28 profiles are loaded.
28 profiles are in enforce mode.
   /sbin/dhclient
   /snap/core/7270/usr/lib/snapd/snap-confine
   /snap/core/7270/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/lxc-start
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/libvirtd
   /usr/sbin/libvirtd//qemu_bridge_helper
   /usr/sbin/ntpd
   /usr/sbin/tcpdump
   libvirt-17e19e53-4859-4640-800f-50c671330edd
   libvirt-254b5857-e6fe-4b03-8cbc-925037a07499
   libvirt-38ca8f6f-4eaa-4aa9-9e6c-05c9b9634e9a
   libvirt-4ff8af4f-c4ab-42f4-8adf-ecc79cf48235
   libvirt-f16dd208-195d-4b89-8f35-cb9f51a45d4d
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
   man_filter
   man_groff
   snap-update-ns.core
   snap.core.hook.configure
   virt-aa-helper
0 profiles are in complain mode.
6 processes have profiles defined.
6 processes are in enforce mode.
   /usr/sbin/libvirtd (3085) 
   libvirt-17e19e53-4859-4640-800f-50c671330edd (4925) 
   libvirt-254b5857-e6fe-4b03-8cbc-925037a07499 (5072) 
   libvirt-38ca8f6f-4eaa-4aa9-9e6c-05c9b9634e9a (5363) 
   libvirt-4ff8af4f-c4ab-42f4-8adf-ecc79cf48235 (5601) 
   libvirt-f16dd208-195d-4b89-8f35-cb9f51a45d4d (5219) 
0 processes are in complain mode. 
0 processes are unconfined but have a profile defined.

[MaxiReglisse]

as suggested by Programster, we can find the concerned libvirt ID in the most recent syslogs:

Aug  3 16:29:33 kvm540 kernel: [94840.327053] audit: type=1400 audit(1564842573.153:1156): apparmor="DENIED" operation="open" profile="libvirt-17e19e53-4859-4640-800f-50c671330edd" name="/var/lib/libvirt/images/themawebprod.bimg-20190803-162932" pid=4925 comm="qemu-system-x86" requested_mask="wrc" denied_mask="wrc" fsuid=64055 ouid=64055

then he proposed a workaround:

sudo aa-complain /etc/apparmor.d/libvirt/libvirt-17e19e53-4859-4640-800f-50c671330edd

and it works, the themawebprod VM was successfully saved!

[DEB] Snapshotting block devices for 'themawebprod' using suffix 'bimg-20190803-164943'
Formatting '/var/lib/libvirt/images/themawebprod.bimg-20190803-164943', fmt=qcow2 size=17179869184 backing_file=/var/lib/libvirt/images/themawebprod.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
Formatting '/var/lib/libvirt/images/themawebprod-1.bimg-20190803-164943', fmt=qcow2 size=161061273600 backing_file=/var/lib/libvirt/images/themawebprod-1.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
[VER] Snapshot for block devices of 'themawebprod' successful

but no one answered him when he asked the question whether there was a more elegant solution ...

[MaxiReglisse]

if I understand correctly, it would be enough to execute the command aa-complain for each VM ID:

root@kvm540:~ # grep uuid /etc/libvirt/qemu/*.xml
/etc/libvirt/qemu/catalyse-clone.xml:  <uuid>1146b82c-6639-42cc-9e3d-7df4d336b008</uuid>
/etc/libvirt/qemu/catalyse.xml:  <uuid>5815309f-3a0c-4855-9efa-884fcba24b2f</uuid>
/etc/libvirt/qemu/owncloud.xml:  <uuid>f16dd208-195d-4b89-8f35-cb9f51a45d4d</uuid> 
/etc/libvirt/qemu/salinois.xml:  <uuid>254b5857-e6fe-4b03-8cbc-925037a07499</uuid>
/etc/libvirt/qemu/themawebprod.xml:  <uuid>17e19e53-4859-4640-800f-50c671330edd</uuid>
/etc/libvirt/qemu/themaweb.xml:  <uuid>38ca8f6f-4eaa-4aa9-9e6c-05c9b9634e9a</uuid>
/etc/libvirt/qemu/ubuntukvm.xml:  <uuid>eb944d9a-f923-42d5-889c-effb797025d0</uuid>
/etc/libvirt/qemu/ubuntuserver3.xml:  <uuid>ac7f96b4-6247-4905-8230-a4205eb6d157</uuid>
/etc/libvirt/qemu/ubuntuserver_clone.xml:  <uuid>4ff8af4f-c4ab-42f4-8adf-ecc79cf48235</uuid>

is there another solution?

[MaxiReglisse]

virsh domuuid convert a domain name to domain UUID. consequently just use the following command:

aa-complain /etc/apparmor.d/libvirt/libvirt-`virsh domuuid <domain>`

foreach domain name.

[cboltz]

For the records - aa-complain puts the profile into complain/"learning" mode, which means to a) allow everything and b) to log what is not allowed in the profile.

Therefore aa-complain is a workaround, but not a real solution.

AFAIK the libvirt AppArmor profiles are autogenerated by libvirt (no idea where/when exactly), so the real solution would be to generate profiles that allow everything that is needed.

[MaxiReglisse]

the problem with aa-complain is that when restarting the server, the VMs become in "enforce" mode, instead of "complain".

consequently, after many attempts to configure apparmor, the simplest solution at the moment is to disable it: i set security_driver = "none" in /etc/libvirt/qemu.conf, instead of security_driver = "apparmor", and the VM backups are done correctly.

an issue is open in the gitlab apparmor project