Libvirt VMs don't run with KVM

[rohityadavcloud]

I'm trying to see if my M1 Mac Mini with Ubuntu Asahi (Ubuntu 22.04 server) can run VMs with KVM. Defining the domain [1] with kvm domain type is throwing error:

error: Failed to define domain from vm.xml
error: unsupported configuration: Emulator '/usr/bin/qemu-system-aarch64' does not support virt type 'kvm'

This however, works, if I change the domain type to 'qemu' and change the CPU mode to a host model such as cortex-a57 etc. but the VM is slow as qemu emulates the CPU and KVM can't be used.

I upgraded the linux kernel from 5.19 to 6.3 (ubuntu-asahi pkg) but still the same issue. I repeated the same on a Raspberry Pi 4 (4GB model) arm64 host with Ubuntu 22.04 but Linux kernel 5.15.0-1033-raspi and this is working as expected (I tried both terminal and creating/checking with cockpit-machines). Any hints, pointers, tips cc @tobhe ? Thanks.

Additional detail about the system:

root@macmini:~# uname -a
Linux macmini 6.3.0-13-asahi #1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 14 19:48:48 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
root@macmini:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:    22.04
Codename:   jammy
root@macmini:~# virsh version
Compiled against library: libvirt 8.0.0
Using library: libvirt 8.0.0
Using API: QEMU 8.0.0
Running hypervisor: QEMU 6.2.0

root@macmini:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@macmini:~# lsmod | grep kvm
root@macmini:~# lscpu -e
CPU NODE SOCKET CORE L1d:L1i:L2 ONLINE    MAXMHZ   MINMHZ MHZ
  0    0      0    0 0:0:0         yes 2064.0000 600.0000   -
  1    0      0    1 1:1:0         yes 2064.0000 600.0000   -
  2    0      0    2 2:2:0         yes 2064.0000 600.0000   -
  3    0      0    3 3:3:0         yes 2064.0000 600.0000   -
  4    0      0    0 4:4:1         yes 3204.0000 600.0000   -
  5    0      0    1 5:5:1         yes 3204.0000 600.0000   -
  6    0      0    2 6:6:1         yes 3204.0000 600.0000   -
  7    0      0    3 7:7:1         yes 3204.0000 600.0000   -

[1] Domain vm.xml example:

<domain type='kvm'>
<name>VM1</name>
<uuid>238fd6e4-9296-4d37-a3c1-9e84ed2abfdb</uuid>
<description>Debian GNU/Linux 5.0 (64-bit)</description>
<cpu mode='host-passthrough'></cpu><sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>238fd6e4-9296-4d37-a3c1-9e84ed2abfdb</entry>
</system>
</sysinfo>
<os>
<type  arch='aarch64' machine='virt'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
<boot dev='cdrom'/>
<boot dev='hd'/>
</os>
<on_reboot>restart</on_reboot>
<memory>524288</memory>
<currentMemory>524288</currentMemory>
<devices>
<memballoon model='virtio'>
<stats period='0'/>
</memballoon>
</devices>
<vcpu current="1">1</vcpu>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<cputune>
<shares>500</shares>
</cputune>
<clock offset='utc'>
<timer name='kvmclock' >
</timer>
</clock>
<devices>
<emulator></emulator>
<controller type='usb' index='0' model='qemu-xhci'>
<alias name='usb'/><address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</controller>
<watchdog model='i6300esb' action='none'/>
<console type='pty'>
<target port='0'/>
</console>
<disk  device='disk' type='file'>
<driver name='qemu' type='qcow2' cache='none' />
<source file='/mnt/c08cc71e-bd16-3ee4-afc9-6e18093b17f0/14920d53-65fa-4d58-85ac-6e337b78f567'/>
<target dev='vda' bus='virtio'/>
<serial>14920d5365fa4d5885ac</serial>
</disk>
<disk  device='cdrom' type='file'>
<driver name='qemu' type='raw' />
<source file=''/>
<target dev='hdc' bus='scsi'/>
</disk>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes' listen='192.168.1.30' passwd='LEDO2xrg'/>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/s-39-VM.org.qemu.guest_agent.0'/>
<address type='virtio-serial'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='keyboard' bus='usb'/>
<input type='mouse' bus='usb'/>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:c9:c6'/>
<model type='virtio'/>
<rom bar='off' file=''/><link state='up'/>
</interface>
<interface type='bridge'>
<source bridge='cloudbr0'/>
<mac address='1e:00:b8:00:00:03'/>
<model type='virtio'/>
<rom bar='off' file=''/><link state='up'/>
</interface>
<interface type='bridge'>
<source bridge='cloudbr0'/>
<mac address='1e:00:3d:00:00:0b'/>
<model type='virtio'/>
<rom bar='off' file=''/><link state='up'/>
</interface>
</devices>
</domain>

[rohityadavcloud]

On further debugging, I found it wasn't able to start and use KVM:

From libvirtd logs:

2023-07-24 08:43:41.734+0000: 7121: debug : virCommandRunAsync:2607 : About to run LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /usr/bin/qemu-system-aarch64 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/qmp-HIZ771/qmp.monitor,server=on,wait=off -pidfile /var/lib/libvirt/qemu/qmp-HIZ771/qmp.pid -daemonize
2023-07-24 08:43:41.735+0000: 7121: debug : virCommandRunAsync:2609 : Command result 0, with PID 16324
2023-07-24 08:43:41.753+0000: 7121: debug : virCommandRun:2453 : Result exit status 0, stdout: '' stderr: '2023-07-24 08:43:41.735+0000: 16324: debug : virExecCommon:462 : Setting child uid:gid to 64055:109 with caps 2
2023-07-24 08:43:41.735+0000: 16324: debug : virSetUIDGIDWithCaps:1122 : Added 'dac_override' to child capabilities' set
ioctl(KVM_CREATE_VM) failed: 22 Invalid argument
qemu-system-aarch64: failed to initialize kvm: Invalid argument
qemu-system-aarch64: falling back to tcg

I even upgraded to Ubuntu server 23.04 with qemu 8.x, still same issue. However, I found if I run qemu directly on shell it seems to be starting, with:

/usr/bin/qemu-system-aarch64 -accel kvm -M virt -cpu host -m 2G -smp 2 -bios /usr/share/AAVMF/AAVMF_CODE.fd -drive if=none,file=/export/primary/disk,id=hd0 -vnc :1 -monitor stdio

It appears much of this could be to do with libvirtd, and not necessarily the kernel.

[tobhe]

Thanks for the research! I have just read the backlog in #asahi and saw that you got some good support there. Fwiw I use lxd VMs on 23.04 on my m2 air and those work just fine, so a bug in libvirt doesn't seem unlikely.

[rohityadavcloud]

It might be a combination of some parameters or something libvirtd/qemu isn't able to check properly. Yes, I asked a few questions and the asahi community is really helpful (logs https://oftc.irclog.whitequark.org/asahi/2023-07-24). I was able to strace the libvirtd parameter, and it appears libvirtd on M1 mac mini fails to get KVM to use as accelerator; the same libvirtd behaves differently on a Raspberry Pi4. I scp'd the qemu-system-aarch64 binaries from RPi4 but still same issue. I tried the qemu args option in the domain xml, but libvirt seems to be ignoring those. I'll revisit this again in future. (would like to keep the ticket open for couple of days more if I'm near a solution I'll share and close it).

Using qemu (tcg) works, but KVM acceleration would be preferable.

[tobhe]

I did some digging on the upstream bug tracker and https://gitlab.com/libvirt/libvirt/-/issues/365 sounds like a similar issue.

[rohityadavcloud]

Ha I found the same just few mins ago, came here to close it. You're right @tobhe it's an upstream/libvirtd issue. Closing on the linked issue - https://gitlab.com/libvirt/libvirt/-/issues/365

[tobhe]

So it turns out this was actually a qemu bug caused by kvm using -M none. It was fixed in https://lists.nongnu.org/archive/html/qemu-devel/2023-07/msg05127.html. I will see if we can backport the fix into the Ubuntu package.

[rohityadavcloud]

Ah great- thanks @tobhe that would be awesome! It was Hector Martin who did further investigation and filed the issue https://gitlab.com/libvirt/libvirt/-/issues/365#note_1501797497 but glad to see it's been already fixed.

[rohityadavcloud]

Thought to update/share that CloudStack + libvirt + KVM is working fine with latest Ubuntu Asahi on Mac mini M2 pro:

https://twitter.com/RohitYadavCloud/status/1747993477147664410