Ubuntu 19.04 - libvirt unable to use GVT-G device despite QEMU working sucessfully

[ThePraeceps]

Hey all,

I've managed to get GVT-g working sucessfully using the following script:

export QEMU_AUDIO_DRV=pa
export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_TIMER_PERIOD=99
export QEMU_PA_SERVER=/run/user/1000/pulse/native
# Start QEMU
qemu-system-x86_64 \
    -enable-kvm \
    -m 8G \
    -smp cores=2,threads=2,sockets=1,maxcpus=4 \
    -cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
    -machine type=pc,accel=kvm,kernel_irqchip=on \
    -global PIIX4_PM.disable_s3=1 \
    -global PIIX4_PM.disable_s4=1 \
    -name windows-gvt-g-guest \
    -soundhw hda \
    -display gtk,gl=on,grab-on-hover=on -full-screen \
    -vga none \
    -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:00:02.0/7692b8c6-80ca-11e9-94d2-a3b643d1d384,display=on,x-igd-opregion=on \
    -usb \
    -device usb-tablet \
    -device virtio-serial-pci \
    -device virtserialport,chardev=spicechannel0,name=com.redhat.spice.0 \
    -chardev spicevmc,id=spicechannel0,name=vdagent \
    -drive file=/var/lib/libvirt/images/win-gvtg.qcow2,format=qcow2,if=virtio,media=disk,l2-cache-size=8M \
    -drive file=/root/Images/Win10_1903_V1_English_x64.iso,if=ide,media=cdrom \
    -drive file=/root/Images/virtio-win-0.1.141.iso,if=ide,media=cdrom\
    -netdev tap,ifname=vm1,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,id=net0 \
    -device virtio-net-pci,netdev=net0 \

This works as intended and is working fully on my system.

However, if I create what I feel like is a equivelent libvirt XML.

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win-gvtg</name>
  <uuid>bb6d193f-41bd-47fe-b62e-a02855113e3b</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static' current='1'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <topology sockets='1' cores='3' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win-gvtg.qcow2'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/root/Images/virtio-win-0.1.141.iso'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:59:60:85'/>
      <source bridge='hosting'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='4cf2a3a8-d3be-435a-a5a5-71c26edc606b'/>
      </virtualport>
      <target dev='win-persistent'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' port='5905' autoport='no'>
      <listen type='address'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:00:02.0/7692b8c6-80ca-11e9-94d2-a3b643d1d384,display=on,x-igd-opregion=on'/>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
    <qemu:env name='QEMU_AUDIO_DAC_FIXED_FREQ' value='48000'/>
    <qemu:env name='QEMU_AUDIO_DAC_TRY_POLL' value='0'/>
    <qemu:env name='QEMU_AUDIO_ADC_FIXED_FREQ' value='48000'/>
    <qemu:env name='QEMU_AUDIO_ADC_TRY_POLL' value='0'/>
    <qemu:env name='QEMU_AUDIO_TIMER_PERIOD' value='1000'/>
    <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/>
  </qemu:commandline>
</domain>

I get the following error:

2019-06-08T01:19:38.580966Z qemu-system-x86_64: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:00:02.0/7692b8c6-80ca-11e9-94d2-a3b643d1d384,display=on,x-igd-opregion=on: vfio 7692b8c6-80ca-11e9-94d2-a3b643d1d384: failed to open /dev/vfio/14: No such file or directory

Despite the file definitely existing and I can boot successfully straight after with QEMU. I'm passing QEMU arguments rather than defining it in XML as the QEMU version I'm running doesn't support the arguments and compiling it from source has caused me issues in the past.

root@Apollo:/# ls -la /dev/vfio/ total 0 drwxr-xr-x 2 root root 80 Jun 8 02:19 . drwxr-xr-x 21 root root 4640 Jun 8 02:17 .. crw-rw---- 1 root kvm 238, 0 Jun 8 02:19 14 crw-rw-rw- 1 root root 10, 196 Jun 8 02:17 vfio

There are not any AppArmor denied logs in /var/log/syslog and even if I disable AppArmor for libvirt it still does not work.

QEMU is configured to use the kvm group and even if I run it under root it still doesn't work.

Does anyone have any ideas?

Version info:

virsh # version
Compiled against library: libvirt 5.0.0
Using library: libvirt 5.0.0
Using API: QEMU 5.0.0
Running hypervisor: QEMU 3.1.0

[adevur]

When I use GVT-g with libvirt, I add this to the XML configuration of the virtual machine, in order to add the virtual graphics device to the VM:

    <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
      <source>
        <address uuid='5bfcd486-c5eb-43e4-a5ee-97366690106f'/>
      </source>
      <rom bar='off'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>

Replace 5bfcd486-c5eb-43e4-a5ee-97366690106f with the actual UUID of your virtual graphics device.

P.S.: You can also (optionally) change <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> if you want your VM to see the vGPU at a particular PCIe slot. You shouldn't have problems if you leave this slot as is in the code above.

[ThePraeceps]

I attempted to do that as described in the dma-buf guide, but the latest version of either libvirt or qemu (Documentation says "Since 3.2" so I'm assuming qemu as libvirt is 5.0.0) doesn't support mdev host dev devices so I am unable to do that.

[adevur]

Mmh, I don't quite understand actually. First of all, I forgot to mention that I'm using GVT-g with libvirt 4.5.0 and QEMU 2.12.0 on a CentOS 7.6 computer.

Anyway, I don't see any reasons why your libvirt version shouldn't support this type of device: libvirt documentation says that libvirt supports hostdev devices of type mdev and model vfio-pci since version 4.4.0. And libvirt changelog doesn't mention changes to that in its 5.x.x releases.

P.S.: my VM uses Q35 chipset and UEFI bios (OVMF), if it can help.

My only suggestion for now is to try with a live ISO of CentOS 7.6, in order to see if it's a software-related issue only.

Your issue could also somehow depend on the way you created the vGPU or on the way you set up your environment (GVT-g drivers, parameters to pass to Linux during boot, ...).

[ThePraeceps]

Interesting, I thought dma-buf doesn't support UEFI. I'm pretty certain that the environment is correct because like I said I can run the QEMU command line version just fine.

I think I'm using I440FX so I'll give q35 a try too.

[ChristophSchmidpeter]

As you get it working with qemu, but not with libvirt, it might be related to #81.