Support specifying a ROM file for PCI devices to support passthrough of laptop nVidia GPUs

[chriswoope]

At least some nVidia GPUs in Optimus laptops work with GPU passthrough, but only if the video card ROM is passed manually, since getting it requires making an ACPI call on the host.

Unfortunately, while qemu in the stubdomain supports passing romfile= to device_add via QMP, libxl seems to have no support for passing that option; furthermore, when using romfile= the file needs to be in the stubdomain file system.

Hacking vchan-socket-proxy to add romfile= to the device_add call and patching the stubdomain to add the ROM makes nouveau in the VM properly recognize at least my card when passed through, but obviously it's not a very clean solution.

So, my suggestion would be to:

1. Patch qemu to add a new "rombase64" option that passes a Base64-encoded ROM via QMP
2. Patch libxl to recognize "romfile" for domains with a stubdomain, read the file, base64 encode it and pass via QMP
3. If needed (probably not?), patch libvirt to pass romfile to libxl when the libvirt xml contains
4. Patch qubes pci.xml to specify when a parameter is passed
5. If needed (probably not?), patch qvm-pci to support the parameter for the ROM file
6. Ideally patch qvm-pci and/or qubes-qube-manager to have Qubes automatically pick up the ROM file from a standard place (e.g. /var/lib/qubes/pci-roms/VENDOR_DEVICE_PCIID.rom)

Then it would be nice to provide an automatic setup tool for GPU passthrough, which in this case needs to:

1. If the GPU was passed through, require a reboot
2. Attach the GPU to dom0 if needed (unbind from pciback, rebind to the GPU driver such as nouveau)
3. Load the video driver in dom0, dump the BIOS by reading /sys/kernel/debug/dri/\<id>/vbios.rom and put it in the proper place
4. Add the rd.qubes.hide_pci options to hide the GPU in dom0

As further work, it should be possible to have custom code that would extract the BIOS automatically on the fly without having to load a DRI driver, but that might not be worth the effort.

[chriswoope]

If anyone is trying to get GPU passthrough working right now on an Optimus laptop, this is the current hack that I'm using:

#!/bin/bash
mv /usr/bin/vchan-socket-proxy /usr/bin/vchan-socket-proxy.orig
cat -<<EOF > /usr/bin/vchan-socket-proxy
#!/bin/bash
vchan-socket-proxy.orig "$1" "$2" "$3" "$4.internal" &
nc -k --listen --unixsock "$4" --sh-exec "sed -u -e 's|\"hostaddr\":\"0000:02:00.0\"|\"hostaddr\":\"0000:02:00.0\",\"romfile\":\"/share/gpubios\"|'|nc --unixsock $4.internal"
EOF

for stubdom in /usr/libexec/xen/boot/qemu-stubdom-linux-rootfs /usr/libexec/xen/boot/qemu-stubdom-linux-full-rootfs; do
    dir="$(mktemp -d)"
    cd "$dir"
    if ! test -e "$stubdom.orig"; then
        mv "$stubdom" "$stubdom.orig"
    fi
    zcat "$stubdom.orig"|cpio -idm
    cp -a /usr/libexec/xen/boot/gpubios ./share/gpubios
    find .|cpio -o -c|gzip -9 > "$stubdom"
done

You need to correct the PCI ID, dump the BIOS to /usr/libexec/xen/boot/gpubios as described above, and then in the VM you need to disable default kernel opts and remove nomodeset from the kernelopts, and compile qubes-gui-agent with Glamor support from https://github.com/QubesOS/qubes-gui-agent-linux/pull/150, enable Render in the /etc/xorg/xorg-qubes.conf.template, set LIBVA_DRIVER_NAME=nouveau for VA-API and it should "just work".

You can also use bumblebee with optirun instead of glamor (which can let you hotplug the GPU in theory, but I couldn't get mine to hot attach), but in that case you need to compile it from https://github.com/Bumblebee-Project/Bumblebee/pull/1088 and also compile a patched xorg where in xf86RandR12.c you need to replace if (!pScreen->isGPU) { with if (!pScreen->isGPU && config->output && config->output[0]) { since it segfaults otherwise.

It works fine with Linux nouveau, haven't tested the nVidia binary driver on Linux or Windows yet.