QaidVoid / Complete-Single-GPU-Passthrough

Single GPU VFIO Passthrough Guide
779 stars 38 forks source link

Getting kicked out of KDE, but nothing starts #42

Closed mineLdiver closed 5 months ago

mineLdiver commented 7 months ago

Arch Linux, RTX 4060 (+ GTX 970, but it's isolated for a Windows 7 VM, which doesn't work either so whatever lol) VT-d enabled, IOMMU enabled (checked through dmesg).

I followed this guide to the end, got the VM installed, hooks written, and scripts added.

When I start the VM, I get kicked out of KDE to the Linux's boot screen (I suppose?) where the last line says Starting Terminate Plymouth Boot Screen, and then nothing happens and I have to force shutdown my system. It seems like the disk is busy for a few seconds after that, but then, it's just idling on the boot screen.

I tried with and without the KDE (Wayland) line in start.sh, no effect.

One thing though, I can't for the life of me remove Display Spice and Video. Video just doesn't have the remove button, and when I do remove it from XML, it comes back. Removing Display Spice however throws an error, and editing XML throws the same error, not allowing me to save the configuration:

Error Removing Device: unsupported configuration: chardev 'spicevmc' not supported without spice graphics

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/delete.py", line 338, in remove_devobj_internal
    vm.remove_device(devobj)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 641, in remove_device
    self._redefine_xmlobj(xmlobj)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 391, in _redefine_xmlobj
    self._redefine_xml_internal(origxml, newxml)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 374, in _redefine_xml_internal
    self._define(newxml)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1137, in _define
    self.conn.define_domain(xml)
  File "/usr/share/virt-manager/virtManager/connection.py", line 554, in define_domain
    return self._backend.defineXML(xml)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/libvirt.py", line 4543, in defineXML
    raise libvirtError('virDomainDefineXML() failed')
libvirt.libvirtError: unsupported configuration: chardev 'spicevmc' not supported without spice graphics

Here's also my XML:

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>win10</name>
  <uuid>1be073bd-cee6-4de7-9b34-fdfea4a318f3</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">8388608</memory>
  <currentMemory unit="KiB">8388608</currentMemory>
  <vcpu placement="static">8</vcpu>
  <os firmware="efi">
    <type arch="x86_64" machine="pc-q35-8.2">hvm</type>
    <firmware>
      <feature enabled="no" name="enrolled-keys"/>
      <feature enabled="yes" name="secure-boot"/>
    </firmware>
    <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd</loader>
    <nvram template="/usr/share/edk2/x64/OVMF_VARS.4m.fd">/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode="custom">
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vendor_id state="on" value="whatever"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <smm state="on"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on"/>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" discard="unmap"/>
      <source file="/var/lib/libvirt/images/win10.qcow2"/>
      <target dev="vda" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/mine_diver/Downloads/en-us_windows_10_consumer_editions_version_22h2_updated_march_2024_x64_dvd_2ff6c8a4.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/mine_diver/Downloads/virtio-win-0.1.248.iso"/>
      <target dev="sdc" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="2"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x1c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0x1d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
    </controller>
    <controller type="pci" index="15" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="15" port="0x1e"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x6"/>
    </controller>
    <controller type="pci" index="16" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:6a:51:4f"/>
      <source network="default"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
      <image compression="off"/>
    </graphics>
    <audio id="1" type="spice"/>
    <video>
      <model type="none"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
      </source>
      <rom file="/home/mine_diver/vbios.rom"/>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x02" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x0e6a"/>
        <product id="0x02c0"/>
      </source>
      <address type="usb" bus="0" port="4"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc08b"/>
      </source>
      <address type="usb" bus="0" port="5"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <watchdog model="itco" action="reset"/>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value="-device"/>
    <qemu:arg value="ich9-intel-hda,bus=pcie.0,addr=0x1b"/>
    <qemu:arg value="-device"/>
    <qemu:arg value="hda-micro,audiodev=hda"/>
    <qemu:arg value="-audiodev"/>
    <qemu:arg value="pa,id=hda,server=/run/user/1000/pulse/native"/>
  </qemu:commandline>
</domain>

Also, I should probably mention that my host system doesn't seem to support UEFI at all, it's just legacy BIOS, with no mention of UEFI whatsoever.

mineLdiver commented 7 months ago

Tried running the scripts on my phone through SSH, and it seems like everything executes just fine except for the efi-framebuffer part, which I guess makes sense given that I'm using BIOS. Gonna try comment that out and try again.

Also noticed that the boot screen cursor stops blinking after a couple of seconds after executing the script, so I suppose that indicates GPU getting properly detached and the image is just a ghost.

mineLdiver commented 7 months ago

Well, tested the new script through SSH, and the error is gone, however, when testing it with the VM, nothing seems to have changed.

I did notice some weird short flash on the terminal when running the VM compared to just running the script through SSH, but maybe it's just a random glitch.

mineLdiver commented 7 months ago

Another note - I didn't patch vBIOS. When I opened it in a hex editor, there already wasn't anything before hex 55, so I googled and found out that newer GPUs might not need patching, so I just used the unmodified dumped vBIOS.

QaidVoid commented 7 months ago

Were you able to remove Display Spice? This might be causing the issue (USB Redirector or whatever it's called).

<redirdev bus="usb" type="spicevmc">
  <address type="usb" bus="0" port="2"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
  <address type="usb" bus="0" port="3"/>
</redirdev>

You may also want to look at the VM log for more information on why the VM might be failing.

mineLdiver commented 7 months ago

Thanks for the response. No, I still couldn't remove Spice for some reason.

I'm currently in the process of reinstalling my system, so I'll try again from scratch after that and come back here.

amiablechief commented 5 months ago

You could try to follow this search result and see if it helps?

https://www.reddit.com/r/VFIO/comments/vpm0wq/cannot_remove_display_spice_and_video_qxl/

mineLdiver commented 5 months ago

Oh, nevermind actually, I was able to follow a different guide that was referenced on Arch Wiki - https://gitlab.com/risingprismtv/single-gpu-passthrough/-/wikis/1)-Preparations

And it worked fine. I have an issue with pipewire not working after entering the VM, but I've got a soundcard so I think I'll just figure out how to passthrough that. Thanks.