Doesn't seem to work for 7950x3d on PVE 7.4-3

[schickeneder]

I followed the instruction to implement iGPU passthrough on Ubuntu 22.04 with a 7950x3d CPU.

Regardless of the Machine and BIOS combinations I will get "amdgpu: Unable to locate a BIOS ROM" (after extracting and also trying the one in the repo) And depending on the settings I will also get "Invalid PCI ROM header signature: expecting 0xaa55, got 0x4556"

Perhaps it is this problem: https://forum.proxmox.com/threads/amd-gpu-firmware-bios-missing-amdgpu-fatal-error.134739/ but I can't implement that fix because Asrock doesn't have a supporting beta bios version for B650D4U-2L2T/BCM as far as I can tell

[kar200]

Sorry to use your thread but came here to say it worked for me on the 7800x3d + Asus B650m-Plus Wifi board.

I am using Gentoo Linux but I am sure it would work on other distro (for my setup).

I am using Kernel 6.8.5 and Gnome with libvirt.

@schickeneder , where does the message popup? Sorry to sound dumb but did you copy the files to a location where the hyoervisor has access? I recall having a similar issue in the past where the actual files could not be accessed by libvirt.

I used the vbios extracted (cat command or the C code here as hey are the same) and I used the "AMDGopDriver.rom" from here.

My libvirt config looks like this. I added the PCI's using libvirts add command (to avoid bus conflict) then just updated the rombar line.

<hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x0f" slot="0x00" function="0x0"/>
      </source>
      <rom bar="on" file="/var/lib/libvirt/images/vbios_1002_164e.bin"/>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0" multifunction="on"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x0f" slot="0x00" function="0x1"/>
      </source>
      <rom bar="on" file="/var/lib/libvirt/images/AMDGopDriver.rom"/>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>

I am using Windows 11 with Q35 and UEFI. The output throtouhg hdmi is working perfctly (27in 144hz) and as a bonus the sound comes from the HDMI connection!

[schickeneder]

@kar200 Those errors were in Linux dmesg. I was trying to pass through to Ubuntu VMs I also setup a Windows VM using the instructions on the page with Seabios and get error 43. So it seems like the issue persists across OSes.

Here's one of the Ubuntu configs:

agent: 1
args: -cpu 'host,-hypervisor,kvm=off'
bios: seabios
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
efidisk0: local-lvm:vm-104-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:0d:00.0,pcie=1,rombar=0,romfile=vbios_7950x3d.bin
hostpci1: 0000:0d:00.1,pcie=1,rombar=0
ide2: local:iso/ubuntu-22.04.4-desktop-amd64.iso,media=cdrom,size=4899762K
machine: q35
memory: 4096
meta: creation-qemu=7.2.0,ctime=1712759293
name: Ubu22-GPU
net0: virtio=2A:D5:73:45:7F:F0,bridge=vmbr1,firewall=1,queues=4
numa: 0
ostype: l26
scsi0: local-lvm:vm-104-disk-1,iothread=1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=1cb30df5-c783-4ea6-a8fe-e39380f16956
sockets: 1
vmgenid: 8cdbfbe1-9e31-4955-a804-e37548608469

[kar200]

@schickeneder romfile doesn't seem to have the full path to the file. Try to first boot without the romfile option (just to make sure the rest of the syntax is correct) and then use the full path to the file.

Edit: It seems it's the way promox has it. IF you placed the file in /usr/share/kvm it should work

https://pve.proxmox.com/wiki/PCI_Passthrough#The_.27romfile.27_option

[schickeneder]

I've tried it with both paths. I initially didn't specify full path--as per the guide, but it is in /usr/share/kvm with same permissions as all the other files there. Same result:

Also updated to newest beta version of BIOS. No change.

With OVMF (UEFI) and i440fx, I get similar but slightly different errors:

Not sure why it shows the wrong header, because I verified that the first bytes are 0xaa55, not 0x4556..

[kar200]

I think I might have reached my modest knowledge in the area. I will just add a couple of things.

I am using the UFI bios by the way (default option in libvirt for windows 11). Apparently you have to also add the AMDGopDriver.rom option to the sound card and define the GPU as "mutlifunction".

I have a 6800xt GPU as my main gpu . I have a single monitor and use the DP for my Linux. I use the HDMI for the vm. With my hdmi plugged in during boot I noticed that Ubuntu gives me errors. WIndows seems to boot first with the basic display driver as I get the horrible resolution. After a few seconds from booting it seems it sort of setup the AMD driver and I get everything working (sound as well). I have a feeling it's to do with the reset bug service installed.

When I tried the same with Ubuntu I got errors. I tried again booting the host PC with the monitor unplugged - Booted from the Ubuntu ISO installer and then connected the HDMI seemed to have worked.

I did the same with the windows and then booted booted the Ubuntu ISO from the same VM seems to have worked. There is definately something getting fixed by the AMD reset bug.

I would say try always to reboot with the Igpu cable removed. Try and connect it after the vm boots or just before. Make sure also the driver in use for the igpu is vfio-pci.

Here are some other options I set

cat /etc/modprobe.d/vfio.conf           
softdep drm pre: vfio-pci
softdep amdgpu pre: vfi-pci
softdep snd_hda_intel pre: vfi-pci

cat /boot/refind_linux.conf  
"Gentoo Gnome"  "root=PARTUUID=c0036deb-5560-43f2-b98d-8b973d0a0b39 rootflags=subvol=@gentoo_gnome rw iommu=pt vfio-pci.ids=1002:164e,1002:1640"

I am using refind by the way it's just to show the kernel options of iommu=pt and the binding of vfio-pci.ids.

This is my complete XML from libvirt. Only changed a few things from default (the ones I mentionned above) plus evdev for sharin mouse and keyboard and cpu options.

[wrobelda]

I have the same issue here.

For the record, this is not KVM failing to locate BIOS binary file on host filesystem — you can confirm that by changing the romfile to something non-existent and the VM will fail to start.

I am on Ryzen 9 7900 here.

FYI, the 'Invalid PCI ROM header signature: expecting 0xaa55, got 0x4556" is most likely due to the fact that the BIOS is designed for PCIe, so it won't work if passed through as regular PCI device.