search

HikariKnight / quickpassthrough

A project to remove the complexity of setting up GPU passthrough for qemu
MIT License
681 stars 18 forks source link

Audio Controller not getting using kernel driver vfio-pci #22

Closed bytemaestr0 closed 5 months ago

bytemaestr0 commented 5 months ago

i have 2 gpus and virtualization enabled, so it should work this is the output of 

lspci -k | grep -E "vfio|NVIDIA"
07:00.0 VGA compatible controller: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)
        Kernel driver in use: vfio-pci
07:00.1 Audio device: NVIDIA Corporation GA106 High Definition Audio Controller (rev a1)

also, i checked dmesg and it showed that both devices are being added:

[    6.910118] vfio_pci: add [10de:2504[ffffffff:ffffffff]] class 0x000000/00000000
[    6.910137] vfio_pci: add [10de:228e[ffffffff:ffffffff]] class 0x000000/00000000
vfio_pci.ids=10de:2504,10de:228e

(they are both correct, i triple checked) and also in /etc/modprobe.d/vfio.conf i also added options vfio-pci ids=10de:2504,10de:228e but it didnt work either

HikariKnight commented 5 months ago

hello @bytemaestr0 can you provide the output from utils/ls-iommu -grk and post that here along with the debug.log. the utils folder is inside the folder that has the quickpassthrough binary

If you do not have the ls-iommu binary just start quickpassthrough once

note: also i updated your message for readability by using code blocks ``` put text here to put in the code block ``` will result in

put text here to put in the code block
bytemaestr0 commented 5 months ago

the output of utils/ls-iommu -grk is:

IOMMU Group  12: 0000:01:00.0 USB controller [0c03] Advanced Micro Devices, Inc. [AMD] unknown [1022:43d0] (rev 01)
        Subsystem: ASMedia Technology Inc. unknown [1b21:1142]
        Kernel driver in use: xhci_hcd
IOMMU Group  12: 0000:01:00.1 SATA controller [0106] Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset SATA Controller [1b21:1062]
        Kernel driver in use: ahci
IOMMU Group  12: 0000:01:00.2 PCI bridge [0604] Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Bridge [1b21:0201]
        Kernel driver in use: pcieport
IOMMU Group  12: 0000:02:00.0 PCI bridge [0604] Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Port [1b21:3306]
        Kernel driver in use: pcieport
IOMMU Group  12: 0000:02:01.0 PCI bridge [0604] Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Port [1b21:3306]
        Kernel driver in use: pcieport
IOMMU Group  12: 0000:02:02.0 PCI bridge [0604] Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Port [1b21:3306]
        Kernel driver in use: pcieport
IOMMU Group  12: 0000:02:03.0 PCI bridge [0604] Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Port [1b21:3306]
        Kernel driver in use: pcieport
IOMMU Group  12: 0000:02:04.0 PCI bridge [0604] Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Port [1b21:3306]
        Kernel driver in use: pcieport
IOMMU Group  12: 0000:02:09.0 PCI bridge [0604] Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Port [1b21:3306]
        Kernel driver in use: pcieport
IOMMU Group  12: 0000:03:00.0 Ethernet controller [0200] Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
        Subsystem: ASUSTeK Computer Inc. PRIME B450M-A Motherboard [1043:8677]
        Kernel driver in use: r8168
IOMMU Group  12: 0000:07:00.0 VGA compatible controller [0300] NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GA106 [GeForce RTX 3060 Lite Hash Rate] [1462:397d]
        Kernel driver in use: vfio-pci
IOMMU Group  12: 0000:07:00.1 Audio device [0403] NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GA106 High Definition Audio Controller [1462:397d]
        Kernel driver in use: snd_hda_intel
IOMMU Group  12: 0000:08:00.0 Non-Volatile memory controller [0108] Kingston Technology Company, Inc. A2000 NVMe SSD SM2263EN [2646:2263] (rev 03)
        Subsystem: Kingston Technology Company, Inc. A2000 NVMe SSD SM2263EN [2646:2263]
        Kernel driver in use: nvme
IOMMU Group  13: 0000:09:00.0 VGA compatible controller [0300] Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] [1002:67ff] (rev cf)
        Subsystem: Sapphire Technology Limited Radeon RX 560 [1da2:e348]
        Kernel driver in use: amdgpu
IOMMU Group  13: 0000:09:00.1 Audio device [0403] Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]
        Subsystem: Sapphire Technology Limited Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1da2:aae0]
        Kernel driver in use: snd_hda_intel

the debug.log is:

2024/03/24 16:59:31 
/etc/modprobe.d found on the system
Creating config/etc/modprobe.d

2024/03/24 16:59:31 
/etc/default found on the system
Creating config/etc/default

2024/03/24 16:59:31 
/etc/dracut.conf.d found on the system
Creating config/etc/dracut.conf.d

2024/03/24 16:59:31 
/etc/default/grub found on the system
Creating config/etc/default/grub

2024/03/24 16:59:31 
Executing: utils/ls-iommu -g -F vendor:,prod_name,optional_revision:,device_id

2024/03/24 16:59:31 
ls-iommu query returned
IOMMU Group  12: NVIDIA Corporation: GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1): [10de:2504]
IOMMU Group  13: Advanced Micro Devices, Inc. [AMD/ATI]: Baffin [Radeon RX 550 640SP / RX 560/560X] (rev cf): [1002:67ff]

2024/03/24 16:59:34 
Executing: utils/ls-iommu -g -r -i 12 -F vendor:,prod_name,optional_revision:,device_id

2024/03/24 16:59:36 
ls-iommu query returned
IOMMU Group  12: Advanced Micro Devices, Inc. [AMD]: 400 Series Chipset PCIe Bridge (rev 01): [1022:43c6]
IOMMU Group  12: Advanced Micro Devices, Inc. [AMD]: 400 Series Chipset PCIe Port (rev 01): [1022:43c7]
IOMMU Group  12: Advanced Micro Devices, Inc. [AMD]: 400 Series Chipset SATA Controller (rev 01): [1022:43c8]
IOMMU Group  12: Advanced Micro Devices, Inc. [AMD]: unknown (rev 01): [1022:43d0]
IOMMU Group  12: Kingston Technology Company, Inc.: A2000 NVMe SSD SM2263EN (rev 03): [2646:2263]
IOMMU Group  12: NVIDIA Corporation: GA106 High Definition Audio Controller (rev a1): [10de:228e]
IOMMU Group  12: NVIDIA Corporation: GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1): [10de:2504]
IOMMU Group  12: Realtek Semiconductor Co., Ltd.: RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15): [10ec:8168]

2024/03/24 16:59:45 
Executing: utils/ls-iommu -g -rr -i 12 -F vendor:,prod_name,optional_revision:,device_id

2024/03/24 16:59:48 
ls-iommu query returned
IOMMU Group  12: Advanced Micro Devices, Inc. [AMD]: 400 Series Chipset PCIe Bridge (rev 01): [1022:43c6]
IOMMU Group  12: Advanced Micro Devices, Inc. [AMD]: 400 Series Chipset PCIe Port (rev 01): [1022:43c7]
IOMMU Group  12: Advanced Micro Devices, Inc. [AMD]: 400 Series Chipset SATA Controller (rev 01): [1022:43c8]
IOMMU Group  12: Advanced Micro Devices, Inc. [AMD]: unknown (rev 01): [1022:43d0]
IOMMU Group  12: Kingston Technology Company, Inc.: A2000 NVMe SSD SM2263EN (rev 03): [2646:2263]
IOMMU Group  12: NVIDIA Corporation: GA106 High Definition Audio Controller (rev a1): [10de:228e]
IOMMU Group  12: NVIDIA Corporation: GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1): [10de:2504]
IOMMU Group  12: Realtek Semiconductor Co., Ltd.: RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15): [10ec:8168]

2024/03/24 17:00:12 
Executing: utils/ls-iommu -g -i 12 --rom

2024/03/24 17:00:12 
ls-iommu query returned
/sys/devices/pci0000:00/0000:00:01.3/0000:01:00.2/0000:02:04.0/0000:07:00.0/rom

2024/03/24 17:00:12 
Writing utils/dump_vbios.sh

2024/03/24 17:00:21 
Adding vfio_pci.disable_vga=1 to config/kernel_args

2024/03/24 17:00:21 
User has disabled vfio video output on host, adding disable_vga=1 to the optional hardcoded vfio_pci options

2024/03/24 17:00:21 
Writing config/etc/modprobe.d/vfio.conf:
## This is an autogenerated file that stubs your graphic card for use with vfio
## This file should be placed inside /etc/modprobe.d/
# Uncomment the line below to "hardcode" your graphic card to be bound to the vfio-pci driver.
# In most cases this should not be neccessary, it will also prevent you from turning off vfio in the bootloader.
#options vfio_pci ids=10de:228e,10de:2504 disable_vga=1

# Make sure vfio_pci is loaded before these modules: nvidia, nouveau, amdgpu and radeon
softdep nvidia pre: vfio vfio_pci
softdep nouveau pre: vfio vfio_pci
softdep amdgpu pre: vfio vfio_pci
softdep radeon pre: vfio vfio_pci

2024/03/24 17:00:21 
Writing to config/etc/dracut.conf.d/vfio.conf:
add_drivers+=" vfio_pci vfio vfio_iommu_type1 "

2024/03/24 17:00:21 
Configuring grub2 manually

2024/03/24 17:00:21 
Read /etc/default/grub:
# GRUB boot loader configuration

GRUB_DEFAULT='0'
GRUB_TIMEOUT='5'
GRUB_DISTRIBUTOR='EndeavourOS'
GRUB_CMDLINE_LINUX_DEFAULT='nowatchdog nvme_load=YES resume=UUID=2b98b356-0833-443f-8ce8-d6800e9cb6ac loglevel=3'
GRUB_CMDLINE_LINUX="iommu=pt amd_iommu=on vfio_pci.ids=10de:2504,10de:228e vfio_pci.disable_vga=1 rd.driver.pre=vfio_pci"

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
#GRUB_ENABLE_CRYPTODISK=y

# Set to 'countdown' or 'hidden' to change timeout behavior,
# press ESC key to display menu.
GRUB_TIMEOUT_STYLE=menu

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `videoinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY='true'

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
#GRUB_COLOR_NORMAL="light-blue/black"
#GRUB_COLOR_HIGHLIGHT="light-cyan/blue"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
GRUB_BACKGROUND='/usr/share/endeavouros/splash.png'
#GRUB_THEME="/path/to/gfxtheme"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

# Uncomment to make GRUB remember the last selection. This requires
# setting 'GRUB_DEFAULT=saved' above.
#GRUB_SAVEDEFAULT=true

# Uncomment to disable submenus in boot menu
GRUB_DISABLE_SUBMENU='false'

# Probing for other operating systems is disabled for security reasons. Read
# documentation on GRUB_DISABLE_OS_PROBER, if still want to enable this
# functionality install os-prober and uncomment to detect and include other
# operating systems.
GRUB_DISABLE_OS_PROBER=false

2024/03/24 17:00:21 
Replacing line in config/etc/default/grub:
GRUB_CMDLINE_LINUX="iommu=pt amd_iommu=on vfio_pci.ids=10de:2504,10de:228e vfio_pci.disable_vga=1 rd.driver.pre=vfio_pci"
With:
GRUB_CMDLINE_LINUX="iommu=pt amd_iommu=on vfio_pci.ids=10de:228e,10de:2504 vfio_pci.disable_vga=1 rd.driver.pre=vfio_pci"

2024/03/24 17:00:27 
Applying grub2 changes

2024/03/24 17:00:27 
Executing command:
sudo cp -v "config/etc/default/grub" /etc/default/grub

2024/03/24 17:00:27 
'config/etc/default/grub' -> '/etc/default/grub'

2024/03/24 17:00:29 
Generating grub configuration file ...
Found background: /usr/share/endeavouros/splash.png
Found linux image: /boot/vmlinuz-linux-lts
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux-lts.img
Found fallback initrd image(s) in /boot:  amd-ucode.img initramfs-linux-lts-fallback.img
Found linux image: /boot/vmlinuz-linux
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux.img
Found fallback initrd image(s) in /boot:  amd-ucode.img initramfs-linux-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Adding boot menu entry for UEFI Firmware Settings ...
done

2024/03/24 17:00:29 
'config/etc/modprobe.d/vfio.conf' -> '/etc/modprobe.d/vfio.conf'

2024/03/24 17:00:29 
'config/etc/dracut.conf.d/vfio.conf' -> '/etc/dracut.conf.d/vfio'

2024/03/24 17:00:29 
Executing: sudo dracut -f -v --kver 6.6.22-1-lts

2024/03/24 17:00:29 
dracut: Executing: /usr/bin/dracut -f -v --kver 6.6.22-1-lts
dracut: Can't write to /boot/efi/843f973e33514afdb46423d177c1bd29/6.6.22-1-lts: Directory /boot/efi/843f973e33514afdb46423d177c1bd29/6.6.22-1-lts does not exist or is not accessible.

it seems that the problem might have something to do with dracut but im not an expert.

HikariKnight commented 5 months ago

Well this is easy, the card you have selected will never work in the PCIe slot it is in, because it is in IOMMU group 12 which is shared with a bunch of other stuff, not sure why the audio controller doesnt bind to vfio-pci, but it might be related?

either way you might have better luck if you swap the nvidia card to use the pcie slot that the amd card is using.

The dialog where you select the gpu clearly states that the list should only contain the GPU and anything associated with it. image

bytemaestr0 commented 5 months ago

i mean i thought that if i went over to advanced, then manual select, it would somehow isolate those 2 from iommu group. that was my mistake. im going to look for a way to set it to another group, thanks for all the help!

HikariKnight commented 5 months ago

i mean i thought that if i went over to advanced, then manual select, it would somehow isolate those 2 from iommu group. that was my mistake. im going to look for a way to set it to another group, thanks for all the help!

no the advanced one is for situations where the other stuff in the group is fine to be passed through, but you shouldnt bind it to vfio-pci or in situations where you have an intel ARC card which will share the vendor ID with intel chipsets and the intel cpu and anything else from intel (which causes the relative checks to be unreliable, so you have to supply it manually then) 😄