search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.27k stars 14.25k forks source link

NixOS installer & nixos-generate-config don’t work with QEMU vIOMMU #340086

Open Zocker1999NET opened 2 months ago

Zocker1999NET commented 2 months ago

Describe the bug

When using NixOS on QEMU (in my case Proxmox) with vIOMMU enabled, the official NixOS installers cannot boot (keeps stuck in Stage 1 because it cannot find its IDE/SATA/SCSI disks). Also, after managing to boot a modified NixOS installer iso, nixos-generate-config fails to identify that the virtio_iommu kernel module is required for booting.

Proxmox VM configuration ![image](https://github.com/user-attachments/assets/b65d23b2-6e06-4004-b4a0-6b20ad64e6f2) ![image](https://github.com/user-attachments/assets/294700a3-4837-4f67-8225-621c554f7b8b) I also tested it with the default machine type `i440fx`, but that shows the same behavior as with `q35`, at least as long as `vIOMMU` is enabled.

Steps To Reproduce

Steps to reproduce the behavior of the installer:

  1. Have a Proxmox setup & configure a VM as described above
  2. Download the NixOS installation ISO (any variant) & "insert" that to the VM
  3. Try to boot the VM

Steps to reproduce the behavior of nixos-generate-config:

  1. Boot a modified NixOS installation (requires boot.initrd.availableKernelModules = [ "virtio_iommu" ];)
  2. Execute nixos-generate-config
  3. Execute nixos-rebuild repl
  4. Check with builtins.elem "virtio_iommu" config.boot.initrd.availableKernelModules

Expected behavior

The VM just boots, especially because the installer is supposed to work on any hardware.

Screenshots

NixOS installer trying to boot ![image](https://github.com/user-attachments/assets/03ad8721-2321-4889-a7e2-9ba1832ecf4e)

Additional context

I have worked a lot of time into investigating this & fixed it for myself. As already mentioned, the missing culprit is virtio_iommu, hence by adding that module to boot.initrd.availableKernelModules, NixOS boots in these VMs.

However, I opened an issue aside of just creating PR because I think it would be reasonable to fix both the installer & nixos-generate-config at the same time, otherwise there might be some people which wonder more about that the installer boots but not an installed system later on. And I’m unsure how to approach patching nixos-generate-config, so I’m asking for help here.

To aid fixing, I executed nix shell nixpkgs#pciutils --command "lspci" -v in the VM with vIOMMU disabled & enabled, producing following outputs & diff:

`lspci -v` with vIOMMU disabled ``` 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0 Kernel modules: intel_agp 00:01.0 VGA compatible controller: Device 1234:1111 (rev 02) (prog-if 00 [VGA controller]) Subsystem: Red Hat, Inc. Device 1100 Flags: bus master, fast devsel, latency 0 Memory at 80000000 (32-bit, prefetchable) [size=16M] Memory at 8224b000 (32-bit, non-prefetchable) [size=4K] Expansion ROM at 000c0000 [disabled] [size=128K] Kernel driver in use: bochs-drm Kernel modules: bochs 00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 16 I/O ports at 9100 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 17 I/O ports at 90e0 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 18 I/O ports at 90c0 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03) (prog-if 20 [EHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 19 Memory at 8224a000 (32-bit, non-prefetchable) [size=4K] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 47 Memory at 82240000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+ Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode]) Subsystem: Red Hat, Inc. Device 0000 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at 82249000 (32-bit, non-prefetchable) [size=4K] Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 1000-1fff [size=4K] [16-bit] Memory behind bridge: 82000000-821fffff [size=2M] [32-bit] Prefetchable memory behind bridge: c000000000-c01fffffff [size=512M] [32-bit] Capabilities: [54] Express Root Port (Slot+), IntMsgNum 0 Capabilities: [48] MSI-X: Enable+ Count=1 Masked- Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000 Capabilities: [100] Advanced Error Reporting Capabilities: [148] Access Control Services Kernel driver in use: pcieport 00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode]) Subsystem: Red Hat, Inc. Device 0000 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at 82248000 (32-bit, non-prefetchable) [size=4K] Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 I/O behind bridge: 2000-2fff [size=4K] [16-bit] Memory behind bridge: 81e00000-81ffffff [size=2M] [32-bit] Prefetchable memory behind bridge: c020000000-c03fffffff [size=512M] [32-bit] Capabilities: [54] Express Root Port (Slot+), IntMsgNum 0 Capabilities: [48] MSI-X: Enable+ Count=1 Masked- Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000 Capabilities: [100] Advanced Error Reporting Capabilities: [148] Access Control Services Kernel driver in use: pcieport 00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode]) Subsystem: Red Hat, Inc. Device 0000 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at 82247000 (32-bit, non-prefetchable) [size=4K] Bus: primary=00, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 3000-3fff [size=4K] [16-bit] Memory behind bridge: 81c00000-81dfffff [size=2M] [32-bit] Prefetchable memory behind bridge: c040000000-c05fffffff [size=512M] [32-bit] Capabilities: [54] Express Root Port (Slot+), IntMsgNum 0 Capabilities: [48] MSI-X: Enable+ Count=1 Masked- Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000 Capabilities: [100] Advanced Error Reporting Capabilities: [148] Access Control Services Kernel driver in use: pcieport 00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode]) Subsystem: Red Hat, Inc. Device 0000 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at 82246000 (32-bit, non-prefetchable) [size=4K] Bus: primary=00, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 4000-4fff [size=4K] [16-bit] Memory behind bridge: 81a00000-81bfffff [size=2M] [32-bit] Prefetchable memory behind bridge: c060000000-c07fffffff [size=512M] [32-bit] Capabilities: [54] Express Root Port (Slot+), IntMsgNum 0 Capabilities: [48] MSI-X: Enable+ Count=1 Masked- Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000 Capabilities: [100] Advanced Error Reporting Capabilities: [148] Access Control Services Kernel driver in use: pcieport 00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 16 I/O ports at 90a0 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 17 I/O ports at 9080 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 18 I/O ports at 9060 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03) (prog-if 20 [EHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 19 Memory at 82245000 (32-bit, non-prefetchable) [size=4K] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode]) Flags: bus master, 66MHz, fast devsel, latency 0 Bus: primary=00, secondary=05, subordinate=09, sec-latency=0 I/O behind bridge: 6000-8fff [size=12K] [16-bit] Memory behind bridge: 81000000-818fffff [size=9M] [32-bit] Prefetchable memory behind bridge: c080000000-c0ffffffff [size=2G] [32-bit] Capabilities: [50] Subsystem: Device 0000:0000 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0 Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) (prog-if 01 [AHCI 1.0]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 30 I/O ports at 9040 [size=32] Memory at 82244000 (32-bit, non-prefetchable) [size=4K] Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [a8] SATA HBA v1.0 Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 16 I/O ports at 9000 [size=64] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode]) Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 21 Memory at 81800000 (64-bit, non-prefetchable) [size=256] Bus: primary=05, secondary=06, subordinate=06, sec-latency=0 I/O behind bridge: 8000-8fff [size=4K] [16-bit] Memory behind bridge: 81600000-817fffff [size=2M] [32-bit] Prefetchable memory behind bridge: c0e0000000-c0ffffffff [size=512M] [32-bit] Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [48] Slot ID: 0 slots, First+, chassis 01 Capabilities: [40] Hot-plug capable 05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode]) Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 22 Memory at 81801000 (64-bit, non-prefetchable) [size=256] Bus: primary=05, secondary=07, subordinate=07, sec-latency=0 I/O behind bridge: 7000-7fff [size=4K] [16-bit] Memory behind bridge: 81400000-815fffff [size=2M] [32-bit] Prefetchable memory behind bridge: c0c0000000-c0dfffffff [size=512M] [32-bit] Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [48] Slot ID: 0 slots, First+, chassis 02 Capabilities: [40] Hot-plug capable 05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode]) Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 11 Memory at 81802000 (64-bit, non-prefetchable) [size=256] Bus: primary=05, secondary=08, subordinate=08, sec-latency=0 I/O behind bridge: [disabled] [16-bit] Memory behind bridge: 81200000-813fffff [size=2M] [32-bit] Prefetchable memory behind bridge: c0a0000000-c0bfffffff [size=512M] [32-bit] Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [48] Slot ID: 0 slots, First+, chassis 03 Capabilities: [40] Hot-plug capable 05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode]) Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 20 Memory at 81803000 (64-bit, non-prefetchable) [size=256] Bus: primary=05, secondary=09, subordinate=09, sec-latency=0 I/O behind bridge: 6000-6fff [size=4K] [16-bit] Memory behind bridge: 81000000-811fffff [size=2M] [32-bit] Prefetchable memory behind bridge: c080000000-c09fffffff [size=512M] [32-bit] Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [48] Slot ID: 0 slots, First+, chassis 04 Capabilities: [40] Hot-plug capable 06:12.0 Ethernet controller: Red Hat, Inc. Virtio network device Subsystem: Red Hat, Inc. Device 0001 Physical Slot: 18-1 Flags: bus master, fast devsel, latency 0, IRQ 23 I/O ports at 8040 [size=64] Memory at 81601000 (32-bit, non-prefetchable) [size=4K] Memory at c0e0004000 (64-bit, prefetchable) [size=16K] Expansion ROM at 81640000 [disabled] [size=256K] Capabilities: [98] MSI-X: Enable+ Count=4 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci 06:13.0 Ethernet controller: Red Hat, Inc. Virtio network device Subsystem: Red Hat, Inc. Device 0001 Physical Slot: 19-1 Flags: bus master, fast devsel, latency 0, IRQ 20 I/O ports at 8000 [size=64] Memory at 81600000 (32-bit, non-prefetchable) [size=4K] Memory at c0e0000000 (64-bit, prefetchable) [size=16K] Expansion ROM at 81680000 [disabled] [size=256K] Capabilities: [98] MSI-X: Enable+ Count=4 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci 07:1d.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG Subsystem: Red Hat, Inc. Device 0004 Physical Slot: 29-2 Flags: bus master, fast devsel, latency 0, IRQ 23 I/O ports at 7000 [size=32] Memory at 81400000 (32-bit, non-prefetchable) [size=4K] Memory at c0c0000000 (64-bit, prefetchable) [size=16K] Capabilities: [98] MSI-X: Enable+ Count=2 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci 09:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI Subsystem: Red Hat, Inc. Device 0008 Physical Slot: 1-3 Flags: bus master, fast devsel, latency 0, IRQ 21 I/O ports at 6040 [size=64] Memory at 81001000 (32-bit, non-prefetchable) [size=4K] Memory at c080004000 (64-bit, prefetchable) [size=16K] Capabilities: [98] MSI-X: Enable+ Count=5 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci 09:02.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI Subsystem: Red Hat, Inc. Device 0008 Physical Slot: 2-3 Flags: bus master, fast devsel, latency 0, IRQ 22 I/O ports at 6000 [size=64] Memory at 81000000 (32-bit, non-prefetchable) [size=4K] Memory at c080000000 (64-bit, prefetchable) [size=16K] Capabilities: [98] MSI-X: Enable+ Count=5 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci ```
`lspci -v` with vIOMMU enabled ``` 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IOMMU group 8 Kernel modules: intel_agp 00:01.0 VGA compatible controller: Device 1234:1111 (rev 02) (prog-if 00 [VGA controller]) Subsystem: Red Hat, Inc. Device 1100 Flags: bus master, fast devsel, latency 0, IOMMU group 9 Memory at 80000000 (32-bit, prefetchable) [size=16M] Memory at 8224b000 (32-bit, non-prefetchable) [size=4K] Expansion ROM at 000c0000 [disabled] [size=128K] Kernel driver in use: bochs-drm Kernel modules: bochs 00:02.0 Unclassified device [00ff]: Red Hat, Inc. Device 1057 (rev 01) Subsystem: Red Hat, Inc. Device 1100 Flags: bus master, fast devsel, latency 0, IRQ 22 Memory at c100000000 (64-bit, prefetchable) [size=16K] Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci 00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 5 I/O ports at 9100 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 5 I/O ports at 90e0 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 18, IOMMU group 5 I/O ports at 90c0 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03) (prog-if 20 [EHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 19, IOMMU group 5 Memory at 8224a000 (32-bit, non-prefetchable) [size=4K] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 47, IOMMU group 10 Memory at 82240000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+ Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode]) Subsystem: Red Hat, Inc. Device 0000 Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 1 Memory at 82249000 (32-bit, non-prefetchable) [size=4K] Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 1000-1fff [size=4K] [16-bit] Memory behind bridge: 82000000-821fffff [size=2M] [32-bit] Prefetchable memory behind bridge: c000000000-c01fffffff [size=512M] [32-bit] Capabilities: [54] Express Root Port (Slot+), IntMsgNum 0 Capabilities: [48] MSI-X: Enable+ Count=1 Masked- Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000 Capabilities: [100] Advanced Error Reporting Capabilities: [148] Access Control Services Kernel driver in use: pcieport 00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode]) Subsystem: Red Hat, Inc. Device 0000 Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 2 Memory at 82248000 (32-bit, non-prefetchable) [size=4K] Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 I/O behind bridge: 2000-2fff [size=4K] [16-bit] Memory behind bridge: 81e00000-81ffffff [size=2M] [32-bit] Prefetchable memory behind bridge: c020000000-c03fffffff [size=512M] [32-bit] Capabilities: [54] Express Root Port (Slot+), IntMsgNum 0 Capabilities: [48] MSI-X: Enable+ Count=1 Masked- Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000 Capabilities: [100] Advanced Error Reporting Capabilities: [148] Access Control Services Kernel driver in use: pcieport 00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode]) Subsystem: Red Hat, Inc. Device 0000 Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 3 Memory at 82247000 (32-bit, non-prefetchable) [size=4K] Bus: primary=00, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 3000-3fff [size=4K] [16-bit] Memory behind bridge: 81c00000-81dfffff [size=2M] [32-bit] Prefetchable memory behind bridge: c040000000-c05fffffff [size=512M] [32-bit] Capabilities: [54] Express Root Port (Slot+), IntMsgNum 0 Capabilities: [48] MSI-X: Enable+ Count=1 Masked- Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000 Capabilities: [100] Advanced Error Reporting Capabilities: [148] Access Control Services Kernel driver in use: pcieport 00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode]) Subsystem: Red Hat, Inc. Device 0000 Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 4 Memory at 82246000 (32-bit, non-prefetchable) [size=4K] Bus: primary=00, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 4000-4fff [size=4K] [16-bit] Memory behind bridge: 81a00000-81bfffff [size=2M] [32-bit] Prefetchable memory behind bridge: c060000000-c07fffffff [size=512M] [32-bit] Capabilities: [54] Express Root Port (Slot+), IntMsgNum 0 Capabilities: [48] MSI-X: Enable+ Count=1 Masked- Capabilities: [40] Subsystem: Red Hat, Inc. Device 0000 Capabilities: [100] Advanced Error Reporting Capabilities: [148] Access Control Services Kernel driver in use: pcieport 00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 6 I/O ports at 90a0 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 6 I/O ports at 9080 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03) (prog-if 00 [UHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 18, IOMMU group 6 I/O ports at 9060 [size=32] Kernel driver in use: uhci_hcd Kernel modules: uhci_hcd 00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03) (prog-if 20 [EHCI]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 19, IOMMU group 6 Memory at 82245000 (32-bit, non-prefetchable) [size=4K] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode]) Flags: bus master, 66MHz, fast devsel, latency 0, IOMMU group 0 Bus: primary=00, secondary=05, subordinate=09, sec-latency=0 I/O behind bridge: 6000-8fff [size=12K] [16-bit] Memory behind bridge: 81000000-818fffff [size=9M] [32-bit] Prefetchable memory behind bridge: c080000000-c0ffffffff [size=2G] [32-bit] Capabilities: [50] Subsystem: Device 0000:0000 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IOMMU group 7 Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) (prog-if 01 [AHCI 1.0]) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 46, IOMMU group 7 I/O ports at 9040 [size=32] Memory at 82244000 (32-bit, non-prefetchable) [size=4K] Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [a8] SATA HBA v1.0 Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) Subsystem: Red Hat, Inc. QEMU Virtual Machine Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 7 I/O ports at 9000 [size=64] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode]) Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 21 Memory at 81800000 (64-bit, non-prefetchable) [size=256] Bus: primary=05, secondary=06, subordinate=06, sec-latency=0 I/O behind bridge: 8000-8fff [size=4K] [16-bit] Memory behind bridge: 81600000-817fffff [size=2M] [32-bit] Prefetchable memory behind bridge: c0e0000000-c0ffffffff [size=512M] [32-bit] Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [48] Slot ID: 0 slots, First+, chassis 01 Capabilities: [40] Hot-plug capable 05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode]) Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 22 Memory at 81801000 (64-bit, non-prefetchable) [size=256] Bus: primary=05, secondary=07, subordinate=07, sec-latency=0 I/O behind bridge: 7000-7fff [size=4K] [16-bit] Memory behind bridge: 81400000-815fffff [size=2M] [32-bit] Prefetchable memory behind bridge: c0c0000000-c0dfffffff [size=512M] [32-bit] Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [48] Slot ID: 0 slots, First+, chassis 02 Capabilities: [40] Hot-plug capable 05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode]) Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 11 Memory at 81802000 (64-bit, non-prefetchable) [size=256] Bus: primary=05, secondary=08, subordinate=08, sec-latency=0 I/O behind bridge: [disabled] [16-bit] Memory behind bridge: 81200000-813fffff [size=2M] [32-bit] Prefetchable memory behind bridge: c0a0000000-c0bfffffff [size=512M] [32-bit] Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [48] Slot ID: 0 slots, First+, chassis 03 Capabilities: [40] Hot-plug capable 05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode]) Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 20 Memory at 81803000 (64-bit, non-prefetchable) [size=256] Bus: primary=05, secondary=09, subordinate=09, sec-latency=0 I/O behind bridge: 6000-6fff [size=4K] [16-bit] Memory behind bridge: 81000000-811fffff [size=2M] [32-bit] Prefetchable memory behind bridge: c080000000-c09fffffff [size=512M] [32-bit] Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [48] Slot ID: 0 slots, First+, chassis 04 Capabilities: [40] Hot-plug capable 06:12.0 Ethernet controller: Red Hat, Inc. Virtio network device Subsystem: Red Hat, Inc. Device 0001 Physical Slot: 18-1 Flags: bus master, fast devsel, latency 0, IRQ 23 I/O ports at 8040 [size=64] Memory at 81601000 (32-bit, non-prefetchable) [size=4K] Memory at c0e0004000 (64-bit, prefetchable) [size=16K] Expansion ROM at 81640000 [disabled] [size=256K] Capabilities: [98] MSI-X: Enable+ Count=4 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci 06:13.0 Ethernet controller: Red Hat, Inc. Virtio network device Subsystem: Red Hat, Inc. Device 0001 Physical Slot: 19-1 Flags: bus master, fast devsel, latency 0, IRQ 20 I/O ports at 8000 [size=64] Memory at 81600000 (32-bit, non-prefetchable) [size=4K] Memory at c0e0000000 (64-bit, prefetchable) [size=16K] Expansion ROM at 81680000 [disabled] [size=256K] Capabilities: [98] MSI-X: Enable+ Count=4 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci 07:1d.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG Subsystem: Red Hat, Inc. Device 0004 Physical Slot: 29-2 Flags: bus master, fast devsel, latency 0, IRQ 23 I/O ports at 7000 [size=32] Memory at 81400000 (32-bit, non-prefetchable) [size=4K] Memory at c0c0000000 (64-bit, prefetchable) [size=16K] Capabilities: [98] MSI-X: Enable+ Count=2 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci 09:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI Subsystem: Red Hat, Inc. Device 0008 Physical Slot: 1-3 Flags: bus master, fast devsel, latency 0, IRQ 21 I/O ports at 6040 [size=64] Memory at 81001000 (32-bit, non-prefetchable) [size=4K] Memory at c080004000 (64-bit, prefetchable) [size=16K] Capabilities: [98] MSI-X: Enable+ Count=5 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci 09:02.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI Subsystem: Red Hat, Inc. Device 0008 Physical Slot: 2-3 Flags: bus master, fast devsel, latency 0, IRQ 22 I/O ports at 6000 [size=64] Memory at 81000000 (32-bit, non-prefetchable) [size=4K] Memory at c080000000 (64-bit, prefetchable) [size=16K] Capabilities: [98] MSI-X: Enable+ Count=5 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Kernel modules: virtio_pci ```
diff of both `lspci -v` outputs left/right = without/with vIOMMU ```diff 3c3 < Flags: bus master, fast devsel, latency 0 --- > Flags: bus master, fast devsel, latency 0, IOMMU group 8 8c8 < Flags: bus master, fast devsel, latency 0 --- > Flags: bus master, fast devsel, latency 0, IOMMU group 9 14a15,26 > 00:02.0 Unclassified device [00ff]: Red Hat, Inc. Device 1057 (rev 01) > Subsystem: Red Hat, Inc. Device 1100 > Flags: bus master, fast devsel, latency 0, IRQ 22 > Memory at c100000000 (64-bit, prefetchable) [size=16K] > Capabilities: [84] Vendor Specific Information: VirtIO: > Capabilities: [70] Vendor Specific Information: VirtIO: Notify > Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg > Capabilities: [50] Vendor Specific Information: VirtIO: ISR > Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg > Kernel driver in use: virtio-pci > Kernel modules: virtio_pci > 17c29 < Flags: bus master, fast devsel, latency 0, IRQ 16 --- > Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 5 24c36 < Flags: bus master, fast devsel, latency 0, IRQ 17 --- > Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 5 31c43 < Flags: bus master, fast devsel, latency 0, IRQ 18 --- > Flags: bus master, fast devsel, latency 0, IRQ 18, IOMMU group 5 38c50 < Flags: bus master, fast devsel, latency 0, IRQ 19 --- > Flags: bus master, fast devsel, latency 0, IRQ 19, IOMMU group 5 45c57 < Flags: bus master, fast devsel, latency 0, IRQ 47 --- > Flags: bus master, fast devsel, latency 0, IRQ 47, IOMMU group 10 53c65 < Flags: bus master, fast devsel, latency 0, IRQ 16 --- > Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 1 68c80 < Flags: bus master, fast devsel, latency 0, IRQ 16 --- > Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 2 83c95 < Flags: bus master, fast devsel, latency 0, IRQ 16 --- > Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 3 98c110 < Flags: bus master, fast devsel, latency 0, IRQ 16 --- > Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 4 113c125 < Flags: bus master, fast devsel, latency 0, IRQ 16 --- > Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 6 120c132 < Flags: bus master, fast devsel, latency 0, IRQ 17 --- > Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 6 127c139 < Flags: bus master, fast devsel, latency 0, IRQ 18 --- > Flags: bus master, fast devsel, latency 0, IRQ 18, IOMMU group 6 134c146 < Flags: bus master, fast devsel, latency 0, IRQ 19 --- > Flags: bus master, fast devsel, latency 0, IRQ 19, IOMMU group 6 140c152 < Flags: bus master, 66MHz, fast devsel, latency 0 --- > Flags: bus master, 66MHz, fast devsel, latency 0, IOMMU group 0 149c161 < Flags: bus master, fast devsel, latency 0 --- > Flags: bus master, fast devsel, latency 0, IOMMU group 7 155c167 < Flags: bus master, fast devsel, latency 0, IRQ 30 --- > Flags: bus master, fast devsel, latency 0, IRQ 46, IOMMU group 7 165c177 < Flags: bus master, fast devsel, latency 0, IRQ 16 --- > Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 7 ```

It is obvious that a new device called 00:02.0 Unclassified device [00ff]: Red Hat, Inc. Device 1057 (rev 01) appears when vIOMMU is enabled. However, I have no idea which are the unique identifying bits on this output which should be used in nixos-generate-config to it can add virtio_iommu when required.

Notify maintainers

The nixos-install-tools has no maintainers listed, so I list the last 5 comitters to nixos-generate-config.pl: @bjornfor @anthonyroussel @Atemu @Mic92 @lheckemann

Add a :+1: reaction to issues you find important.

Zocker1999NET commented 2 months ago

Oh, I just discovered that nixos-generate-config also refers to https://github.com/NixOS/nixpkgs/blob/671844dcef9cdb5a77574bf9289b58f01d238b4d/nixos/modules/profiles/qemu-guest.nix when detecting QEMU, meaning that should be sufficient to add the module there as well. However, I’m unsure if this is sensible given this profile is used in a lot of places.

Mic92 commented 2 months ago

How does one simulate this with plain qemu?

Zocker1999NET commented 2 months ago

How does one simulate this with plain qemu?

I have literally no idea on how to do that, as I only use QEMU via Proxmox interface. But maybe the command line that Proxmox ultimately executes might be helpful? (I replaced sensitive data and ids)

/usr/bin/kvm -id 111 -name nixtest,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/111.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/111.pid -daemonize -smbios type=1,uuid=e255bb8e-d3c1-4773-a987-71c1957366c8 -drive if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd -drive if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/dev/zvol/pve/Usedata/VMs/vm-111-disk-0,size=540672 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/111.vnc,password=on -cpu host,+kvm_pv_eoi,+kvm_pv_unhalt -m 16384 -object memory-backend-ram,id=ram-node0,size=16384M -numa node,nodeid=0,cpus=0-3,memdev=ram-node0 -object iothread,id=iothread-virtioscsi0 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device vmgenid,guid=f3366558-c223-4987-a816-be7bbc12e4d6 -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device VGA,id=vga,bus=pcie.0,addr=0x1 -chardev socket,path=/var/run/qemu-server/111.qga,server=on,wait=off,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0,max-bytes=1024,period=1000,bus=pci.1,addr=0x1d -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on -iscsi initiator-name=iqn.1993-08.org.debian:01:e744c39177a4 -device virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1,iothread=iothread-virtioscsi0 -drive file=/dev/zvol/pve/Usedata/VMs/vm-111-disk-1,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap -device scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,rotation_rate=1,bootindex=101 -device virtio-scsi-pci,id=virtioscsi1,bus=pci.3,addr=0x2 -drive file=/var/lib/vz/template/iso/nixos-minimal-24.05.3642.883180e6550c-x86_64-linux.iso,if=none,id=drive-scsi1,media=cdrom,aio=io_uring -device scsi-cd,bus=virtioscsi1.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,bootindex=100 -netdev type=tap,id=net0,ifname=tap111i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:1D:8E:2E,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=102 -device virtio-iommu-pci -machine type=q35+pve0

Or you might utilize nested virtualization as setting up a Proxmox (which is enough for testing purposes) is a matter "inserting ISO & pressing okay until a WebUI starts" which is possible in 3 to 5 minutes.