search

xcp-ng / xcp

Entry point for issues and wiki. Also contains some scripts and sources.
https://xcp-ng.org
1.31k stars 74 forks source link

PCI Passthrough: xen-pciback.hide yet still visible to host #387

Closed jeremfg closed 3 years ago

jeremfg commented 4 years ago

Fresh install of xcp-ng 8.1.0 on a X11SDV-8C-TP8F motherboard from supermicro

I'm trying to setup PCI passthrough for a few devices I want to pass to VMs. However, there seems to be a conflict between host and guest, both "seeing" the devices and thus not working.

I'm focusing on three 1GbE NICs here, denoted by IDs (0000:66:00.1)(0000:66:00.2)(0000:66:00.3)

Here is what I see on the host:

[03:38 kepler ~]# xl pci-assignable-list
0000:00:17.0
0000:b7:00.2
0000:66:00.2
0000:b7:00.0
0000:b7:00.3
0000:66:00.3
0000:b7:00.1
0000:66:00.1
[03:38 kepler ~]# lspci
00:00.0 Host bridge: Intel Corporation Sky Lake-E DMI3 Registers (rev 04)
00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
00:05.2 System peripheral: Intel Corporation Sky Lake-E RAS (rev 04)
00:05.4 PIC: Intel Corporation Sky Lake-E IOAPIC (rev 04)
00:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
00:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
00:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
00:11.0 Unassigned class [ff00]: Intel Corporation C620 Series Chipset Family MROM 0 (rev 04)
00:11.1 Unassigned class [ff00]: Intel Corporation C620 Series Chipset Family MROM 1 (rev 04)
00:11.5 SATA controller: Intel Corporation C620 Series Chipset Family SSATA Controller [AHCI mode] (rev 04)
00:14.0 USB controller: Intel Corporation C620 Series Chipset Family USB 3.0 xHCI Controller (rev 04)
00:14.2 Signal processing controller: Intel Corporation C620 Series Chipset Family Thermal Subsystem (rev 04)
00:16.0 Communication controller: Intel Corporation C620 Series Chipset Family MEI Controller #1 (rev 04)
00:16.1 Communication controller: Intel Corporation C620 Series Chipset Family MEI Controller #2 (rev 04)
00:16.4 Communication controller: Intel Corporation C620 Series Chipset Family MEI Controller #3 (rev 04)
00:17.0 SATA controller: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] (rev 04)
00:1c.0 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #1 (rev f4)
00:1c.4 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #5 (rev f4)
00:1d.0 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #9 (rev f4)
00:1f.0 ISA bridge: Intel Corporation Device a1c8 (rev 04)
00:1f.2 Memory controller: Intel Corporation C620 Series Chipset Family Power Management Controller (rev 04)
00:1f.4 SMBus: Intel Corporation C620 Series Chipset Family SMBus (rev 04)
00:1f.5 Serial bus controller [0c80]: Intel Corporation C620 Series Chipset Family SPI Controller (rev 04)
02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
16:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
16:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
16:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
16:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
16:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
16:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
16:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
16:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
16:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
16:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
16:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
64:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port C (rev 04)
64:03.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port D (rev 04)
64:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
64:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
64:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
64:08.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:09.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0a.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0a.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0a.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0a.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0a.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0a.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 04)
64:0a.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 04)
64:0a.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 04)
64:0b.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 04)
64:0b.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 04)
64:0b.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 04)
64:0b.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 04)
64:0c.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0c.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0c.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0c.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0c.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
64:0c.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 04)
64:0c.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 04)
64:0c.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 04)
64:0d.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 04)
64:0d.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 04)
64:0d.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 04)
64:0d.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 04)
65:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
66:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
66:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
66:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
66:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
b2:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 04)
b2:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
b2:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
b2:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
b2:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
b2:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
b2:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
b2:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
b2:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
b2:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
b2:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
b3:00.0 PCI bridge: Intel Corporation Device 37c0 (rev 04)
b4:00.0 PCI bridge: Intel Corporation Device 37c2 (rev 04)
b4:02.0 PCI bridge: Intel Corporation Device 37c4 (rev 04)
b4:03.0 PCI bridge: Intel Corporation Device 37c5 (rev 04)
b5:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
b6:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
b7:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04)
b7:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04)
b7:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04)
b7:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04)
[03:38 kepler ~]# lspci -tv
-+-[0000:b2]-+-00.0-[b3-b8]----00.0-[b4-b8]--+-00.0-[b5]----00.0  Intel Corporation C62x Chipset QuickAssist Technology
 |           |                               +-02.0-[b6]----00.0  Intel Corporation C62x Chipset QuickAssist Technology
 |           |                               \-03.0-[b7-b8]--+-00.0  Intel Corporation Ethernet Connection X722 for 10GBASE-T
 |           |                                               +-00.1  Intel Corporation Ethernet Connection X722 for 10GBASE-T
 |           |                                               +-00.2  Intel Corporation Ethernet Connection X722 for 10GbE SFP+
 |           |                                               \-00.3  Intel Corporation Ethernet Connection X722 for 10GbE SFP+
 |           +-05.0  Intel Corporation Sky Lake-E VT-d
 |           +-05.2  Intel Corporation Sky Lake-E RAS Configuration Registers
 |           +-05.4  Intel Corporation Sky Lake-E IOxAPIC Configuration Registers
 |           +-12.0  Intel Corporation Sky Lake-E M3KTI Registers
 |           +-12.1  Intel Corporation Sky Lake-E M3KTI Registers
 |           +-12.2  Intel Corporation Sky Lake-E M3KTI Registers
 |           +-15.0  Intel Corporation Sky Lake-E M2PCI Registers
 |           +-16.0  Intel Corporation Sky Lake-E M2PCI Registers
 |           +-16.4  Intel Corporation Sky Lake-E M2PCI Registers
 |           \-17.0  Intel Corporation Sky Lake-E M2PCI Registers
 +-[0000:64]-+-02.0-[65]----00.0  Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
 |           +-03.0-[66-67]--+-00.0  Intel Corporation I350 Gigabit Network Connection
 |           |               +-00.1  Intel Corporation I350 Gigabit Network Connection
 |           |               +-00.2  Intel Corporation I350 Gigabit Network Connection
 |           |               \-00.3  Intel Corporation I350 Gigabit Network Connection
 |           +-05.0  Intel Corporation Sky Lake-E VT-d
 |           +-05.2  Intel Corporation Sky Lake-E RAS Configuration Registers
 |           +-05.4  Intel Corporation Sky Lake-E IOxAPIC Configuration Registers
 |           +-08.0  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-09.0  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.0  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.1  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.2  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.3  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.4  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0a.5  Intel Corporation Sky Lake-E LM Channel 1
 |           +-0a.6  Intel Corporation Sky Lake-E LMS Channel 1
 |           +-0a.7  Intel Corporation Sky Lake-E LMDP Channel 1
 |           +-0b.0  Intel Corporation Sky Lake-E DECS Channel 2
 |           +-0b.1  Intel Corporation Sky Lake-E LM Channel 2
 |           +-0b.2  Intel Corporation Sky Lake-E LMS Channel 2
 |           +-0b.3  Intel Corporation Sky Lake-E LMDP Channel 2
 |           +-0c.0  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0c.1  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0c.2  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0c.3  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0c.4  Intel Corporation Sky Lake-E Integrated Memory Controller
 |           +-0c.5  Intel Corporation Sky Lake-E LM Channel 1
 |           +-0c.6  Intel Corporation Sky Lake-E LMS Channel 1
 |           +-0c.7  Intel Corporation Sky Lake-E LMDP Channel 1
 |           +-0d.0  Intel Corporation Sky Lake-E DECS Channel 2
 |           +-0d.1  Intel Corporation Sky Lake-E LM Channel 2
 |           +-0d.2  Intel Corporation Sky Lake-E LMS Channel 2
 |           \-0d.3  Intel Corporation Sky Lake-E LMDP Channel 2
 +-[0000:16]-+-05.0  Intel Corporation Sky Lake-E VT-d
 |           +-05.2  Intel Corporation Sky Lake-E RAS Configuration Registers
 |           +-05.4  Intel Corporation Sky Lake-E IOxAPIC Configuration Registers
 |           +-08.0  Intel Corporation Sky Lake-E CHA Registers
 |           +-08.1  Intel Corporation Sky Lake-E CHA Registers
 |           +-08.2  Intel Corporation Sky Lake-E CHA Registers
 |           +-08.3  Intel Corporation Sky Lake-E CHA Registers
 |           +-08.4  Intel Corporation Sky Lake-E CHA Registers
 |           +-08.5  Intel Corporation Sky Lake-E CHA Registers
 |           +-08.6  Intel Corporation Sky Lake-E CHA Registers
 |           +-08.7  Intel Corporation Sky Lake-E CHA Registers
 |           +-09.0  Intel Corporation Sky Lake-E CHA Registers
 |           +-09.1  Intel Corporation Sky Lake-E CHA Registers
 |           +-09.2  Intel Corporation Sky Lake-E CHA Registers
 |           +-09.3  Intel Corporation Sky Lake-E CHA Registers
 |           +-09.4  Intel Corporation Sky Lake-E CHA Registers
 |           +-09.5  Intel Corporation Sky Lake-E CHA Registers
 |           +-09.6  Intel Corporation Sky Lake-E CHA Registers
 |           +-09.7  Intel Corporation Sky Lake-E CHA Registers
 |           +-0a.0  Intel Corporation Sky Lake-E CHA Registers
 |           +-0a.1  Intel Corporation Sky Lake-E CHA Registers
 |           +-0e.0  Intel Corporation Sky Lake-E CHA Registers
 |           +-0e.1  Intel Corporation Sky Lake-E CHA Registers
 |           +-0e.2  Intel Corporation Sky Lake-E CHA Registers
 |           +-0e.3  Intel Corporation Sky Lake-E CHA Registers
 |           +-0e.4  Intel Corporation Sky Lake-E CHA Registers
 |           +-0e.5  Intel Corporation Sky Lake-E CHA Registers
 |           +-0e.6  Intel Corporation Sky Lake-E CHA Registers
 |           +-0e.7  Intel Corporation Sky Lake-E CHA Registers
 |           +-0f.0  Intel Corporation Sky Lake-E CHA Registers
 |           +-0f.1  Intel Corporation Sky Lake-E CHA Registers
 |           +-0f.2  Intel Corporation Sky Lake-E CHA Registers
 |           +-0f.3  Intel Corporation Sky Lake-E CHA Registers
 |           +-0f.4  Intel Corporation Sky Lake-E CHA Registers
 |           +-0f.5  Intel Corporation Sky Lake-E CHA Registers
 |           +-0f.6  Intel Corporation Sky Lake-E CHA Registers
 |           +-0f.7  Intel Corporation Sky Lake-E CHA Registers
 |           +-10.0  Intel Corporation Sky Lake-E CHA Registers
 |           +-10.1  Intel Corporation Sky Lake-E CHA Registers
 |           +-1d.0  Intel Corporation Sky Lake-E CHA Registers
 |           +-1d.1  Intel Corporation Sky Lake-E CHA Registers
 |           +-1d.2  Intel Corporation Sky Lake-E CHA Registers
 |           +-1d.3  Intel Corporation Sky Lake-E CHA Registers
 |           +-1e.0  Intel Corporation Sky Lake-E PCU Registers
 |           +-1e.1  Intel Corporation Sky Lake-E PCU Registers
 |           +-1e.2  Intel Corporation Sky Lake-E PCU Registers
 |           +-1e.3  Intel Corporation Sky Lake-E PCU Registers
 |           +-1e.4  Intel Corporation Sky Lake-E PCU Registers
 |           +-1e.5  Intel Corporation Sky Lake-E PCU Registers
 |           \-1e.6  Intel Corporation Sky Lake-E PCU Registers
 \-[0000:00]-+-00.0  Intel Corporation Sky Lake-E DMI3 Registers
             +-04.0  Intel Corporation Sky Lake-E CBDMA Registers
             +-04.1  Intel Corporation Sky Lake-E CBDMA Registers
             +-04.2  Intel Corporation Sky Lake-E CBDMA Registers
             +-04.3  Intel Corporation Sky Lake-E CBDMA Registers
             +-04.4  Intel Corporation Sky Lake-E CBDMA Registers
             +-04.5  Intel Corporation Sky Lake-E CBDMA Registers
             +-04.6  Intel Corporation Sky Lake-E CBDMA Registers
             +-04.7  Intel Corporation Sky Lake-E CBDMA Registers
             +-05.0  Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers
             +-05.2  Intel Corporation Sky Lake-E RAS
             +-05.4  Intel Corporation Sky Lake-E IOAPIC
             +-08.0  Intel Corporation Sky Lake-E Ubox Registers
             +-08.1  Intel Corporation Sky Lake-E Ubox Registers
             +-08.2  Intel Corporation Sky Lake-E Ubox Registers
             +-11.0  Intel Corporation C620 Series Chipset Family MROM 0
             +-11.1  Intel Corporation C620 Series Chipset Family MROM 1
             +-11.5  Intel Corporation C620 Series Chipset Family SSATA Controller [AHCI mode]
             +-14.0  Intel Corporation C620 Series Chipset Family USB 3.0 xHCI Controller
             +-14.2  Intel Corporation C620 Series Chipset Family Thermal Subsystem
             +-16.0  Intel Corporation C620 Series Chipset Family MEI Controller #1
             +-16.1  Intel Corporation C620 Series Chipset Family MEI Controller #2
             +-16.4  Intel Corporation C620 Series Chipset Family MEI Controller #3
             +-17.0  Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode]
             +-1c.0-[01]--
             +-1c.4-[02-03]----00.0-[03]----00.0  ASPEED Technology, Inc. ASPEED Graphics Family
             +-1d.0-[04]--
             +-1f.0  Intel Corporation Device a1c8
             +-1f.2  Intel Corporation C620 Series Chipset Family Power Management Controller
             +-1f.4  Intel Corporation C620 Series Chipset Family SMBus
             \-1f.5  Intel Corporation C620 Series Chipset Family SPI Controller
[03:38 kepler ~]#

As you can see, lspci still show those devices, even though they should have been "hidden" thanks to xen-pciback.hide. As further proof, these devices appear in command xl pci-assignable-list. I've also checked the content of /boot/grub/grub.cfg and everything seems fine.

XCP-ng Center and XOCE both show the NICs, even properly identifying their status if a cable is plugged in or not. The VM I've assigned these devices (a fresh install of OPNsense) also see these interfaces, as if the PCI Passthrough was successful. But I can't ping or open the webUI even after ensuring the proper interface assignment is good. So while FreeBSD can see them, they aren't working.

I've been able to pass regular virtual interfaces, or even been able to make SR-IOV work with this VM. But PCI Passthrough doesn't seem to work as expected.

Any ideas? The behavior seems really weird to me, that both host and guest see the devices.

Daniel-Furcht commented 4 years ago

I use PCI Passthrough on XCP-NG 81 whitout problems. lspci shows the the pci raid device, but disks on this device are only visible to assigned VM. (FreeBSD) In case of network device, i think you need a physical cable as virtual switch on host cannot connect to the interface.

jeremfg commented 4 years ago

In case of network device, i think you need a physical cable as virtual switch on host cannot connect to the interface.

What do you mean? I've successfully passed NICs before in Xen Server 6.5. I expect the host to NOT see the interfaces I'm trying to pass to a VM. I'm not trying to do anything "virtual" here.

blackkanto commented 4 years ago

In my experience using PCI passthrough with XenServer/XCP-ng, the devices will still show on the host. However, instead of having the normal kernel driver that you'd expect i.e. "igb" for the Intel i350, it shows "pciback".

Running dmesg | grep pciback should show all the BDFs you added to xen-pciback.hide=

Running lspci -s 66:00.0 -v should show Kernel driver in use: pciback

More in depth info here: https://wiki.xenproject.org/wiki/Xen_PCI_Passthrough

Have you tested another OS to confirm using PCI Passthrough is working successfully with the i350?

stormi commented 3 years ago

Closing as the initial question received an answer.