search

coreos / bugs

Issue tracker for CoreOS Container Linux
https://coreos.com/os/eol/
147 stars 30 forks source link

Network interfaces renamed after upgrade to 2079.3.0 #2578

Closed gigatexal closed 5 years ago

gigatexal commented 5 years ago

Issue Report

Bug

Container Linux Version

2079.3.0

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
...
BUG_REPORT_URL="https://issues.coreos.com"

Environment

What hardware/cloud provider/hypervisor is being used to run Container Linux?

Bare Metal K8s with Cillium for networking and Rook for storage

Expected Behavior

No changes per the release notes found here https://coreos.com/releases/#2079.3.0

Actual Behavior

network interface ids changed and broke things

Reproduction Steps

  1. Run version 2023.5.0
  2. Upgrade to 2079.3.0
  3. See issues.

Other Information

https://github.com/coreos/fedora-coreos-tracker/issues/173

bgilbert commented 5 years ago

Those release notes are just saying that there were no changes between 2079.2.0 and 2079.3.0. We'll use clearer wording in the future. For the full set of changes between 2023.5.0 and 2079.3.0, see the releases page. In this case, the systemd 241 update was probably the culprit.

In order to fix the regression, we'll need more information from you. What were the interface names before and after the upgrade? Could you post the output of lspci -v and udevadm info -e? Could you post any files in /etc/systemd/network/*?

I'll retitle this bug for clarity.

gigatexal commented 5 years ago

We’ll try to get you that output. Thanks for clarifying too. I believe we already downgraded to the previous version so I’m not sure we have a node with the version with changes. I’ll see what I can do.

trevex commented 5 years ago

This is the requested output for a compute node, where the renaming of the interfaces atleast results in a working node. For more complex nodes like network nodes, the BGP configuration, sysctl and iptables settings completely fall apart. All nodes have Mellanox NICs and in the case of the compute nodes the name was always enp59s0f0 and enp59s0f1 and now they are enp1s0f*. This is worse for other nodes with four addtional NICs.

/etc/systemd/network/09-blacklist.network

[Match]
Name=veth* tun* lxc* cilium* kube-ipvs* kube-bridge kube-dummy-if
[Link]
Unmanaged=yes

/etc/systemd/network/10-enp59s0f0.network

[Match]
Name=enp59s0f0
[Network]
Address=10.7.0.58/30
DNS=8.8.8.8
[Route]
Gateway=10.7.0.57

/etc/systemd/network/10-enp59s0f1.network

[Match]
Name=enp59s0f1
[Network]
Address=10.7.1.58/30
DNS=8.8.8.8
[Route]
Gateway=10.7.1.57

/etc/systemd/network/10-lo.network

[Match]
Name=lo
[Network]
Address=10.7.32.41/32
[Route]

lspci -v

00:00.0 Host bridge: Intel Corporation Device 2020 (rev 04)
    Subsystem: Super Micro Computer Inc Device 0962
    Flags: fast devsel
    Capabilities: [90] Express Root Port (Slot-), MSI 00
    Capabilities: [e0] Power Management version 3
    Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
    Capabilities: [144] Vendor Specific Information: ID=0004 Rev=1 Len=03c <?>
    Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
    Capabilities: [250] #19
    Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
    Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
    Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>

00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 35
    Memory at 383ffff2c000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 37
    Memory at 383ffff28000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 35
    Memory at 383ffff24000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 37
    Memory at 383ffff20000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 35
    Memory at 383ffff1c000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 37
    Memory at 383ffff18000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 35
    Memory at 383ffff14000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 37
    Memory at 383ffff10000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

00:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

00:05.2 System peripheral: Intel Corporation Device 2025 (rev 04)
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

00:05.4 PIC: Intel Corporation Device 2026 (rev 04) (prog-if 20 [IO(X)-APIC])
    Subsystem: Intel Corporation Device 2026
    Flags: bus master, fast devsel, latency 0
    Memory at 9d20a000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3

00:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E Ubox Registers
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

00:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E Ubox Registers
    Flags: fast devsel

00:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E Ubox Registers
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

00:11.0 Unassigned class [ff00]: Intel Corporation Device a1ec (rev 09)
    Subsystem: Super Micro Computer Inc Device 0962
    Flags: bus master, fast devsel, latency 0
    Capabilities: [80] Power Management version 3

00:11.1 Unassigned class [ff00]: Intel Corporation Device a1ed (rev 09)
    Subsystem: Super Micro Computer Inc Device 0962
    Flags: bus master, fast devsel, latency 0
    Capabilities: [80] Power Management version 3

00:11.5 SATA controller: Intel Corporation Lewisburg SSATA Controller [AHCI mode] (rev 09) (prog-if 01 [AHCI 1.0])
    Subsystem: Super Micro Computer Inc Lewisburg SSATA Controller [AHCI mode]
    Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 69
    Memory at 9d206000 (32-bit, non-prefetchable) [size=8K]
    Memory at 9d209000 (32-bit, non-prefetchable) [size=256]
    I/O ports at 0800 [size=8]
    I/O ports at 0810 [size=4]
    I/O ports at 0820 [size=32]
    Memory at 9d180000 (32-bit, non-prefetchable) [size=512K]
    Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [70] Power Management version 3
    Capabilities: [a8] SATA HBA v1.0
    Capabilities: [b0] PCI Advanced Features
    Kernel driver in use: ahci
    Kernel modules: ahci

00:14.0 USB controller: Intel Corporation Lewisburg USB 3.0 xHCI Controller (rev 09) (prog-if 30 [XHCI])
    Subsystem: Super Micro Computer Inc Lewisburg USB 3.0 xHCI Controller
    Flags: bus master, medium devsel, latency 0, IRQ 68
    Memory at 383ffff00000 (64-bit, non-prefetchable) [size=64K]
    Capabilities: [70] Power Management version 2
    Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
    Kernel driver in use: xhci_hcd
    Kernel modules: xhci_pci

00:14.2 Signal processing controller: Intel Corporation Device a1b1 (rev 09)
    Subsystem: Super Micro Computer Inc Device 0962
    Flags: bus master, fast devsel, latency 0, IRQ 11
    Memory at 383ffff33000 (64-bit, non-prefetchable) [size=4K]
    Capabilities: [50] Power Management version 3
    Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-

00:16.0 Communication controller: Intel Corporation Lewisburg CSME: HECI #1 (rev 09)
    Subsystem: Super Micro Computer Inc Lewisburg CSME: HECI
    Flags: bus master, fast devsel, latency 0, IRQ 218
    Memory at 383ffff32000 (64-bit, non-prefetchable) [size=4K]
    Capabilities: [50] Power Management version 3
    Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Kernel driver in use: mei_me
    Kernel modules: mei_me

00:16.1 Communication controller: Intel Corporation Lewisburg CSME: HECI #2 (rev 09)
    Subsystem: Super Micro Computer Inc Lewisburg CSME: HECI
    Flags: bus master, fast devsel, latency 0, IRQ 11
    Memory at 383ffff31000 (64-bit, non-prefetchable) [size=4K]
    Capabilities: [50] Power Management version 3
    Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+

00:16.4 Communication controller: Intel Corporation Lewisburg CSME: HECI #3 (rev 09)
    Subsystem: Super Micro Computer Inc Lewisburg CSME: HECI
    Flags: bus master, fast devsel, latency 0, IRQ 11
    Memory at 383ffff30000 (64-bit, non-prefetchable) [size=4K]
    Capabilities: [50] Power Management version 3
    Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+

00:17.0 SATA controller: Intel Corporation Lewisburg SATA Controller [AHCI mode] (rev 09) (prog-if 01 [AHCI 1.0])
    Subsystem: Super Micro Computer Inc Lewisburg SATA Controller [AHCI mode]
    Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 128
    Memory at 9d204000 (32-bit, non-prefetchable) [size=8K]
    Memory at 9d208000 (32-bit, non-prefetchable) [size=256]
    I/O ports at 0840 [size=8]
    I/O ports at 0850 [size=4]
    I/O ports at 0860 [size=32]
    Memory at 9d100000 (32-bit, non-prefetchable) [size=512K]
    Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [70] Power Management version 3
    Capabilities: [a8] SATA HBA v1.0
    Capabilities: [b0] PCI Advanced Features
    Kernel driver in use: ahci
    Kernel modules: ahci

00:1c.0 PCI bridge: Intel Corporation Lewisburg PCI Express Root Port #1 (rev f9) (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0, IRQ 24
    Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
    I/O behind bridge: 00001000-00001fff
    Memory behind bridge: 90000000-901fffff
    Prefetchable memory behind bridge: 0000380000000000-00003800001fffff
    Capabilities: [40] Express Root Port (Slot+), MSI 00
    Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [90] Subsystem: Super Micro Computer Inc Lewisburg PCI Express Root Port
    Capabilities: [a0] Power Management version 3
    Kernel driver in use: pcieport

00:1c.5 PCI bridge: Intel Corporation Lewisburg PCI Express Root Port #6 (rev f9) (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0, IRQ 25
    Bus: primary=00, secondary=02, subordinate=03, sec-latency=0
    I/O behind bridge: 00003000-00003fff
    Memory behind bridge: 9c000000-9d0fffff
    Capabilities: [40] Express Root Port (Slot+), MSI 00
    Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [90] Subsystem: Super Micro Computer Inc Lewisburg PCI Express Root Port
    Capabilities: [a0] Power Management version 3
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Access Control Services
    Capabilities: [220] #19
    Kernel driver in use: pcieport

00:1f.0 ISA bridge: Intel Corporation Lewisburg LPC Controller (rev 09)
    Subsystem: Super Micro Computer Inc Lewisburg LPC Controller
    Flags: bus master, fast devsel, latency 0

00:1f.2 Memory controller: Intel Corporation Lewisburg PMC (rev 09)
    Subsystem: Super Micro Computer Inc Lewisburg PMC
    Flags: bus master, fast devsel, latency 0
    Memory at 9d200000 (32-bit, non-prefetchable) [size=16K]

00:1f.4 SMBus: Intel Corporation Lewisburg SMBus (rev 09)
    Subsystem: Super Micro Computer Inc Lewisburg SMBus
    Flags: medium devsel, IRQ 16
    Memory at 380000200000 (64-bit, non-prefetchable) [size=256]
    I/O ports at 0780 [size=32]
    Kernel driver in use: i801_smbus
    Kernel modules: i2c_i801

00:1f.5 Serial bus controller [0c80]: Intel Corporation Lewisburg SPI Controller (rev 09)
    Subsystem: Super Micro Computer Inc Lewisburg SPI Controller
    Flags: fast devsel
    Memory at fe010000 (32-bit, non-prefetchable) [size=4K]

02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04) (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0, IRQ 10
    Bus: primary=02, secondary=03, subordinate=03, sec-latency=32
    I/O behind bridge: 00003000-00003fff
    Memory behind bridge: 9c000000-9d0fffff
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [78] Power Management version 3
    Capabilities: [80] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
    Capabilities: [c0] Subsystem: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge
    Capabilities: [100] Virtual Channel
    Capabilities: [800] Advanced Error Reporting

03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41) (prog-if 00 [VGA controller])
    DeviceName: ASPEED Video AST2500
    Subsystem: Super Micro Computer Inc ASPEED Graphics Family
    Flags: bus master, medium devsel, latency 0, IRQ 10
    Memory at 9c000000 (32-bit, non-prefetchable) [size=16M]
    Memory at 9d000000 (32-bit, non-prefetchable) [size=128K]
    I/O ports at 3000 [size=128]
    [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+

17:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
    Physical Slot: 6
    Flags: bus master, fast devsel, latency 0, IRQ 27
    Bus: primary=17, secondary=18, subordinate=18, sec-latency=0
    I/O behind bridge: 00005000-00005fff
    Memory behind bridge: aae00000-aaefffff
    Prefetchable memory behind bridge: 0000384000000000-00003840000fffff
    Capabilities: [40] Subsystem: Super Micro Computer Inc Sky Lake-E PCI Express Root Port 1A
    Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
    Capabilities: [90] Express Root Port (Slot+), MSI 00
    Capabilities: [e0] Power Management version 3
    Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
    Capabilities: [110] Access Control Services
    Capabilities: [148] Advanced Error Reporting
    Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
    Capabilities: [250] #19
    Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
    Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
    Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
    Kernel driver in use: pcieport

17:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

17:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

17:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
    Subsystem: Intel Corporation Device 2036
    Flags: bus master, fast devsel, latency 0
    Memory at aaf00000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3

17:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

17:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

17:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

17:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

17:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

17:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

17:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

17:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

18:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    Subsystem: Super Micro Computer Inc I350 Gigabit Network Connection
    Flags: bus master, fast devsel, latency 0, IRQ 57
    Memory at aae20000 (32-bit, non-prefetchable) [size=128K]
    I/O ports at 5020 [size=32]
    Memory at aae44000 (32-bit, non-prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
    Capabilities: [a0] Express Endpoint, MSI 00
    Capabilities: [e0] Vital Product Data
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Device Serial Number ac-1f-6b-ff-ff-b1-73-ce
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1a0] Transaction Processing Hints
    Capabilities: [1c0] Latency Tolerance Reporting
    Capabilities: [1d0] Access Control Services
    Kernel driver in use: igb
    Kernel modules: igb

18:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    Subsystem: Super Micro Computer Inc I350 Gigabit Network Connection
    Flags: bus master, fast devsel, latency 0, IRQ 70
    Memory at aae00000 (32-bit, non-prefetchable) [size=128K]
    I/O ports at 5000 [size=32]
    Memory at aae40000 (32-bit, non-prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
    Capabilities: [a0] Express Endpoint, MSI 00
    Capabilities: [e0] Vital Product Data
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Device Serial Number ac-1f-6b-ff-ff-b1-73-ce
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1a0] Transaction Processing Hints
    Capabilities: [1d0] Access Control Services
    Kernel driver in use: igb
    Kernel modules: igb

3a:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
    Physical Slot: 1
    Flags: bus master, fast devsel, latency 0, IRQ 29
    Bus: primary=3a, secondary=3b, subordinate=3b, sec-latency=0
    Memory behind bridge: b8500000-b86fffff
    Prefetchable memory behind bridge: 000038bffc000000-000038bfffffffff
    Capabilities: [40] Subsystem: Super Micro Computer Inc Sky Lake-E PCI Express Root Port 1A
    Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
    Capabilities: [90] Express Root Port (Slot+), MSI 00
    Capabilities: [e0] Power Management version 3
    Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
    Capabilities: [110] Access Control Services
    Capabilities: [148] Advanced Error Reporting
    Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
    Capabilities: [250] #19
    Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
    Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
    Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
    Kernel driver in use: pcieport

3a:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
    Subsystem: Intel Corporation Device 2036
    Flags: bus master, fast devsel, latency 0
    Memory at b8700000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3

3a:08.0 System peripheral: Intel Corporation Device 2066 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

3a:09.0 System peripheral: Intel Corporation Device 2066 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

3a:0a.0 System peripheral: Intel Corporation Device 2040 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0a.1 System peripheral: Intel Corporation Device 2041 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0a.2 System peripheral: Intel Corporation Device 2042 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

3a:0a.3 System peripheral: Intel Corporation Device 2043 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0a.4 System peripheral: Intel Corporation Device 2044 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0a.5 System peripheral: Intel Corporation Device 2045 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0a.6 System peripheral: Intel Corporation Device 2046 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

3a:0a.7 System peripheral: Intel Corporation Device 2047 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0b.0 System peripheral: Intel Corporation Device 2048 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0b.1 System peripheral: Intel Corporation Device 2049 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0b.2 System peripheral: Intel Corporation Device 204a (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

3a:0b.3 System peripheral: Intel Corporation Device 204b (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0c.0 System peripheral: Intel Corporation Device 2040 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0c.1 System peripheral: Intel Corporation Device 2041 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0c.2 System peripheral: Intel Corporation Device 2042 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

3a:0c.3 System peripheral: Intel Corporation Device 2043 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0c.4 System peripheral: Intel Corporation Device 2044 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0c.5 System peripheral: Intel Corporation Device 2045 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0c.6 System peripheral: Intel Corporation Device 2046 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

3a:0c.7 System peripheral: Intel Corporation Device 2047 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0d.0 System peripheral: Intel Corporation Device 2048 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0d.1 System peripheral: Intel Corporation Device 2049 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3a:0d.2 System peripheral: Intel Corporation Device 204a (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

3a:0d.3 System peripheral: Intel Corporation Device 204b (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

3b:00.0 Ethernet controller: Mellanox Technologies MT27640 Family
    Subsystem: Mellanox Technologies MT27640 Family
    Flags: bus master, fast devsel, latency 0, IRQ 67
    Memory at 38bffe000000 (64-bit, prefetchable) [size=32M]
    Expansion ROM at b8600000 [disabled] [size=1M]
    Capabilities: [60] Express Endpoint, MSI 00
    Capabilities: [48] Vital Product Data
    Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
    Capabilities: [c0] Vendor Specific Information: Len=18 <?>
    Capabilities: [40] Power Management version 3
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1c0] #19
    Capabilities: [230] Access Control Services
    Kernel driver in use: mlx5_core
    Kernel modules: mlx5_core

3b:00.1 Ethernet controller: Mellanox Technologies MT27640 Family
    Subsystem: Mellanox Technologies MT27640 Family
    Flags: bus master, fast devsel, latency 0, IRQ 173
    Memory at 38bffc000000 (64-bit, prefetchable) [size=32M]
    Expansion ROM at b8500000 [disabled] [size=1M]
    Capabilities: [60] Express Endpoint, MSI 00
    Capabilities: [48] Vital Product Data
    Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
    Capabilities: [c0] Vendor Specific Information: Len=18 <?>
    Capabilities: [40] Power Management version 3
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [230] Access Control Services
    Kernel driver in use: mlx5_core
    Kernel modules: mlx5_core

5d:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

5d:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

5d:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
    Subsystem: Intel Corporation Device 2036
    Flags: bus master, fast devsel, latency 0
    Memory at c5f00000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3

5d:0e.0 Performance counters: Intel Corporation Device 2058 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

5d:0e.1 System peripheral: Intel Corporation Device 2059 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [100] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

5d:0f.0 Performance counters: Intel Corporation Device 2058 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

5d:0f.1 System peripheral: Intel Corporation Device 2059 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [100] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

5d:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

5d:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers
    Flags: fast devsel
    Kernel driver in use: skx_uncore

5d:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers
    Flags: fast devsel
    Kernel driver in use: skx_uncore

5d:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M2PCI Registers
    Flags: fast devsel

5d:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M2PCI Registers
    Flags: fast devsel

5d:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M2PCI Registers
    Flags: fast devsel

80:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 45
    Memory at 393ffff1c000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

80:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 47
    Memory at 393ffff18000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

80:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 45
    Memory at 393ffff14000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

80:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 47
    Memory at 393ffff10000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

80:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 45
    Memory at 393ffff0c000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

80:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 47
    Memory at 393ffff08000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

80:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 45
    Memory at 393ffff04000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

80:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
    Flags: bus master, fast devsel, latency 0, IRQ 47
    Memory at 393ffff00000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
    Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3
    Kernel driver in use: ioatdma

80:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

80:05.2 System peripheral: Intel Corporation Device 2025 (rev 04)
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

80:05.4 PIC: Intel Corporation Device 2026 (rev 04) (prog-if 20 [IO(X)-APIC])
    Subsystem: Intel Corporation Device 2026
    Flags: bus master, fast devsel, latency 0
    Memory at d3700000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3

80:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E Ubox Registers
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

80:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E Ubox Registers
    Flags: fast devsel

80:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E Ubox Registers
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

85:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

85:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

85:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
    Subsystem: Intel Corporation Device 2036
    Flags: bus master, fast devsel, latency 0
    Memory at e0f00000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3

85:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E CHA Registers
    Flags: fast devsel

85:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

85:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

85:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

85:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

85:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

85:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

85:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E PCU Registers
    Flags: fast devsel

ae:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
    Subsystem: Intel Corporation Device 2036
    Flags: bus master, fast devsel, latency 0
    Memory at ee700000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3

ae:08.0 System peripheral: Intel Corporation Device 2066 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

ae:09.0 System peripheral: Intel Corporation Device 2066 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

ae:0a.0 System peripheral: Intel Corporation Device 2040 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0a.1 System peripheral: Intel Corporation Device 2041 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0a.2 System peripheral: Intel Corporation Device 2042 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

ae:0a.3 System peripheral: Intel Corporation Device 2043 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0a.4 System peripheral: Intel Corporation Device 2044 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0a.5 System peripheral: Intel Corporation Device 2045 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0a.6 System peripheral: Intel Corporation Device 2046 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

ae:0a.7 System peripheral: Intel Corporation Device 2047 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0b.0 System peripheral: Intel Corporation Device 2048 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0b.1 System peripheral: Intel Corporation Device 2049 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0b.2 System peripheral: Intel Corporation Device 204a (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

ae:0b.3 System peripheral: Intel Corporation Device 204b (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0c.0 System peripheral: Intel Corporation Device 2040 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0c.1 System peripheral: Intel Corporation Device 2041 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0c.2 System peripheral: Intel Corporation Device 2042 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

ae:0c.3 System peripheral: Intel Corporation Device 2043 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0c.4 System peripheral: Intel Corporation Device 2044 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0c.5 System peripheral: Intel Corporation Device 2045 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0c.6 System peripheral: Intel Corporation Device 2046 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

ae:0c.7 System peripheral: Intel Corporation Device 2047 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0d.0 System peripheral: Intel Corporation Device 2048 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0d.1 System peripheral: Intel Corporation Device 2049 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

ae:0d.2 System peripheral: Intel Corporation Device 204a (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

ae:0d.3 System peripheral: Intel Corporation Device 204b (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

d7:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
    Physical Slot: 14
    Flags: bus master, fast devsel, latency 0, IRQ 31
    Bus: primary=d7, secondary=d8, subordinate=d8, sec-latency=0
    Memory behind bridge: fbe00000-fbefffff
    Prefetchable memory behind bridge: 000039c000000000-000039c0001fffff
    Capabilities: [40] Subsystem: Super Micro Computer Inc Sky Lake-E PCI Express Root Port 1A
    Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
    Capabilities: [90] Express Root Port (Slot+), MSI 00
    Capabilities: [e0] Power Management version 3
    Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
    Capabilities: [110] Access Control Services
    Capabilities: [148] Advanced Error Reporting
    Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
    Capabilities: [250] #19
    Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
    Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
    Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
    Kernel driver in use: pcieport

d7:01.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1B (rev 04) (prog-if 00 [Normal decode])
    Physical Slot: 15
    Flags: bus master, fast devsel, latency 0, IRQ 32
    Bus: primary=d7, secondary=d9, subordinate=d9, sec-latency=0
    Memory behind bridge: ee800000-ee9fffff
    Prefetchable memory behind bridge: 000039c000200000-000039c0003fffff
    Capabilities: [40] Subsystem: Super Micro Computer Inc Sky Lake-E PCI Express Root Port 1B
    Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
    Capabilities: [90] Express Root Port (Slot+), MSI 00
    Capabilities: [e0] Power Management version 3
    Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
    Capabilities: [110] Access Control Services
    Capabilities: [148] Advanced Error Reporting
    Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
    Capabilities: [250] #19
    Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
    Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
    Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
    Kernel driver in use: pcieport

d7:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
    Physical Slot: 16
    Flags: bus master, fast devsel, latency 0, IRQ 33
    Bus: primary=d7, secondary=da, subordinate=da, sec-latency=0
    Memory behind bridge: eea00000-eebfffff
    Prefetchable memory behind bridge: 000039c000400000-000039c0005fffff
    Capabilities: [40] Subsystem: Super Micro Computer Inc Sky Lake-E PCI Express Root Port 1C
    Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
    Capabilities: [90] Express Root Port (Slot+), MSI 00
    Capabilities: [e0] Power Management version 3
    Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
    Capabilities: [110] Access Control Services
    Capabilities: [148] Advanced Error Reporting
    Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
    Capabilities: [250] #19
    Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
    Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
    Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
    Kernel driver in use: pcieport

d7:03.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1D (rev 04) (prog-if 00 [Normal decode])
    Physical Slot: 17
    Flags: bus master, fast devsel, latency 0, IRQ 34
    Bus: primary=d7, secondary=db, subordinate=db, sec-latency=0
    I/O behind bridge: 0000f000-0000ffff
    Memory behind bridge: eec00000-eedfffff
    Prefetchable memory behind bridge: 000039c000600000-000039c0007fffff
    Capabilities: [40] Subsystem: Super Micro Computer Inc Sky Lake-E PCI Express Root Port 1D
    Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
    Capabilities: [90] Express Root Port (Slot+), MSI 00
    Capabilities: [e0] Power Management version 3
    Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
    Capabilities: [110] Access Control Services
    Capabilities: [148] Advanced Error Reporting
    Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
    Capabilities: [250] #19
    Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
    Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
    Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
    Kernel driver in use: pcieport

d7:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

d7:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

d7:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
    Subsystem: Intel Corporation Device 2036
    Flags: bus master, fast devsel, latency 0
    Memory at fbf00000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [e0] Power Management version 3

d7:0e.0 Performance counters: Intel Corporation Device 2058 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

d7:0e.1 System peripheral: Intel Corporation Device 2059 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [100] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

d7:0f.0 Performance counters: Intel Corporation Device 2058 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Kernel driver in use: skx_uncore

d7:0f.1 System peripheral: Intel Corporation Device 2059 (rev 04)
    Subsystem: Intel Corporation Device 0000
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [100] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

d7:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers
    Flags: fast devsel
    Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

d7:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers
    Flags: fast devsel
    Kernel driver in use: skx_uncore

d7:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M3KTI Registers
    Flags: fast devsel
    Kernel driver in use: skx_uncore

d7:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M2PCI Registers
    Flags: fast devsel

d7:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M2PCI Registers
    Flags: fast devsel

d7:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
    Subsystem: Intel Corporation Sky Lake-E M2PCI Registers
    Flags: fast devsel

d8:00.0 Non-Volatile memory controller: Intel Corporation Device 0a54 (prog-if 02 [NVM Express])
    Subsystem: Intel Corporation Device 4802
    Physical Slot: 14-1
    Flags: bus master, fast devsel, latency 0, IRQ 55
    Memory at fbe10000 (64-bit, non-prefetchable) [size=16K]
    Expansion ROM at fbe00000 [disabled] [size=64K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI-X: Enable+ Count=129 Masked-
    Capabilities: [60] Express Endpoint, MSI 00
    Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [150] Virtual Channel
    Capabilities: [180] Power Budgeting <?>
    Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [270] Device Serial Number 55-cd-2e-41-4f-db-1f-46
    Capabilities: [2a0] #19
    Capabilities: [2d0] Latency Tolerance Reporting
    Capabilities: [310] L1 PM Substates
    Kernel driver in use: nvme
    Kernel modules: nvme

udevadm omitted for now output was ~13000 lines.

lucab commented 5 years ago

@gigatexal @trevex thanks for the follow up. udevadm info /sys/class/net/<$IFNAME> should produce a reasonably smaller output. The output of udevadm test /sys/class/net/<$IFNAME> would also help double-checking udev config.

The network card(s) at 3b:00.0 is indeed reasonable to be called enp59s0*, but I don't have specific knowledge about Mellanox card. (A gut feeling is that this may be related to its ARI capabilities)

While we track down what's going on, you may try experimenting and getting back the old behavior via https://www.freedesktop.org/software/systemd/man/systemd-udevd.service.html#net.naming-scheme=.

trevex commented 5 years ago

I haven't gotten around setting net.naming-scheme yet, but in the mean time I was digging through the systemd news and there are several changes from 238 to 241 which are related:

CHANGES WITH 241:
...
        * A new network device NamePolicy "keep" is implemented for link files,
          and used by default in 99-default.link (the fallback configuration
          provided by systemd). With this policy, if the network device name
          was already set by userspace, the device will not be renamed again.
          This matches the naming scheme that was implemented before
          systemd-240. If naming-scheme < 240 is specified, the "keep" policy
          is also enabled by default, even if not specified. Effectively, this
          means that if naming-scheme >= 240 is specified, network devices will
          be renamed according to the configuration, even if they have been
          renamed already, if "keep" is not specified as the naming policy in
          the .link file. The 99-default.link file provided by systemd includes
          "keep" for backwards compatibility, but it is recommended for user
          installed .link files to *not* include it.

          The "kernel" policy, which keeps kernel names declared to be
          "persistent", now works again as documented.
...
CHANGES WITH 240:
...

        * Predictable interface naming has been extended to cover InfiniBand
NICs. They will be exposed with an "ib" prefix.
...
        * systemd-udevd's network naming logic now understands a new
          net.naming-scheme= kernel command line switch, which may be used to
          pick a specific version of the naming scheme. This helps stabilizing
          interface names even as systemd/udev are updated and the naming logic
is improved.
...
        * The naming scheme for network devices was changed to always rename
          devices, even if they were already renamed by userspace. The "kernel"
          policy was changed to only apply as a fallback, if no other naming
policy took effect.
...
CHANGES WITH 239:

        * NETWORK INTERFACE DEVICE NAMING CHANGES: systemd-udevd's "net_id"
          builtin will name network interfaces differently than in previous
          versions for virtual network interfaces created with SR-IOV and NPAR
          and for devices where the PCI network controller device does not have
a slot number associated.
...

Source: https://github.com/systemd/systemd/blob/master/NEWS

This would indeed lead me to believe that systemd is the "culprit".

dm0- commented 5 years ago

The naming scheme has been reverted in the alpha and beta channels, if you want to test it.

trevex commented 5 years ago

We will, thank you. I will come back with more information once we tested it.

bgilbert commented 5 years ago

This should be fixed in stable 2079.5.0, due shortly. Thanks for reporting.