search

edk2-porting / edk2-rk3588

EDK2 UEFI firmware for Rockchip RK3588 platforms
456 stars 90 forks source link

PCIe does not detect switches #21

Closed strongtz closed 1 year ago

strongtz commented 1 year ago

The current PCIe driver used in edk2 is quite problematic. PCIe won't even work properly in mainline Linux booted with device tree.

In the example below, four NVMe drives are behind an ASM2824 PCIe switch, which is connected to pcie3x4 controller. But only one NVMe drive works properly in both UEFI and Linux.

However, if we remove PCIe DXE from edk2, PCIe will work prefectly fine in Linux.

dmesg with PCIe DXE:

 rockchip-dw-pcie fe150000.pcie: host bridge /pcie@fe150000 ranges:
 rockchip-dw-pcie fe150000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
 rockchip-dw-pcie fe150000.pcie:      MEM 0x00f0200000..0x00f0ffffff -> 0x00f0200000
 rockchip-dw-pcie fe150000.pcie:      MEM 0x0900000000..0x093fffffff -> 0x0900000000
 rockchip-dw-pcie fe150000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
 rockchip-dw-pcie fe150000.pcie: PCIe Gen.3 x4 link up
 rockchip-dw-pcie fe150000.pcie: PCI host bridge to bus 0000:00
 pci_bus 0000:00: root bus resource [bus 00-0f]
 pci_bus 0000:00: root bus resource [io  0x100000-0x1fffff] (bus address [0xf0100000-0xf01fffff])
 pci_bus 0000:00: root bus resource [mem 0xf0200000-0xf0ffffff]
 pci_bus 0000:00: root bus resource [mem 0x900000000-0x93fffffff pref]
 pci 0000:00:00.0: [1d87:3588] type 01 class 0x060400
 pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
 pci 0000:00:00.0: reg 0x14: [mem 0x00000000-0x3fffffff]
 pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
 pci 0000:00:00.0: supports D1 D2
 pci 0000:00:00.0: PME# supported from D0 D1 D3hot
 pci_bus 0000:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
 pci 0000:01:00.0: [1b21:2824] type 01 class 0x060400
 pci 0000:01:00.0: enabling Extended Tags
 pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
 pci 0000:01:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:00.0 (capable of 63.008 Gb/s with 8.0 GT/s PCIe x8 link)
 pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
 pci 0000:01:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
 pci 0000:02:00.0: [1b21:2824] type 01 class 0x060400
 pci 0000:02:00.0: enabling Extended Tags
 pci 0000:02:04.0: [1b21:2824] type 01 class 0x060400
 pci 0000:02:04.0: enabling Extended Tags
 pci 0000:02:04.0: PME# supported from D0 D3hot D3cold
 pci 0000:02:08.0: [1b21:2824] type 01 class 0x060400
 pci 0000:02:08.0: enabling Extended Tags
 pci 0000:02:0c.0: [1b21:2824] type 01 class 0xffffff
 pci 0000:02:0c.0: enabling Extended Tags
 pci 0000:02:0c.0: PME# supported from D0 D3hot D3cold
 pci 0000:02:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
 pci 0000:02:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
 pci 0000:02:08.0: bridge configuration invalid ([bus 00-00]), reconfiguring
 pci 0000:02:0c.0: bridge configuration invalid ([bus 00-00]), reconfiguring
 pci 0000:03:00.0: [1e4b:1202] type 00 class 0x010802
 pci 0000:03:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
 pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
 pci 0000:04:00.0: [1e4b:1202] type 00 class 0x010802
 pci 0000:04:00.0: PME# supported from D0 D3hot D3cold
 pci 0000:04:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
 pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
 pci 0000:05:00.0: [1e4b:1202] type 00 class 0x010802
 pci 0000:05:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
 pci 0000:05:00.0: PME# supported from D0 D3hot D3cold
 pci_bus 0000:05: busn_res: [bus 05-ff] end is updated to 05
 pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 05
 pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000]
 pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x40000000]
 pci 0000:00:00.0: BAR 1: no space for [mem size 0x40000000]
 pci 0000:00:00.0: BAR 1: failed to assign [mem size 0x40000000]
 pci 0000:00:00.0: BAR 14: assigned [mem 0xf0200000-0xf03fffff]
 pci 0000:00:00.0: BAR 6: assigned [mem 0xf0400000-0xf040ffff pref]
 pci 0000:01:00.0: BAR 14: assigned [mem 0xf0200000-0xf03fffff]
 pci 0000:02:00.0: BAR 14: assigned [mem 0xf0200000-0xf02fffff]
 pci 0000:02:08.0: BAR 14: assigned [mem 0xf0300000-0xf03fffff]
 pci 0000:03:00.0: BAR 0: assigned [mem 0xf0200000-0xf0203fff 64bit]
 pci 0000:02:00.0: PCI bridge to [bus 03]
 pci 0000:02:00.0:   bridge window [mem 0xf0200000-0xf02fffff]
 pci 0000:02:04.0: PCI bridge to [bus 04]
 pci 0000:05:00.0: BAR 0: assigned [mem 0xf0300000-0xf0303fff 64bit]
 pci 0000:02:08.0: PCI bridge to [bus 05]
 pci 0000:02:08.0:   bridge window [mem 0xf0300000-0xf03fffff]
 pci 0000:01:00.0: PCI bridge to [bus 02-05]
 pci 0000:01:00.0:   bridge window [mem 0xf0200000-0xf03fffff]
 pci 0000:00:00.0: PCI bridge to [bus 01-ff]
 pci 0000:00:00.0:   bridge window [mem 0xf0200000-0xf03fffff]
 pcieport 0000:00:00.0: PME: Signaling with IRQ 87
 pcieport 0000:00:00.0: AER: enabled with IRQ 88
 pcieport 0000:01:00.0: enabling device (0000 -> 0002)
 pcieport 0000:02:00.0: enabling device (0000 -> 0002)
 pcieport 0000:02:00.0: no suspend buffer for LTR; ASPM issues possible after resume
 pcieport 0000:02:04.0: of_irq_parse_pci: failed with rc=134
 pcieport 0000:02:08.0: enabling device (0000 -> 0002)
 pcieport 0000:02:08.0: buffer not found in pci_save_pcie_state
 nvme nvme0: pci function 0000:03:00.0
 nvme nvme0: pci function 0000:05:00.0
 scsi host0: uas
 nvme nvme0: missing or invalid SUBNQN field.
 nvme nvme0: allocated 32 MiB host memory buffer.
 nvme nvme0: 8/0/0 default/read/poll queues
 nvme nvme0: Ignoring bogus Namespace Identifiers
  nvme0n1: p1 p2 p3

[root@sakuramist-nas-3588 ~]# lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0000:01:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01)
0000:02:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01)
0000:02:04.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01)
0000:02:08.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01)
0000:02:0c.0 Unassigned class [ffff]: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev ff)
0000:03:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (rev 01)
0000:04:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (rev 01)
0000:05:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (rev 01)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 04)

dmesg without PCIe DXE:

 rockchip-dw-pcie fe150000.pcie: host bridge /pcie@fe150000 ranges:
 rockchip-dw-pcie fe150000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
 rockchip-dw-pcie fe150000.pcie:      MEM 0x00f0200000..0x00f0ffffff -> 0x00f0200000
 rockchip-dw-pcie fe150000.pcie:      MEM 0x0900000000..0x093fffffff -> 0x0900000000
 rockchip-dw-pcie fe150000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
 rockchip-dw-pcie fe150000.pcie: PCIe Gen.3 x4 link up
 rockchip-dw-pcie fe150000.pcie: PCI host bridge to bus 0000:00
 pci_bus 0000:00: root bus resource [bus 00-0f]
 pci_bus 0000:00: root bus resource [io  0x100000-0x1fffff] (bus address [0xf0100000-0xf01fffff])
 pci_bus 0000:00: root bus resource [mem 0xf0200000-0xf0ffffff]
 pci_bus 0000:00: root bus resource [mem 0x900000000-0x93fffffff pref]
 pci 0000:00:00.0: [1d87:3588] type 01 class 0x060400
 pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
 pci 0000:00:00.0: reg 0x14: [mem 0x00000000-0x3fffffff]
 pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
 pci 0000:00:00.0: supports D1 D2
 pci 0000:00:00.0: PME# supported from D0 D1 D3hot
 pci_bus 0000:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
 pci 0000:01:00.0: [1b21:2824] type 01 class 0x060400
 pci 0000:01:00.0: enabling Extended Tags
 pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
 pci 0000:01:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:00.0 (capable of 63.008 Gb/s with 8.0 GT/s PCIe x8 link)
 pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
 pci 0000:02:00.0: [1b21:2824] type 01 class 0x060400
 pci 0000:02:00.0: enabling Extended Tags
 pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
 pci 0000:02:04.0: [1b21:2824] type 01 class 0x060400
 pci 0000:02:04.0: enabling Extended Tags
 pci 0000:02:04.0: PME# supported from D0 D3hot D3cold
 pci 0000:02:08.0: [1b21:2824] type 01 class 0x060400
 pci 0000:02:08.0: enabling Extended Tags
 pci 0000:02:08.0: PME# supported from D0 D3hot D3cold
 pci 0000:02:0c.0: [1b21:2824] type 01 class 0x060400
 pci 0000:02:0c.0: enabling Extended Tags
 pci 0000:02:0c.0: PME# supported from D0 D3hot D3cold
 pci 0000:02:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
 pci 0000:02:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
 pci 0000:02:08.0: bridge configuration invalid ([bus 00-00]), reconfiguring
 pci 0000:02:0c.0: bridge configuration invalid ([bus 00-00]), reconfiguring
 pci 0000:03:00.0: [1e4b:1202] type 00 class 0x010802
 pci 0000:03:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
 pci 0000:03:00.0: PME# supported from D0 D3hot D3cold
 pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
 pci 0000:04:00.0: [1e4b:1202] type 00 class 0x010802
 pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
 pci 0000:04:00.0: PME# supported from D0 D3hot D3cold
 pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
 pci 0000:05:00.0: [1e4b:1202] type 00 class 0x010802
 pci 0000:05:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
 pci 0000:05:00.0: PME# supported from D0 D3hot D3cold
 pci_bus 0000:05: busn_res: [bus 05-ff] end is updated to 05
 pci 0000:06:00.0: [1e4b:1202] type 00 class 0x010802
 pci 0000:06:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
 pci 0000:06:00.0: PME# supported from D0 D3hot D3cold
 pci_bus 0000:06: busn_res: [bus 06-ff] end is updated to 06
 pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 06
 pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000]
 pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x40000000]
 pci 0000:00:00.0: BAR 1: no space for [mem size 0x40000000]
 pci 0000:00:00.0: BAR 1: failed to assign [mem size 0x40000000]
 pci 0000:00:00.0: BAR 14: assigned [mem 0xf0200000-0xf05fffff]
 pci 0000:00:00.0: BAR 6: assigned [mem 0xf0600000-0xf060ffff pref]
 pci 0000:01:00.0: BAR 14: assigned [mem 0xf0200000-0xf05fffff]
 pci 0000:02:00.0: BAR 14: assigned [mem 0xf0200000-0xf02fffff]
 pci 0000:02:04.0: BAR 14: assigned [mem 0xf0300000-0xf03fffff]
 pci 0000:02:08.0: BAR 14: assigned [mem 0xf0400000-0xf04fffff]
 pci 0000:02:0c.0: BAR 14: assigned [mem 0xf0500000-0xf05fffff]
 pci 0000:03:00.0: BAR 0: assigned [mem 0xf0200000-0xf0203fff 64bit]
 pci 0000:02:00.0: PCI bridge to [bus 03]
 pci 0000:02:00.0:   bridge window [mem 0xf0200000-0xf02fffff]
 pci 0000:04:00.0: BAR 0: assigned [mem 0xf0300000-0xf0303fff 64bit]
 pci 0000:02:04.0: PCI bridge to [bus 04]
 pci 0000:02:04.0:   bridge window [mem 0xf0300000-0xf03fffff]
 pci 0000:05:00.0: BAR 0: assigned [mem 0xf0400000-0xf0403fff 64bit]
 pci 0000:02:08.0: PCI bridge to [bus 05]
 pci 0000:02:08.0:   bridge window [mem 0xf0400000-0xf04fffff]
 pci 0000:06:00.0: BAR 0: assigned [mem 0xf0500000-0xf0503fff 64bit]
 pci 0000:02:0c.0: PCI bridge to [bus 06]
 pci 0000:02:0c.0:   bridge window [mem 0xf0500000-0xf05fffff]
 pci 0000:01:00.0: PCI bridge to [bus 02-06]
 pci 0000:01:00.0:   bridge window [mem 0xf0200000-0xf05fffff]
 pci 0000:00:00.0: PCI bridge to [bus 01-ff]
 pci 0000:00:00.0:   bridge window [mem 0xf0200000-0xf05fffff]
 pcieport 0000:00:00.0: PME: Signaling with IRQ 70
 pcieport 0000:00:00.0: AER: enabled with IRQ 71
 pcieport 0000:01:00.0: enabling device (0000 -> 0002)
 pcieport 0000:02:00.0: enabling device (0000 -> 0002)
 pcieport 0000:02:04.0: enabling device (0000 -> 0002)
 pcieport 0000:02:08.0: enabling device (0000 -> 0002)
 pcieport 0000:02:0c.0: enabling device (0000 -> 0002)
 nvme nvme0: pci function 0000:03:00.0
 nvme 0000:03:00.0: enabling device (0000 -> 0002)
 nvme nvme0: missing or invalid SUBNQN field.
 nvme nvme0: allocated 32 MiB host memory buffer.
 nvme nvme0: 8/0/0 default/read/poll queues
 nvme nvme0: Ignoring bogus Namespace Identifiers
  nvme0n1: p1 p2 p3
 nvme nvme1: pci function 0000:04:00.0
 nvme 0000:04:00.0: enabling device (0000 -> 0002)
 nvme nvme1: missing or invalid SUBNQN field.
 nvme nvme1: allocated 32 MiB host memory buffer.
 nvme nvme1: 8/0/0 default/read/poll queues
 nvme nvme1: Ignoring bogus Namespace Identifiers
  nvme1n1: p1 p2 p3
 nvme nvme2: pci function 0000:05:00.0
 nvme 0000:05:00.0: enabling device (0000 -> 0002)
 nvme nvme2: missing or invalid SUBNQN field.
 nvme nvme2: allocated 32 MiB host memory buffer.
 nvme nvme2: 8/0/0 default/read/poll queues
 nvme nvme2: Ignoring bogus Namespace Identifiers
  nvme2n1: p1 p2 p3
 nvme nvme3: pci function 0000:06:00.0
 nvme 0000:06:00.0: enabling device (0000 -> 0002)
 nvme nvme3: missing or invalid SUBNQN field.
 nvme nvme3: allocated 32 MiB host memory buffer.
 nvme nvme3: 8/0/0 default/read/poll queues
 nvme nvme3: Ignoring bogus Namespace Identifiers
  nvme3n1: p1 p2 p3

[root@sakuramist-nas-3588 ~]# lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0000:01:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01)
0000:02:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01)
0000:02:04.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01)
0000:02:08.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01)
0000:02:0c.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01)
0000:03:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (rev 01)
0000:04:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (rev 01)
0000:05:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (rev 01)
0000:06:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (rev 01)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 04)

[root@sakuramist-nas-3588 ~]# uname -a
Linux sakuramist-nas-3588 6.3.0-rc7-next-20230421-NAS+ #6 SMP PREEMPT Tue Apr 25 17:00:30 CST 2023 aarch64 GNU/Linux
thefiredragon commented 1 year ago

@strongtz could your share your image without PCIe DXE or describe how you removed it? Best regards

strongtz commented 1 year ago

@strongtz could your share your image without PCIe DXE or describe how you removed it? Best regards

diff --git a/edk2-platforms/Platform/Radxa/ROCK5B/ROCK5B.fdf b/edk2-platforms/Platform/Radxa/ROCK5B/ROCK5B.fdf
index 2250f5b0..c83c425d 100644
--- a/edk2-platforms/Platform/Radxa/ROCK5B/ROCK5B.fdf
+++ b/edk2-platforms/Platform/Radxa/ROCK5B/ROCK5B.fdf
@@ -235,14 +235,14 @@ READ_LOCK_STATUS   = TRUE

   #PCIe
-  INF Silicon/Rockchip/Drivers/PciPlatform/PcieInitDxe.inf
+  # INF Silicon/Rockchip/Drivers/PciPlatform/PcieInitDxe.inf

   # Required by PCI
   INF ArmPkg/Drivers/ArmPciCpuIo2Dxe/ArmPciCpuIo2Dxe.inf

   # PCI Support
-  INF MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf
-  INF MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridgeDxe.inf
+  # INF MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf
+  # INF MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridgeDxe.inf
   INF MdeModulePkg/Bus/Pci/NvmExpressDxe/NvmExpressDxe.inf
   #INF MdeModulePkg/Bus/Pci/NvmExpressPei/NvmExpressPei.inf
mariobalanica commented 1 year ago

Is this still an issue with latest changes and Linux booted with a device tree?

ACPI-wise with generic OS drivers I'm afraid we can't do much. For anything more complex than a single device on bus 1, I guess custom ECAM accessors are needed (similar to the old PciExpressLib implementation in UEFI, but that was also broken).

strongtz commented 1 year ago

Is this still an issue with latest changes and Linux booted with a device tree?

ACPI-wise with generic OS drivers I'm afraid we can't do much. For anything more complex than a single device on bus 1, I guess custom ECAM accessors are needed (similar to the old PciExpressLib implementation in UEFI, but that was also broken).

I can confirm that PCIe switch works without any problem on Linux 6.2 + devicetree now. Gonna test ACPI later

strongtz commented 1 year ago

looks like the switch doesn't get recognized in Linux with ACPI

[    0.027417] ACPI: PCI Root Bridge [PCI2] (domain 0002 [bus 01])
[    0.027429] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.027553] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]
[    0.027751] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]
[    0.028029] acpi PNP0A08:00: ECAM area [mem 0x980108000-0x980207fff] reserved by PNP0C02:00
[    0.028049] acpi PNP0A08:00: ECAM at [mem 0x980108000-0x980207fff] for [bus 01]
[    0.028084] ACPI: Remapped I/O 0x00000009bfff0000 to [io  0x0000-0xffff window]
[    0.028158] PCI host bridge to bus 0002:01
[    0.028162] pci_bus 0002:01: root bus resource [mem 0xf2000000-0xf2ffffff window]
[    0.028167] pci_bus 0002:01: root bus resource [mem 0x990000000-0x9bffeffff window]
[    0.028171] pci_bus 0002:01: root bus resource [io  0x0000-0xffff window]
[    0.028175] pci_bus 0002:01: root bus resource [bus 01]
[    0.028223] pci 0002:01:00.0: [10ec:c822] type 00 class 0x028000
[    0.028254] pci 0002:01:00.0: reg 0x10: [io  0x0000-0x00ff]
[    0.028287] pci 0002:01:00.0: reg 0x18: [mem 0xf2000000-0xf200ffff 64bit]
[    0.028453] pci 0002:01:00.0: supports D1 D2
[    0.028457] pci 0002:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.028767] pci_bus 0002:01: on NUMA node 0
[    0.028778] pci 0002:01:00.0: BAR 2: assigned [mem 0x990000000-0x99000ffff 64bit]
[    0.028799] pci 0002:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    0.028809] pci_bus 0002:01: resource 4 [mem 0xf2000000-0xf2ffffff window]
[    0.028813] pci_bus 0002:01: resource 5 [mem 0x990000000-0x9bffeffff window]
[    0.028817] pci_bus 0002:01: resource 6 [io  0x0000-0xffff window]
[    0.028902] ACPI: PCI Root Bridge [PCI4] (domain 0004 [bus 01])
[    0.028911] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.029027] acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug LTR]
[    0.029225] acpi PNP0A08:01: _OSC: OS now controls [PME AER PCIeCapability]
[    0.029533] acpi PNP0A08:01: ECAM area [mem 0xa00108000-0xa00207fff] reserved by PNP0C02:01
[    0.029551] acpi PNP0A08:01: ECAM at [mem 0xa00108000-0xa00207fff] for [bus 01]
[    0.029582] ACPI: Remapped I/O 0x0000000a3fff0000 to [io  0x10000-0x1ffff window]
[    0.029657] PCI host bridge to bus 0004:01
[    0.029660] pci_bus 0004:01: root bus resource [mem 0xf4000000-0xf4ffffff window]
[    0.029665] pci_bus 0004:01: root bus resource [mem 0xa10000000-0xa3ffeffff window]
[    0.029670] pci_bus 0004:01: root bus resource [io  0x10000-0x1ffff window] (bus address [0x0000-0xffff])
[    0.029674] pci_bus 0004:01: root bus resource [bus 01]
[    0.029704] pci 0004:01:00.0: [10ec:8125] type 00 class 0x020000
[    0.029725] pci 0004:01:00.0: reg 0x10: [io  0x10000-0x100ff]
[    0.029751] pci 0004:01:00.0: reg 0x18: [mem 0xf4000000-0xf400ffff 64bit]
[    0.029768] pci 0004:01:00.0: reg 0x20: [mem 0xf4010000-0xf4013fff 64bit]
[    0.029878] pci 0004:01:00.0: supports D1 D2
[    0.029881] pci 0004:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.030182] pci 0004:01:1f.0: [e000:034c] type 7f class 0xffffff
[    0.030214] pci 0004:01:1f.0: unknown header type 7f, ignoring device
[    0.030218] pci_bus 0004:01: on NUMA node 0
[    0.030225] pci 0004:01:00.0: BAR 2: assigned [mem 0xa10000000-0xa1000ffff 64bit]
[    0.030239] pci 0004:01:00.0: BAR 4: assigned [mem 0xa10010000-0xa10013fff 64bit]
[    0.030252] pci 0004:01:00.0: BAR 0: assigned [io  0x10000-0x100ff]
[    0.030260] pci_bus 0004:01: resource 4 [mem 0xf4000000-0xf4ffffff window]
[    0.030264] pci_bus 0004:01: resource 5 [mem 0xa10000000-0xa3ffeffff window]
[    0.030268] pci_bus 0004:01: resource 6 [io  0x10000-0x1ffff window]
[    0.030351] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 01])
[    0.030359] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.030474] acpi PNP0A08:02: _OSC: platform does not support [PCIeHotplug LTR]
[    0.030673] acpi PNP0A08:02: _OSC: OS now controls [PME AER PCIeCapability]
[    0.031013] acpi PNP0A08:02: ECAM area [mem 0x900108000-0x900207fff] reserved by PNP0C02:02
[    0.031030] acpi PNP0A08:02: ECAM at [mem 0x900108000-0x900207fff] for [bus 01]
[    0.031062] ACPI: Remapped I/O 0x000000093fff0000 to [io  0x20000-0x2ffff window]
[    0.031133] PCI host bridge to bus 0000:01
[    0.031137] pci_bus 0000:01: root bus resource [mem 0xf0000000-0xf0ffffff window]
[    0.031141] pci_bus 0000:01: root bus resource [mem 0x910000000-0x93ffeffff window]
[    0.031146] pci_bus 0000:01: root bus resource [io  0x20000-0x2ffff window] (bus address [0x0000-0xffff])
[    0.031150] pci_bus 0000:01: root bus resource [bus 01]
[    0.031189] pci_bus 0000:01: on NUMA node 0
[    0.031193] pci_bus 0000:01: resource 4 [mem 0xf0000000-0xf0ffffff window]
[    0.031197] pci_bus 0000:01: resource 5 [mem 0x910000000-0x93ffeffff window]
[    0.031201] pci_bus 0000:01: resource 6 [io  0x20000-0x2ffff window]
[    0.033225] pnp: PnP ACPI init
[    0.033441] system 00:00: [mem 0x980000000-0x98fffffff window] could not be reserved
[    0.033448] system 00:00: [mem 0xa40800000-0xa40bfffff window] has been reserved
[    0.033514] system 00:01: [mem 0xa00000000-0xa0fffffff window] could not be reserved
[    0.033520] system 00:01: [mem 0xa41000000-0xa413fffff window] has been reserved
[    0.033608] system 00:02: [mem 0x900000000-0x90fffffff window] could not be reserved
[    0.033614] system 00:02: [mem 0xa40000000-0xa403fffff window] has been reserved
$ lspci
0002:01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter
0004:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 04)

btw the mac address of rtl8125 seems to have changed when booting with ACPI, not sure if the following messages are relevant

[    0.030182] pci 0004:01:1f.0: [e000:034c] type 7f class 0xffffff
[    0.030214] pci 0004:01:1f.0: unknown header type 7f, ignoring device
[  463.613817] r8169 0004:01:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM
strongtz commented 1 year ago

I can confirm that PCIe switch works without any problem on Linux 6.2 + devicetree now.

Well, I think there are still hidden problems. The NVMe devices are not detected sometimes with device tree boot. 0~4 drives can get detected on each boot. And even if they're all detected, timeouts could occur.

mariobalanica commented 1 year ago

Do the NVME drives behind your switch work in UEFI if you increase the bus range to 15?: https://github.com/edk2-porting/edk2-rk3588/blob/f85454f8b72feb602fa56b3b3d9b8c5f07d5928c/edk2-rockchip/Silicon/Rockchip/RK3588/Library/Rk3588PciHostBridgeLib/PciHostBridgeLib.c#L195

Same can be done for ACPI:

https://github.com/edk2-porting/edk2-rk3588/blob/f85454f8b72feb602fa56b3b3d9b8c5f07d5928c/edk2-rockchip/Silicon/Rockchip/RK3588/AcpiTables/Pcie3x4.asl#L31-L39

https://github.com/edk2-porting/edk2-rk3588/blob/f85454f8b72feb602fa56b3b3d9b8c5f07d5928c/edk2-rockchip/Silicon/Rockchip/RK3588/AcpiTables/Mcfg.aslc#L38-L42

Set both max range and length to 15.

I got my hands on a board with 2x RTL8125 behind an ASM1182e. Both the switch and NICs appear to enumerate correctly, but the Realtek driver does not load (bad resource map?)

image

Linux dmesg 

[    0.044878] ACPI: bus type PCI registered
[    0.044884] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.045091] Serial: AMBA PL011 UART driver
[    0.050171] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.050178] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.050182] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.050187] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.053726] ACPI: Added _OSI(Module Device)
[    0.053731] ACPI: Added _OSI(Processor Device)
[    0.053735] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.053739] ACPI: Added _OSI(Processor Aggregator Device)
[    0.053743] ACPI: Added _OSI(Linux-Dell-Video)
[    0.053747] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.053751] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    0.054697] ACPI: 5 ACPI AML tables successfully acquired and loaded
[    0.055156] ACPI: Interpreter enabled
[    0.055162] ACPI: Using GIC for interrupt routing
[    0.055185] ACPI: MCFG table detected, 5 entries
[    0.059744] ACPI: PCI Root Bridge [PCI2] (domain 0002 [bus 01-0f])
[    0.059760] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    0.060027] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug SHPCHotplug LTR DPC]
[    0.060269] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]
[    0.060611] acpi PNP0A08:00: ECAM area [mem 0x980108000-0x981007fff] reserved by PNP0C02:00
[    0.060689] acpi PNP0A08:00: ECAM at [mem 0x980108000-0x981007fff] for [bus 01-0f]
[    0.060739] ACPI: Remapped I/O 0x00000009bfff0000 to [io  0x0000-0xffff window]
[    0.060843] PCI host bridge to bus 0002:01
[    0.060851] pci_bus 0002:01: root bus resource [bus 01-0f]
[    0.060857] pci_bus 0002:01: root bus resource [io  0x0000-0xffff window]
[    0.060862] pci_bus 0002:01: root bus resource [mem 0xf2000000-0xf2ffffff window]
[    0.060868] pci_bus 0002:01: root bus resource [mem 0x990000000-0x9bffeffff window]
[    0.060911] pci 0002:01:00.0: [1b21:1182] type 01 class 0x060400
[    0.060967] pci 0002:01:00.0: enabling Extended Tags
[    0.061046] pci 0002:01:00.0: PME# supported from D0 D3hot D3cold
[    0.061336] pci 0002:02:02.0: [1b21:1182] type 01 class 0x060400
[    0.061395] pci 0002:02:02.0: enabling Extended Tags
[    0.061465] pci 0002:02:02.0: PME# supported from D0 D3hot D3cold
[    0.061626] pci 0002:02:06.0: [1b21:1182] type 01 class 0x060400
[    0.061684] pci 0002:02:06.0: enabling Extended Tags
[    0.061754] pci 0002:02:06.0: PME# supported from D0 D3hot D3cold
[    0.061961] pci 0002:02:1f.0: [10ec:8125] type 00 class 0x020000
[    0.061990] pci 0002:02:1f.0: reg 0x10: [io  0x1000-0x10ff]
[    0.062020] pci 0002:02:1f.0: reg 0x18: [mem 0xf2100000-0xf210ffff 64bit]
[    0.062042] pci 0002:02:1f.0: reg 0x20: [mem 0xf2110000-0xf2113fff 64bit]
[    0.062196] pci 0002:02:1f.0: supports D1 D2
[    0.062201] pci 0002:02:1f.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.062601] pci 0002:01:00.0: BAR 14: assigned [mem 0xf2000000-0xf20fffff]
[    0.062609] pci 0002:01:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    0.062618] pci 0002:02:1f.0: BAR 2: assigned [mem 0xf2000000-0xf200ffff 64bit]
[    0.062638] pci 0002:02:1f.0: BAR 4: assigned [mem 0xf2010000-0xf2013fff 64bit]
[    0.062656] pci 0002:02:1f.0: BAR 0: assigned [io  0x1000-0x10ff]
[    0.062666] pci 0002:02:02.0: PCI bridge to [bus 03]
[    0.062681] pci 0002:02:06.0: PCI bridge to [bus 04]
[    0.062696] pci 0002:01:00.0: PCI bridge to [bus 02-04]
[    0.062701] pci 0002:01:00.0:   bridge window [io  0x1000-0x1fff]
[    0.062709] pci 0002:01:00.0:   bridge window [mem 0xf2000000-0xf20fffff]
[    0.062721] pci_bus 0002:01: resource 4 [io  0x0000-0xffff window]
[    0.062726] pci_bus 0002:01: resource 5 [mem 0xf2000000-0xf2ffffff window]
[    0.062731] pci_bus 0002:01: resource 6 [mem 0x990000000-0x9bffeffff window]
[    0.062737] pci_bus 0002:02: resource 0 [io  0x1000-0x1fff]
[    0.062741] pci_bus 0002:02: resource 1 [mem 0xf2000000-0xf20fffff]
[    0.062853] ACPI: PCI Root Bridge [PCI3] (domain 0003 [bus 01])
[    0.062865] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    0.063122] acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug SHPCHotplug LTR DPC]
[    0.063364] acpi PNP0A08:01: _OSC: OS now controls [PME AER PCIeCapability]
[    0.063742] acpi PNP0A08:01: ECAM area [mem 0x9c0108000-0x9c0207fff] reserved by PNP0C02:01
[    0.063759] acpi PNP0A08:01: ECAM at [mem 0x9c0108000-0x9c0207fff] for [bus 01]
[    0.063802] ACPI: Remapped I/O 0x00000009ffff0000 to [io  0x10000-0x1ffff window]
[    0.063895] PCI host bridge to bus 0003:01
[    0.063902] pci_bus 0003:01: root bus resource [bus 01]
[    0.063909] pci_bus 0003:01: root bus resource [io  0x10000-0x1ffff window] (bus address [0x0000-0xffff])
[    0.063914] pci_bus 0003:01: root bus resource [mem 0xf3000000-0xf3ffffff window]
[    0.063920] pci_bus 0003:01: root bus resource [mem 0x9d0000000-0x9fffeffff window]
[    1.702866] pci_bus 0003:01: resource 4 [io  0x10000-0x1ffff window]
[    1.702875] pci_bus 0003:01: resource 5 [mem 0xf3000000-0xf3ffffff window]
[    1.702881] pci_bus 0003:01: resource 6 [mem 0x9d0000000-0x9fffeffff window]
[    1.702996] ACPI: PCI Root Bridge [PCI4] (domain 0004 [bus 01])
[    1.703007] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    1.703264] acpi PNP0A08:02: _OSC: platform does not support [PCIeHotplug SHPCHotplug LTR DPC]
[    1.703506] acpi PNP0A08:02: _OSC: OS now controls [PME AER PCIeCapability]
[    1.703926] acpi PNP0A08:02: ECAM area [mem 0xa00108000-0xa00207fff] reserved by PNP0C02:02
[    1.703944] acpi PNP0A08:02: ECAM at [mem 0xa00108000-0xa00207fff] for [bus 01]
[    1.703988] ACPI: Remapped I/O 0x0000000a3fff0000 to [io  0x20000-0x2ffff window]
[    1.704089] PCI host bridge to bus 0004:01
[    1.704096] pci_bus 0004:01: root bus resource [bus 01]
[    1.704102] pci_bus 0004:01: root bus resource [io  0x20000-0x2ffff window] (bus address [0x0000-0xffff])
[    1.704108] pci_bus 0004:01: root bus resource [mem 0xf4000000-0xf4ffffff window]
[    1.704114] pci_bus 0004:01: root bus resource [mem 0xa10000000-0xa3ffeffff window]
[    3.343057] pci_bus 0004:01: resource 4 [io  0x20000-0x2ffff window]
[    3.343064] pci_bus 0004:01: resource 5 [mem 0xf4000000-0xf4ffffff window]
[    3.343069] pci_bus 0004:01: resource 6 [mem 0xa10000000-0xa3ffeffff window]
[    3.343171] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 01])
[    3.343183] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    3.343450] acpi PNP0A08:03: _OSC: platform does not support [PCIeHotplug SHPCHotplug LTR DPC]
[    3.343692] acpi PNP0A08:03: _OSC: OS now controls [PME AER PCIeCapability]
[    3.344150] acpi PNP0A08:03: ECAM area [mem 0x900108000-0x900207fff] reserved by PNP0C02:03
[    3.344176] acpi PNP0A08:03: ECAM at [mem 0x900108000-0x900207fff] for [bus 01]
[    3.344221] ACPI: Remapped I/O 0x000000093fff0000 to [io  0x30000-0x3ffff window]
[    3.344308] PCI host bridge to bus 0000:01
[    3.344314] pci_bus 0000:01: root bus resource [bus 01]
[    3.344320] pci_bus 0000:01: root bus resource [io  0x30000-0x3ffff window] (bus address [0x0000-0xffff])
[    3.344326] pci_bus 0000:01: root bus resource [mem 0xf0000000-0xf0ffffff window]
[    3.344332] pci_bus 0000:01: root bus resource [mem 0x910000000-0x93ffeffff window]
[    3.344363] pci 0000:01:00.0: [1344:5410] type 00 class 0x010802
[    3.344383] pci 0000:01:00.0: reg 0x10: [mem 0xf0000000-0xf0003fff 64bit]
[    3.344623] pci 0000:01:00.0: BAR 0: assigned [mem 0x910000000-0x910003fff 64bit]
[    3.344637] pci_bus 0000:01: resource 4 [io  0x30000-0x3ffff window]
[    3.344642] pci_bus 0000:01: resource 5 [mem 0xf0000000-0xf0ffffff window]
[    3.344648] pci_bus 0000:01: resource 6 [mem 0x910000000-0x93ffeffff window]
[    3.345005] iommu: Default domain type: Translated
[    3.345010] iommu: DMA domain TLB invalidation policy: strict mode
[    3.345609] SCSI subsystem initialized
[    3.345695] libata version 3.00 loaded.
[    3.345785] vgaarb: loaded
[    3.345846] ACPI: bus type USB registered
[    3.345888] usbcore: registered new interface driver usbfs
[    3.345910] usbcore: registered new interface driver hub
[    3.345929] usbcore: registered new device driver usb
[    3.346015] pps_core: LinuxPPS API ver. 1 registered
[    3.346019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti 
[    3.346029] PTP clock support registered
[    3.346139] EDAC MC: Ver: 3.0.0
[    3.346445] Registered efivars operations
[    3.347236] NetLabel: Initializing
[    3.347241] NetLabel:  domain hash size = 128
[    3.347245] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    3.347281] NetLabel:  unlabeled traffic allowed by default
[    3.347532] clocksource: Switched to clocksource arch_sys_counter
[    3.378984] VFS: Disk quotas dquot_6.6.0
[    3.379031] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    3.379420] AppArmor: AppArmor Filesystem Enabled
[    3.379472] pnp: PnP ACPI init
[    3.379784] system 00:00: [mem 0x980000000-0x98fffffff window] could not be reserved
[    3.379795] system 00:00: [mem 0xa40800000-0xa40bfffff window] has been reserved
[    3.379890] system 00:01: [mem 0x9c0000000-0x9cfffffff window] could not be reserved
[    3.379898] system 00:01: [mem 0xa40c00000-0xa40ffffff window] has been reserved
[    3.379994] system 00:02: [mem 0xa00000000-0xa0fffffff window] could not be reserved
[    3.380002] system 00:02: [mem 0xa41000000-0xa413fffff window] has been reserved
[    3.380082] system 00:03: [mem 0x900000000-0x90fffffff window] could not be reserved
[    3.380090] system 00:03: [mem 0xa40000000-0xa403fffff window] has been reserved
Linux lspci 

0000:01:00.0 Non-Volatile memory controller: Micron Technology Inc Device 5410 (rev 01)
0002:01:00.0 PCI bridge: ASMedia Technology Inc. ASM1182e 2-Port PCIe x1 Gen2 Packet Switch
0002:02:02.0 PCI bridge: ASMedia Technology Inc. ASM1182e 2-Port PCIe x1 Gen2 Packet Switch
0002:02:06.0 PCI bridge: ASMedia Technology Inc. ASM1182e 2-Port PCIe x1 Gen2 Packet Switch
0002:02:1f.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
strongtz commented 1 year ago

Do the NVME drives behind your switch work in UEFI if you increase the bus range to 15?:

No, I don't get anything after patching the code:

Shell> pci
   Seg  Bus  Dev  Func
   ---  ---  ---  ----
    00   00   00    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1D87 Device 3588 Prog Interface 0
    04   00   00    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1D87 Device 3588 Prog Interface 0
    04   01   00    00 ==> Network Controller - Ethernet controller
             Vendor 10EC Device 8125 Prog Interface 0
mariobalanica commented 1 year ago

Can we close this as fixed now?

ACPI support for switches is going to need OS-specific quirks.

strongtz commented 1 year ago

Can we close this as fixed now?

Sure.