bcm2711 PCIe Interrupt Pins B through D not functional

[ErikWlfn]

This pertains to Linux for the Raspberry Pi version 5.10.74 and the included Device Tree definition files. The file "arch/arm/boot/dts/bcm2711.dtsi" only defines an interrupt for PCIe interrupt Pin A. Devices that use interrupt Pins C, D, or E will not function properly because they cannot request interrupts.

Existing definition of interrupt map for PCIe in "bcm2711.dtsi":

interrupt-map-mask = <0x0 0x0 0x0 0x7>;
interrupt-map = <0 0 0 1 &gicv2 GIC_SPI 143
        IRQ_TYPE_LEVEL_HIGH>;

The work around or correction to the interrupt map is this:

interrupt-map-mask = <0x0 0x0 0x0 0xFF>;
interrupt-map =
    <0 0 0 1
        &gicv2 GIC_SPI 143 IRQ_TYPE_LEVEL_HIGH>,
    <0 0 0 2
        &gicv2 GIC_SPI 144 IRQ_TYPE_LEVEL_HIGH>,
    <0 0 0 3
        &gicv2 GIC_SPI 145 IRQ_TYPE_LEVEL_HIGH>,
    <0 0 0 4
        &gicv2 GIC_SPI 146 IRQ_TYPE_LEVEL_HIGH>;

I tested this change on the Raspberry Pi Compute Module 4 and associated IO board. There is not enough documentation for the interrupt controllers in the BCM2711 for me to verify that the assigned interrupt request numbers are valid. The pins do have to use different interrupt request numbers, or the associated interrupt handler is not called.

This problem affects any PCIe peripherals or bus multiplexers that use interrupt Pins C through E, either directly or indirectly due to interrupt "swizzling" (rotation). The problem does not affect the operation of Message Signaled Interrupts (MSI) on the PCIe interface because they use a different set of interrupt request lines.

A modified "bcm2711.dtsi" is attached. Search for initials "EWW".

bcm2711-pcie-int.zip

[pelwell]

A patch was submitted upstream for this in the last week. We'll back-port it when it hits the linux-next branch (and we notice...)

[6by9]

For reference, https://lore.kernel.org/linux-arm-kernel/20211029210927.36800-1-f.fainelli@gmail.com/

[geerlingguy]

Quick question, as a PCIe noob: could this affect the performance of setups that have a PCIe switch and a number of downstream devices? In much of my testing in these scenarios, it seems like the total throughput is much less than the ~3.4 Gbps I can get through a single device.

For example, connecting both a SATA controller (which by itself can get through around 3.3 Gbps) and a 2.5G NIC (which by itself can get through around 3.3 Gbps), when I copy files through the NIC to a SATA drive, total file copy throughput is more like 1.2 Gbps, which is far less than half of the total bandwidth.

I expect overhead losses due to the protocols involved, but testing with Samba and NFS, and testing in a variety of setups, I am always seeing IRQ overhead during these operations (much more so when copying from NIC to a SATA drive) when measured with atop/irqtop/etc.

[P33M]

Sharing interrupt lines between devices means that every device driver that has registered to use the interrupt line must be polled each time any one device raises an interrupt. This significantly increases the amount of work done in hardirq context to figure out which device actually needs attention.

[pelwell]

The upstream patch was accepted, and it's now in rpi-5.10.y as a514fef1c808b2fac84a4d03b16d93d9a6d78ffc.