Open MaciekMachni opened 2 years ago
On 8/12/22 3:41 AM, Maciek Machnikowski @.***> wrote:
The driver loads but fails to allocate interrupts when connecting the TimeCard to the RPi CM4 2GB and 4GB modules. As a result, no TTY ports are present. The same card works properly in the x86 platform. Tested on the in-tree 5.19.0 driver.
|[ 4.278190] ptp_ocp 0000:03:00.0: enabling device (0000 -> 0002) [ 4.279502] ptp_ocp 0000:03:00.0: irq 1 out of range, skipping ts0 [ 4.279531] ptp_ocp 0000:03:00.0: irq 2 out of range, skipping ts1 [ 4.279546] ptp_ocp 0000:03:00.0: irq 6 out of range, skipping ts2 [ 4.279559] ptp_ocp 0000:03:00.0: irq 15 out of range, skipping ts3 [ 4.279572] ptp_ocp 0000:03:00.0: irq 16 out of range, skipping ts4 [ 4.279693] ptp_ocp 0000:03:00.0: irq 11 out of range, skipping signal_out[0] [ 4.279708] ptp_ocp 0000:03:00.0: irq 12 out of range, skipping signal_out[1] [ 4.279721] ptp_ocp 0000:03:00.0: irq 13 out of range, skipping signal_out[2] [ 4.279733] ptp_ocp 0000:03:00.0: irq 14 out of range, skipping signal_out[3] [ 4.280026] ptp_ocp 0000:03:00.0: irq 7 out of range, skipping i2c_ctrl [ 4.280040] ptp_ocp 0000:03:00.0: irq 3 out of range, skipping gnss_port [ 4.280052] ptp_ocp 0000:03:00.0: irq 4 out of range, skipping gnss2_port [ 4.280064] ptp_ocp 0000:03:00.0: irq 5 out of range, skipping mac_port [ 4.280077] ptp_ocp 0000:03:00.0: irq 10 out of range, skipping nmea_port [ 4.280089] ptp_ocp 0000:03:00.0: irq 9 out of range, skipping spi_flash [ 4.359245] ptp_ocp 0000:03:00.0: Version 1.2.0, clock PPS, device ptp1 [ 4.359297] ptp_ocp 0000:03:00.0: Time: 1660295049.472829090, in-sync |
The CM4s' datasheet states that the module should be able to allocate up to 32 vectors, and the lspci confirms it:
|03:00.0 Memory controller: Facebook, Inc. Device 0400 Subsystem: Xilinx Corporation Device 0007 Flags: bus master, fast devsel, latency 0, IRQ 39 Memory at 600000000 (32-bit, non-prefetchable) [size=32M] Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable+ Count=1/32 Maskable- 64bit+ Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00 Kernel driver in use: ptp_ocp Kernel modules: ptp_ocp |
— Reply to this email directly, view it on GitHub https://github.com/opencomputeproject/Time-Appliance-Project/issues/73, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADG5AT65IK6WVBNXIXLRNCTVYYS5DANCNFSM56LHQAZA. You are receiving this because you are subscribed to this thread.Message ID: @.***>
Unfortunately, the CM4 doesn't support multiple MSI vectors.
Hmmm - RPi team claims it supports up to 32 vectors.
Although the problem seems to be present in different devices too. https://forums.raspberrypi.com/viewtopic.php?t=322859
On Aug 17, 2022, at 2:42 AM, Maciek Machnikowski @.***> wrote:
Hmmm - RPi team claims it supports up to 32 vectors.
Although the problem seems to be present in different devices too. https://forums.raspberrypi.com/viewtopic.php?t=322859
MSI and MSI-X are different.
On Aug 17, 2022, at 2:42 AM, Maciek Machnikowski @.***> wrote: Hmmm - RPi team claims it supports up to 32 vectors. Although the problem seems to be present in different devices too. https://forums.raspberrypi.com/viewtopic.php?t=322859 MSI and MSI-X are different.
I know, and I think this is just a typo, and they meant MSI. The pcie-brcmstb.c seems to confirm that:
#define BRCM_INT_PCI_MSI_NR 32
#define BRCM_INT_PCI_MSI_LEGACY_NR 8
...
static int brcm_pcie_enable_msi(struct brcm_pcie *pcie)
{
struct brcm_msi *msi;
int irq, ret;
struct device *dev = pcie->dev;
irq = irq_of_parse_and_map(dev->of_node, 1);
if (irq <= 0) {
dev_err(dev, "cannot map MSI interrupt\n");
return -ENODEV;
}
msi = devm_kzalloc(dev, sizeof(struct brcm_msi), GFP_KERNEL);
if (!msi)
return -ENOMEM;
mutex_init(&msi->lock);
msi->dev = dev;
msi->base = pcie->base;
msi->np = pcie->np;
msi->target_addr = pcie->msi_target_addr;
msi->irq = irq;
msi->legacy = pcie->hw_rev < BRCM_PCIE_HW_REV_33;
/*
* Sanity check to make sure that the 'used' bitmap in struct brcm_msi
* is large enough.
*/
BUILD_BUG_ON(BRCM_INT_PCI_MSI_LEGACY_NR > BRCM_INT_PCI_MSI_NR);
if (msi->legacy) {
msi->intr_base = msi->base + PCIE_INTR2_CPU_BASE;
msi->nr = BRCM_INT_PCI_MSI_LEGACY_NR;
msi->legacy_shift = 24;
} else {
msi->intr_base = msi->base + PCIE_MSI_INTR2_BASE;
msi->nr = BRCM_INT_PCI_MSI_NR;
msi->legacy_shift = 0;
}
ret = brcm_allocate_domains(msi);
if (ret)
return ret;
irq_set_chained_handler_and_data(msi->irq, brcm_pcie_msi_isr, msi);
brcm_msi_set_regs(msi);
pcie->msi = msi;
return 0;
}
...
static int brcm_msi_alloc(struct brcm_msi *msi)
{
int hwirq;
mutex_lock(&msi->lock);
hwirq = bitmap_find_free_region(msi->used, msi->nr, 0);
mutex_unlock(&msi->lock);
return hwirq;
}
Since this is an open issue, will comment on my current view of this. We're working on enabling MSI-X in the Time Card as the primary mechanism for interrupts. I'm hoping that once we get that accomplished, this issue will be resolved since it seems RPi CM4 can support a large number of MSI-X interrupts.
The driver loads but fails to allocate interrupts when connecting the TimeCard to the RPi CM4 2GB and 4GB modules. As a result, no TTY ports are present. The same card works properly in the x86 platform. Tested on the in-tree 5.19.0 driver.
The CM4s' datasheet states that the module should be able to allocate up to 32 vectors, and the lspci confirms it: