ptp_ocp on RPi CM4 doesn't expose NMEA TTY and GNSS TTY

[MaciekMachni]

The driver loads but fails to allocate interrupts when connecting the TimeCard to the RPi CM4 2GB and 4GB modules. As a result, no TTY ports are present. The same card works properly in the x86 platform. Tested on the in-tree 5.19.0 driver.

[    4.278190] ptp_ocp 0000:03:00.0: enabling device (0000 -> 0002)
[    4.279502] ptp_ocp 0000:03:00.0: irq 1 out of range, skipping ts0
[    4.279531] ptp_ocp 0000:03:00.0: irq 2 out of range, skipping ts1
[    4.279546] ptp_ocp 0000:03:00.0: irq 6 out of range, skipping ts2
[    4.279559] ptp_ocp 0000:03:00.0: irq 15 out of range, skipping ts3
[    4.279572] ptp_ocp 0000:03:00.0: irq 16 out of range, skipping ts4
[    4.279693] ptp_ocp 0000:03:00.0: irq 11 out of range, skipping signal_out[0]
[    4.279708] ptp_ocp 0000:03:00.0: irq 12 out of range, skipping signal_out[1]
[    4.279721] ptp_ocp 0000:03:00.0: irq 13 out of range, skipping signal_out[2]
[    4.279733] ptp_ocp 0000:03:00.0: irq 14 out of range, skipping signal_out[3]
[    4.280026] ptp_ocp 0000:03:00.0: irq 7 out of range, skipping i2c_ctrl
[    4.280040] ptp_ocp 0000:03:00.0: irq 3 out of range, skipping gnss_port
[    4.280052] ptp_ocp 0000:03:00.0: irq 4 out of range, skipping gnss2_port
[    4.280064] ptp_ocp 0000:03:00.0: irq 5 out of range, skipping mac_port
[    4.280077] ptp_ocp 0000:03:00.0: irq 10 out of range, skipping nmea_port
[    4.280089] ptp_ocp 0000:03:00.0: irq 9 out of range, skipping spi_flash
[    4.359245] ptp_ocp 0000:03:00.0: Version 1.2.0, clock PPS, device ptp1
[    4.359297] ptp_ocp 0000:03:00.0: Time: 1660295049.472829090, in-sync

The CM4s' datasheet states that the module should be able to allocate up to 32 vectors, and the lspci confirms it:

03:00.0 Memory controller: Facebook, Inc. Device 0400
        Subsystem: Xilinx Corporation Device 0007
        Flags: bus master, fast devsel, latency 0, IRQ 39
        Memory at 600000000 (32-bit, non-prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/32 Maskable- 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
        Kernel driver in use: ptp_ocp
        Kernel modules: ptp_ocp

[jlemon]

On 8/12/22 3:41 AM, Maciek Machnikowski @.***> wrote:

The driver loads but fails to allocate interrupts when connecting the TimeCard to the RPi CM4 2GB and 4GB modules. As a result, no TTY ports are present. The same card works properly in the x86 platform. Tested on the in-tree 5.19.0 driver.

|[ 4.278190] ptp_ocp 0000:03:00.0: enabling device (0000 -> 0002) [ 4.279502] ptp_ocp 0000:03:00.0: irq 1 out of range, skipping ts0 [ 4.279531] ptp_ocp 0000:03:00.0: irq 2 out of range, skipping ts1 [ 4.279546] ptp_ocp 0000:03:00.0: irq 6 out of range, skipping ts2 [ 4.279559] ptp_ocp 0000:03:00.0: irq 15 out of range, skipping ts3 [ 4.279572] ptp_ocp 0000:03:00.0: irq 16 out of range, skipping ts4 [ 4.279693] ptp_ocp 0000:03:00.0: irq 11 out of range, skipping signal_out[0] [ 4.279708] ptp_ocp 0000:03:00.0: irq 12 out of range, skipping signal_out[1] [ 4.279721] ptp_ocp 0000:03:00.0: irq 13 out of range, skipping signal_out[2] [ 4.279733] ptp_ocp 0000:03:00.0: irq 14 out of range, skipping signal_out[3] [ 4.280026] ptp_ocp 0000:03:00.0: irq 7 out of range, skipping i2c_ctrl [ 4.280040] ptp_ocp 0000:03:00.0: irq 3 out of range, skipping gnss_port [ 4.280052] ptp_ocp 0000:03:00.0: irq 4 out of range, skipping gnss2_port [ 4.280064] ptp_ocp 0000:03:00.0: irq 5 out of range, skipping mac_port [ 4.280077] ptp_ocp 0000:03:00.0: irq 10 out of range, skipping nmea_port [ 4.280089] ptp_ocp 0000:03:00.0: irq 9 out of range, skipping spi_flash [ 4.359245] ptp_ocp 0000:03:00.0: Version 1.2.0, clock PPS, device ptp1 [ 4.359297] ptp_ocp 0000:03:00.0: Time: 1660295049.472829090, in-sync |

The CM4s' datasheet states that the module should be able to allocate up to 32 vectors, and the lspci confirms it:

|03:00.0 Memory controller: Facebook, Inc. Device 0400 Subsystem: Xilinx Corporation Device 0007 Flags: bus master, fast devsel, latency 0, IRQ 39 Memory at 600000000 (32-bit, non-prefetchable) [size=32M] Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable+ Count=1/32 Maskable- 64bit+ Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00 Kernel driver in use: ptp_ocp Kernel modules: ptp_ocp |

— Reply to this email directly, view it on GitHub https://github.com/opencomputeproject/Time-Appliance-Project/issues/73, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADG5AT65IK6WVBNXIXLRNCTVYYS5DANCNFSM56LHQAZA. You are receiving this because you are subscribed to this thread.Message ID: @.***>

Unfortunately, the CM4 doesn't support multiple MSI vectors.

[MaciekMachni]

Hmmm - RPi team claims it supports up to 32 vectors.

Although the problem seems to be present in different devices too. https://forums.raspberrypi.com/viewtopic.php?t=322859

[jlemon]

On Aug 17, 2022, at 2:42 AM, Maciek Machnikowski @.***> wrote:

 Hmmm - RPi team claims it supports up to 32 vectors.

Although the problem seems to be present in different devices too. https://forums.raspberrypi.com/viewtopic.php?t=322859

MSI and MSI-X are different.

[MaciekMachni]

On Aug 17, 2022, at 2:42 AM, Maciek Machnikowski @.***> wrote:  Hmmm - RPi team claims it supports up to 32 vectors. Although the problem seems to be present in different devices too. https://forums.raspberrypi.com/viewtopic.php?t=322859 MSI and MSI-X are different.

I know, and I think this is just a typo, and they meant MSI. The pcie-brcmstb.c seems to confirm that:

#define BRCM_INT_PCI_MSI_NR     32
#define BRCM_INT_PCI_MSI_LEGACY_NR  8
...
static int brcm_pcie_enable_msi(struct brcm_pcie *pcie)
{
    struct brcm_msi *msi;
    int irq, ret;
    struct device *dev = pcie->dev;

    irq = irq_of_parse_and_map(dev->of_node, 1);
    if (irq <= 0) {
        dev_err(dev, "cannot map MSI interrupt\n");
        return -ENODEV;
    }

    msi = devm_kzalloc(dev, sizeof(struct brcm_msi), GFP_KERNEL);
    if (!msi)
        return -ENOMEM;

    mutex_init(&msi->lock);
    msi->dev = dev;
    msi->base = pcie->base;
    msi->np = pcie->np;
    msi->target_addr = pcie->msi_target_addr;
    msi->irq = irq;
    msi->legacy = pcie->hw_rev < BRCM_PCIE_HW_REV_33;

    /*
     * Sanity check to make sure that the 'used' bitmap in struct brcm_msi
     * is large enough.
     */
    BUILD_BUG_ON(BRCM_INT_PCI_MSI_LEGACY_NR > BRCM_INT_PCI_MSI_NR);

    if (msi->legacy) {
        msi->intr_base = msi->base + PCIE_INTR2_CPU_BASE;
        msi->nr = BRCM_INT_PCI_MSI_LEGACY_NR;
        msi->legacy_shift = 24;
    } else {
        msi->intr_base = msi->base + PCIE_MSI_INTR2_BASE;
        msi->nr = BRCM_INT_PCI_MSI_NR;
        msi->legacy_shift = 0;
    }

    ret = brcm_allocate_domains(msi);
    if (ret)
        return ret;

    irq_set_chained_handler_and_data(msi->irq, brcm_pcie_msi_isr, msi);

    brcm_msi_set_regs(msi);
    pcie->msi = msi;

    return 0;
}
...
static int brcm_msi_alloc(struct brcm_msi *msi)
{
    int hwirq;

    mutex_lock(&msi->lock);
    hwirq = bitmap_find_free_region(msi->used, msi->nr, 0);
    mutex_unlock(&msi->lock);

    return hwirq;
}

[julianstj1]

Since this is an open issue, will comment on my current view of this. We're working on enabling MSI-X in the Time Card as the primary mechanism for interrupts. I'm hoping that once we get that accomplished, this issue will be resolved since it seems RPi CM4 can support a large number of MSI-X interrupts.