Add TimeHAT V2 Pi 5 HAT from OCP-TAP

[geerlingguy]

OCP-TAP has announced the TimeHAT V2, a Pi 5 HAT that embeds an Intel i226 NIC along with a B-key M.2 slot for cards like GPS/GNSS adapters.

The M.2 slot routes the PPS signal straight to the i226 input, for GPS time synchronization.

The two SMA connectors on the HAT can be configured for PPS in/out (IIRC the Pi 5 only has one PPS pin on it's built-in 1 Gbps NIC, but it is not exposed externally).

The preliminary schematics are available on the OCP-TAP GitHub here, and I have asked if they're considering selling units (or working with someone else to do so), and it sounds like this is a possibility!

From the LinkedIn post at the top of this issue, they also showed a demo of < 10ns sync using PTP with ptp4l:

Here is the TimeHat (open source and available on https://lnkd.in/gWNDJtDn) an add-on for the Raspberry Pi 5 that packs the Intel i226 with two SMA connectors, configurable to do PPS in/out (a feature that is unfortunately missing on the RPi 5). In addition the TimeHat got a connector for the latest RCB m.2 form factor to host a GNSS receiver by routing its PPS signal to the i226 as an input allowing it to become a Time Server for both NTP and PTP.

In this setup two Raspberry Pi 5 boards, each with a TimeHat are getting synchronized (the one on top acts as a time server and the one on the bottom as client) via PTP and both provide a PPS output that is compared (one act as the trigger and one gets measured) on the oscilloscope. The first sigma of the time sync error is under 10ns.

See related: https://pipci.jeffgeerling.com/hats/ocp-tap-timehat.html / #619

[geerlingguy]

I was sent two of these (along with an M.2 u-blox ZED-F9T-00B-01 module, capable of high-accuracy GPS reception.

I hope to get this set up as a time server in my rack soon—maybe I can also work with Masterclock to set up a nice time display as well?

[geerlingguy]

lol, how apropos ShortCircuit just posted a video on the Open Timecard Mini today.

[geerlingguy]

Just booted a new Pi OS install, but for some reason the card is not showing up when I run lspci. I'm wondering if the cable orientation is off if you have it sitting as a HAT (I noticed the photos on the LinkedIn have a cable that's basically flipped opposite of what it would be if you have the HAT on top of the Pi... I think).

[geerlingguy]

Following along with #606 (where we were debugging issues with the i225-V adapter), I've done the following:

1. Run sudo rpi-update
2. Add dtoverlay=pciex1-compat-pi5,mmio-hi to my /boot/firmware/config.txt

After a reboot, I still see:

pi@pi5:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0001:01:00.0 Ethernet controller: Raspberry Pi Ltd RP1 PCIe 2.0 South Bridge

(This is also with dtparam=pciex1 in my /boot/firmware/config.txt file.)

[geerlingguy]

Aha! I was using the little Pineboards FFC with the arrow on the GPIO side of the connection (mostly because I saw there was a tiny triangle on that side of the connector on the HAT, and I love matching triangles:

(In the photo above I just placed the cable atop the other one in the orientation I had it.)

I spun the cable around (did a 180) so the triangle is on the other side:

After booting up, now I see the card:

pi@pi5:~ $ sudo lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0000:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-LM (rev 03)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0001:01:00.0 Ethernet controller: Raspberry Pi Ltd RP1 PCIe 2.0 South Bridge

[geerlingguy]

...surprisingly, it looks like the Intel IGC driver is already present?

pi@pi5:~ $ dmesg | grep igc
[    5.532414] igc 0000:01:00.0: enabling device (0000 -> 0002)
[    5.532448] igc 0000:01:00.0: PCIe PTM not supported by PCIe bus/controller
[    5.592460] igc 0000:01:00.0 (unnamed net_device) (uninitialized): PHC added
[    5.643468] igc 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
[    5.643481] igc 0000:01:00.0 eth1: MAC: 00:a0:c9:00:00:01

pi@pi5:~ $ sudo lspci -vvv
...
0000:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-LM (rev 03)
    Subsystem: Intel Corporation Ethernet Controller I226-LM
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 128 bytes
    Interrupt: pin A routed to IRQ 38
    Region 0: Memory at 1b80000000 (32-bit, non-prefetchable) [size=1M]
    Region 3: Memory at 1b80100000 (32-bit, non-prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Address: 0000000000000000  Data: 0000
        Masking: 00000000  Pending: 00000000
    Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
        Vector table: BAR=3 offset=00000000
        PBA: BAR=3 offset=00002000
    Capabilities: [a0] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
        DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 512 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
        LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s, Width x1
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [140 v1] Device Serial Number 00-a0-c9-ff-ff-00-00-01
    Capabilities: [1c0 v1] Latency Tolerance Reporting
        Max snoop latency: 0ns
        Max no snoop latency: 0ns
    Capabilities: [1f0 v1] Precision Time Measurement
        PTMCap: Requester:+ Responder:- Root:-
        PTMClockGranularity: 4ns
        PTMControl: Enabled:- RootSelected:-
        PTMEffectiveGranularity: Unknown
    Capabilities: [1e0 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
              PortCommonModeRestoreTime=55us PortTPowerOnTime=70us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=55us LTR1.2_Threshold=131072ns
        L1SubCtl2: T_PwrOn=70us
    Kernel driver in use: igc
    Kernel modules: igc

And it's giving me 2.5 Gbps full duplex:

pi@pi5:~ $ sudo ethtool eth1
Settings for eth1:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            2500baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            2500baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 2500Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    MDI-X: off (auto)
    Supports Wake-on: pumbg
    Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
    Link detected: yes

[geerlingguy]

In dmesg, I see the following when I connect a cable:

[  444.905640] igc 0000:01:00.0 eth1: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX

But I don't get an IP address assigned:

pi@pi5:~ $ ip a
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:a0:c9:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ec7b:15ad:149:c0b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

The interface appears in nmtui, and all the settings look correct (auto negotiate an IPv4/IPv6 address, etc.), and if I go to 'Activate' it though, it times out after a while with:

Could not activate connection: Activation failed: IP configuration could not be reserved (no available address, timeout, etc.)

[geerlingguy]

Exploring PTP a bit:

pi@pi5:~ $ sudo apt install linuxptp

pi@pi5:~ $ ethtool -T eth1
Time stamping parameters for eth1:
Capabilities:
    hardware-transmit
    software-transmit
    hardware-receive
    software-receive
    software-system-clock
    hardware-raw-clock
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
    off
    on
Hardware Receive Filter Modes:
    none
    all

pi@pi5:~ $ sudo ptp4l -i eth1 -m
ptp4l[938.535]: selected /dev/ptp0 as PTP clock
ptp4l[938.536]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[938.536]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[946.257]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[946.257]: selected local clock 00a0c9.fffe.000001 as best master
ptp4l[946.257]: port 1: assuming the grand master role
...

Red Hat has some decent docs on Configuring PTP using ptp4l.

[geerlingguy]

To configure PPS signals from the NIC, I should be able to use the testptp software. Going to have to wrap up my testing today but will get back at it soon, and try to sync across two Pi 5s with these TimeHATs!