CM4 is missing IEEE1588-2008 support through BCM54210PE

[tschiemer]

Is this the right place for my bug report? Yes: I asked on the forum, I asked the raspberry foundation through the contact form but didn't get any response - it is very much kernel related as is explained in the forum post.

Describe the bug The CM4 has an undocumented inofficial PHY (BCM54210PE) that supports hardware timestamping (and a hardware clock, actually??), but there seems to be no driver supporting these features.

So CM4 does actually not provide hardware based IEEE1588-2008 support as opposed to what the raspberry pi foundation actually communicates in the CM4 datasheet.

To reproduce sudo ethtool -T eth0 only shows software timestamping capabilities.

Expected behaviour Depends - the raspberry pi foundation never communicated what IEEE1588-2008 features are supported such that is is unclear what exactly can be expected; also the PHY documentation is not available such that possibilities (ie optimal expected behaviour) can not be described.

Expected (according to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-using_ptp):

• eth support for:
  • SOF_TIMESTAMPING_TX_HARDWARE
  • SOF_TIMESTAMPING_RX_HARDWARE
  • (possibly) SOF_TIMESTAMPING_RAW_HARDWARE
• shows PTP Hardware Clock (because BCM54210PE has on-chip clock)
• Hardware Transmit Timestamp Modes ??
• Hardware Receive Filter Modes ??

The CM4 datasheet lists SYNC_IN, SYNC_OUT pins ment for IEEE1588-2008 support but has not further specified this. It actually is a question wether these pins are needed in the first place - because the PHY would seem to have a hardware-clock itself. Further it must be clear wether the PHY's internal clock (if any) is routed to a CM4 pin in any form and if not this should be fixed in future revisions because custom boards for the CM4 might require tight synchronization to a hardware clock.

Actual behaviour sudo ethtool -T eth0 only shows software timestamping capabilities. There is no (documented) synchronizable IEEE1588-2008 clock signal going from the CM4.

System Raspberry Pi Compute Module 4

Raspberry Pi OS Lite, January 11th 2021, Kernel version: 5.4 Ubuntu Server 20.04.2 LTS,

Logs none

Additional context BCM54210PE documentation is required to implement to see actual features provided and configuration options - IEEE802.3-2018 seems to define said capabilities etc. Possibly also required are CM4 schematics and BCM2711 documentation. Also see IEEE802.3-2018 Section 90. Ethernet support for time synchronization protocols

I can try to take care of the implementation, but I would like to have the mentioned documents.

[pelwell]

@ghollingworth?

[tschiemer]

Just compiling a kernel with PTP timestamping support enabled (by default not). Will see if this changes anything.

I'm new to kernel topics, so I missed the obvious. Will report back

[tschiemer]

Built a new kernel according to docs and changed the following settings (through make menuconfig):

CONFIG_NETWORK_PHY_TIMESTAMPING=y

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=m
# CONFIG_DP83640_PHY is not set
# CONFIG_PTP_1588_CLOCK_INES is not set
# CONFIG_PTP_1588_CLOCK_IDT82P33 is not set
# CONFIG_PTP_1588_CLOCK_IDTCM is not set
# end of PTP clock support

Linux cm4-1 5.10.14-v7l-MY_CUSTOM_KERNEL+ #1 SMP Tue Feb 16 14:05:19 GMT 2021 armv7l GNU/Linux

No change observed, ie

pi@cm4-1:~/linux $ ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
    software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
    software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
    software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none

pi@cm4-1:~ $ sudo lshw -class network
  *-network:0
       description: Ethernet interface
       physical id: 2
       logical name: eth0
       serial: dc:a6:32:XX:XX:XX
       size: 1Gbit/s
       capacity: 1Gbit/s
       capabilities: ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=bcmgenet driverversion=5.10.14-v7l-MY_CUSTOM_KERNEL+ duplex=full ip=192.168.2.197 link=yes multicast=yes port=MII speed=1Gbit/s
...

I figure because there is no particular PTP hardware clock (driver) there's no recognized PTP clock (...) - but this would depend on wether BCM54210PE actually would have a hardware clock (?)

bcmgenet does not seem to use phy_devices struct mii_timestamper *mii_ts;

so enabling timestamping has no effect...

For reference: https://github.com/raspberrypi/linux/blob/c9226080e513181ffb3909a905e9c23b8a6e8f62/include/linux/mii_timestamper.h

[ghollingworth]

We're currently discussing getting support for writing a driver from Broadcom, although we have access to the datasheet, to me it's pretty meaningless!

Unfortunately, so is the Linux interface to the clock...

But we'll endeavour to get there

[tschiemer]

Ok, thanks!

Hoping to be hearing more from you!

Just to say, as for the docs, I wouldn't mind signing an NDA to have a look to see if can make sense of it..

[wowczarek]

Damn you Broadcom and your closed datasheets! Timestamping at the PHY level is the best one can get, but not if unimplemented in software...

I also do hope those SYNC_IN and SYNC_OUT pins are actual event inputs/outputs for external sync and were preserved on the I/O board and not repurposed for something else. I've seen countless SBCs where this potential was wasted.

So, @ghollingworth, aside from a PTP driver eventually being developed - do we know if the two pins mentioned (pins 16 and 18 on the CM4) are presented on the I/O board?

Thanks!

[tschiemer]

@wowczarek

From what I see in the CM4IO datasheet (page 7) SYNC_IN, SYNC_OUT are available on J2 pins 8+9

[pelwell]

Damn those open datasheets!

[wowczarek]

@pelwell haha, senior moment :)

[jaycordaro]

I saw @wowczarek message on TimeNuts. I used to work for Broadcom's Ethernet PHY group and worked on the 5421x series including the IEEE1588 part of the chip. I remain on good relations with my former colleagues. There are several packages and two dies which map to the BCM54210 part number. I think the part on the Raspberry Pi (BCM54213PE? is that correct) does not pin out the SYNC_IN and SYNC_OUT pins even though they are on the die (36 pin package). I will check with them. Even if the HW SYNC pins are not brought out, some functionality of the IEEE-1588 portion of the PHY may be able to be enabled.

@ghollingworth, does the datasheet you have access to mention IEEE-1588?

[dp111]

On the CM4 we use the BCM54210PE and the SYNC_IN and SYNC_OUT are exposed to pins on the CM4 connector. The CM4IO board exposes these signals on connector J2. @jaycordaro if the PHY team are looking at a driver for this it would be great.

[jaycordaro]

ok I spoke to some friends who are still at Broadcom. The PHY does support HW timestamping, the register writes required to enable HW timestamping and the SYNC pins are described in an applications note, "IEEE-1588 for Timing Applications," not in the datasheet, so that may explain why the datasheet didn't provide anything useful. Someone who has Broadcom DocSafe access for BCM54210PE could request that document to be added to their account. The Product Line Manager would then approve the request. The PHY team people that I talked to (senior ASIC mgr for the PHY and someone in Marketing) are not aware of any effort inside the company to support a driver but they will try to help.

[ghollingworth]

Thanks to everyone on this... Especially to Philphil on the forums and @jaycordaro for their help! I do have access to the register information and Broadcom Docsafe and am currently trying to play catchup with the PTP kernel interface and ioctl interface to understand what it is I need to implement to get a working timestamping system.

What's clear is that there are a number of interfaces implementing different functionality, just need to decide on the order of implementation.

What is it that people actually want out of this?

1) external signal timestamping (timestamping of the SYNC_IN signal) 2) external event creation (SYNC_OUT periodic output signal) 3) TX and RX packet hardware timestamping (the BRCM MAC contains a FIFO from which you can read hardware timestamps) 4) PTP Hardware clock (which I think is implied with 1 and 2 but maybe not 3

Gordon

[wowczarek]

@ghollingworth actually 4. is implied exactly in 3., at least from the kernel interface point of view, for network sync (IEEE 1588) to work.

What is needed is all of the items you list, but as to the order:

a. 4. is required because this is the interface to controlling the clock. b. 3. is required to sync this clock over the network c. 1. is required to sync with an external hardware pulse such as from GNSS receivers or other quality clocks d. 2. is required for external measurement and characterisation, which is priceless

• Items 1. and 2. are rare opportunities for hardware this common, giving this platform a chance to be the prime off-the-shelf testing ground for time sync related projects *()**
• Items 3. and 4. are the part that any other 1588-compliant chip has driver support for.

So unselfishly I'd say 3,4, then 1,2, but really it's the whole that people / we are / should be after.

*()** While we are at it, a set of u.fl (or ideally SMA) connectors could be considered for these two pins in the next revision of the carrier board instead, and freeing up the pins for something else. That would mean plug and play for serious timing, but it's not a big issue otherwise, we have the header. This is debatable though - a GNSS receiver on a board will also have a pin, but a more serious GNSS driven oscillator will have a coax connector.

Thanks!

[ghollingworth]

Thanks that helps a lot.

If you've got a suggestion for an Ethernet device that does actually implement these functions from the standard set of drivers:

I would suggest one of these (i.e. drivers that reference the ptp_clock_register function)

https://elixir.bootlin.com/linux/latest/C/ident/ptp_clock_register

That would help me with an initial implementation...

Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...

[tschiemer]

many thanks also!

Well, simply put I would think that all features might serve one use case or other and thus - optimally - all should be available to make the CM4 an attractive platform for any of those use cases.

The PTP hardware clock would seem the most reliable/precise possible and thus if it would be available that would be a definite yes.

Sync'ing external components to the PTP clock is - as I would think - also a core feature 1588 capable platforms should offer; an example being where audio ADCs and DACs should be driven by the sampling frequency derived from the network clock (thinking of AES67 here). As I understand from your wording, the periodic signal is adjustable within a particular range/frequency?

Analogously sync'ing the PTP clock to an external component would be quite interesting a feature to have, but as I understand the way SYNC_IN works is just timestamp generation.

Knowing the precise timestamp of an event on an external component would seem as important; ex knowing the precise time of sample acquisition, in particular if external components are not sync'ed to the PTP clock.

So I would consider 1, 2 and 4 to be quite important features to have for a module aiming at deeply embedded applications and would make the CM4 an interesting base indeed.

As for 3 (selective TX/RX hardware timestamping) nothing immediately comes to mind (not working with such tech) but I'm sure there are meaningful use cases.

wowczarek: I'm not sure 3 is required for actual network/ptp syncing, depends a bit how the 1588 functionalities are implemented (on-chip?)

but yes, I agree, 1+2 are indeed rare opportunities for hardware this common and it's indeed the whole package that should be aimed for

[wowczarek]

@tschiemer indeed, so this is why I made a distinction of sorts in the items from Gordon's list. However, IEEE 1588 == network time sync, don't forget. Without hardware packet timestamping (3,4) you cannot have a working PTP stack.

[tschiemer]

Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...

Spontaneous ideas... use a (HW or peripheral) clock for comparison? Observe SYNC_OUT on some external component use SYNC_IN to generate a timestamp and compare values?.... if it's an on-PHY clock, you just have to see how it runs, I guess?

Ja, I'm aware, just don't know what magic the PHY is capable of by itself.. thinking bout it, it likely won't have a PTP stack of it's own.. :/

[wowczarek]

@ghollingworth

If you've got a suggestion for an Ethernet device that does actually implement these functions from the standard set of drivers:

I think the closest are Intel i2xx/i3xx etc. where there are the two SDIO pins that can be ordered to be event inputs / outputs. The Linux PTP/PHC API has a way to grab the input event timestamps I would prefer a /dev/ppsX device, but that's just me.

Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...

Yes, ptp4l for the h/w timestamping and PTP Hardware Clock. I mean really what the "PTP hardware clock" is, "merely" two things, plus some initialisation logic:

1. A counter + register that counts either arbitrary ticks, with maybe another register to hold time of day or a lower/higher register pair for precise and rough time - or like Intel did it, a register/counter that ticks in human time. I am expecting the former. Plus some code to handle rollovers, etc.
2. Another register that controls the counter increment - this is how you steer the clock.

The timestamps captured are obviously latched counter values with a FIFO. So the timestamping driver part will pick these up, associate with frames/packets and attach these as out of band mesages (SCM) to the socket kernel buffer.

So the PHC driver provides an interface to this, allowing us to read time, set time, and steer the clock. If you use ptp4l as a PTP slave ('slave' term is under debate these days), you will utilise h/w timestamps to compute offset and control the hardware clock, this is why timestamps and PHC are one item.

[wowczarek]

@tschiemer

Ja, I'm aware, just don't know what magic the PHY is capable of by itself.. thinking bout it, it likely won't have a PTP stack of it's own.. :/

It won't. The PHY only timestamps packets, that is it, you need a software stack like ptp4l.

[wowczarek]

1. The hardware clock interface or PTP Hardware Clock / PHC, in Linux kernel terms, is only a standard interface for reading, writing and steering increment of the time register(s) on the chip and converting to human time. In hardware terms it's a register or a few, and that is that. This is not the same as the in/out pins; these are an extra.

2. "IEEE 1588 support" means some time register as above, and hardware timestamping, which means only one thing: timestamping packets/frames. Either the ability to timestamp every packet, which Intel, Solarflare and some others support, or, more commonly, only timestamp PTP packets, so the hardware matches a PTP packet using either the multicast group destination MAC address or PTP Ethernet multicast address or link-local variants of those (right at the beginning of frame), or if it's more sophisticated, UDP port 319, which also allows for unicast IP PTP timestamping. Hardware matches these, latches the timestamp counter and pushes that up a FIFO, long story short, they end up being human time attached to packets arriving at the socket. Obviously I don't know this chip and what's in the magic document, so I don't know which of the above matching options it does. There are also things like VLAN tag matching. What I do know is Broadcom's switching ASICs, and their PTP support is solid and extensive.

3. The event input pin should generate a timestamp aligned with the PTP clock on input rise transition (well, typically rise, for most timing applications). I hope it will be that simple. TI's AM335x network subsystem for example does it differently - event input has its own counter, but luckily it can be correlated with the PTP counter on rollover - more work, but it works. The problem with the event input and testing it, is that because it's meant to be handled in the PTP API, it's mostly ignored by people with GNSS receivers and there is near-zero software support for it. Really it should be a /dev/ppsX device.

4. The event output pin should allow, at minimum and by default, a rising pulse of certain width at the top of every second, of a certain width - make it 100 ms to keep all sync applications happy.

Testing 3. and 4, at least preliminary, can be done with two boards back to back, but ideally you need a time interval counter / frequency counter. Time nuts around the world will handle the rest. Two boards, Ethernet back to back, crossed over event pins, one is a PTP master, the other a PTP slave (ptp4l), PTP syncs them, and the inputs will show the clock difference between the two. That's the minimal DUT/SUT setup.

[marcohabeck]

Unfortunately, I was able to find out in December. I put an i210 in the PCIe slot and built a kernel. The PHY clock is synchronized and so is the board. Unfortunately the clock of the CM4 is not good enough and the whole thing runs away after hours and the clock can no longer be synchronized with the PHY .

[wowczarek]

@marcohabeck this is off-topic - but if you mean it runs away after being synchronised once and then leaving it, then welcome to the real world. In a PC or server it would be no different. This hardware can timestamp, but it can't keep time, that's just the way it is. Poor oscillators. Timing cards with oscillators good enough to hold better than hundreds of microseconds per day and few and far between. Oregano/Meinberg and high-end Exablaze are the only ones I know can do decent holdover. Solarflare right behind them. These systems have to be kept actively synchronised all the time, they drift very quickly.

But technically PTP NICs can be steered much more aggressively than the OS clock (I assume this is what you mean by 'clock' vs. PHY), so the CM4 or drivers or software would have had to be severely broken if the OS clock couldn't keep up with the NIC. OS clock control is limited to +/- 512ppm (~half a millisecond per second) adjustment, which is huge. People would have noticed drift of this magnitude. You'd have to check your software stack. Was the PHY (MAC really) synchronised to PTP, or were you just synchronizing the OS clock to the NIC without PTP?

[neggles]

I'm quite happy to help out with hardware-side testing on this stuff; the PHY hardware timestamping support is most of the reason I picked up a CM4+CM4IO board, and it would be absolutely killer for something like a next-gen LimeNET Micro (and for time sync applications in general)

I've got a number of PTP-equipped devices and 1PPS sources - some commercial, some homemade - available for testing, though I doubt you'd have much trouble finding plenty of people who're interested!

[cfdez-tech]

Hi @ghollingworth, I've been working on the ptp 1588 driver for the bcm54210pe, at the moment I've got the full skeleton implemented and the kernel can check configuration available for the PTP and PHC:

root@rpi-cm4:~# ethtool -T eth0
Time sta[   64.191025] DEBUG: Passed bcm54210pe_ts_info 258 
mping parameters for eth0:
Capabilities:
        hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
        hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
        hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
        off                   (HWTSTAMP_TX_OFF)
        on                    (HWTSTAMP_TX_ON)
        one-step-sync         (HWTSTAMP_TX_ONESTEP_SYNC)
Hardware Receive Filter Modes: none

Also I can get and set the PHC using phc_ctl tool:

root@rpi-cm4:~# date
Fri Mar  9 13:03:13 UTC 2018

root@rpi-cm4:~# phc_ctl /dev/ptp0 get
[  174.782856] DEBUG: Passed ptp_clock_gettime 109
[  174.787636] DEBUG: Passed ptp_clock_gettime 114
[  174.792269] DEBUG: bcm54210pe_gettime 176
[  174.796722] DEBUG: Get time 4386 13124 21862 0 0
[  174.801471] DEBUG: Current PCH time 860116326.0
[  174.806145] DEBUG: Passed ptp_clock_adjtime 126
[  174.810797] DEBUG: Passed ptp_clock_gettime 109
[  174.815454] DEBUG: Passed ptp_clock_gettime 114
[  174.820073] DEBUG: bcm54210pe_gettime 176
[  174.824450] DEBUG: Get time 4386 13124 21862 0 0
[  174.829163] DEBUG: Current PCH time 860116326.0
phc_ctl[174.832]: clock time is 1234549248713053542.000000000 or (null)
root@rpi-cm4:~# phc_ctl /dev/ptp0 set
[  176.947297] DEBUG: Passed ptp_clock_gettime 109
[  176.951944] DEBUG: Passed ptp_clock_gettime 114
[  176.956614] DEBUG: bcm54210pe_gettime 176
[  176.961005] DEBUG: Get time 4386 13124 21862 0 0
[  176.965790] DEBUG: Current PCH time 860116326.0
[  176.970450] DEBUG: Passed ptp_clock_adjtime 126
[  176.975122] DEBUG: Passed ptp_clock_settime 98
[  176.979653] DEBUG: Original time 1520600486 102643396
[  176.984846] DEBUG: Set time 0 23202 34214 1552 14020
phc_ctl[176.988]: set clock time to 1520600486.102643396 or Fri Mar  9 13:01:26 2018

root@rpi-cm4:~# phc_ctl /dev/ptp0 get

[  179.635848] DEBUG: Passed ptp_clock_gettime 109
[  179.640502] DEBUG: Passed ptp_clock_gettime 114
[  179.645127] DEBUG: bcm54210pe_gettime 176
[  179.649521] DEBUG: Get time 0 23202 34214 1552 14020
[  179.654601] DEBUG: Current PCH time 1520600486.101725892
[  179.660022] DEBUG: Passed ptp_clock_adjtime 126
[  179.664650] DEBUG: Passed ptp_clock_gettime 109
[  179.669267] DEBUG: Passed ptp_clock_gettime 114
[  179.673883] DEBUG: bcm54210pe_gettime 176
[  179.678254] DEBUG: Get time 0 23202 34214 1552 14020
[  179.683331] DEBUG: Current PCH time 1520600486.101725892
phc_ctl[179.687]: clock time is 1520600486.101725892 or Fri Mar  9 13:01:26 2018

The current blocking point is that, despite reading and setting the PHC using phc_ctl set/get tool, this counter is not increasing its value, so it's not counting. Has any of the contributors in here got a contact in Broadcom in order to know whats happening with the register configuration? I've checked several times and I think it's correct. I cannot get any more ideas from the datasheet.

In parallel, I'm currently developing the time stamping routines in order to get the info from the Broadcom internal FIFO. I think I will have the first draft maybe during next week.

I'll be glad to contribute to this issue. Maybe we can create an branch in order to start collaborating here.

[ghollingworth]

Can you share your code, so I can see your implementation?

[ghollingworth]

I have access to support from Broadcom and have been using it to get the PHY setup correctly, currently I've got it set up timestamping packets and working correctly, but am also having trouble reading back the current timer counter. Whick register set are you using? The original time stamp or the heartbeat time stamp registers?

Gordon

[cfdez-tech]

I'm currently using the Original Time register, I can read and write in it, but it's not counting.

Mi register setup is the following one (extracted from Broadcom .vbs scripts and the other is added from my side in order to configure the SYNC_OUT)

REGISTER, VALUE
0x0F5, 0x0001
0xA10, 0x0101
0xAF8, 0x0101
0xAC0, 0x0F0F
0xAC1, 0x0F0F
0xAC3, 0x0101
0xA21, 0x0001
0xA22, 0x0000
0xA54, 0x0000
0xA55, 0x0000
0xA56, 0x5566
0xA57, 0x3344
0xA58, 0x1122
0xA5C, 0x07C0
0xA5D, 0x07C0
0x82D, 0x407E
0xA5E, 0x0002
0xA5B, 0x0160
0xA11, 0x00FF
0xA19, 0x0000
0xA78, 0x8000
0xA79, 0x400
0xA7A, 0x240
0xA7F, 0xD012 => In order to work with internal sync and SYNC_OUT pulses. We've got nothing connected on SYNC_IN pin.

[cfdez-tech]

Please @ghollingworth tell me how can I send you my code as soon as I've got green light to share it. Thanks, Carlos

[ghollingworth]

email me gordon@raspberrypi.com

[marcohabeck]

The problem isn't just timestamping. You also need FQTSS support. Running away from the clock as described in the post above happens when the clock master changes. Because the clock runs free and can no longer be caught. Therefore it is important to have a stable system clock hardware. This is the only way to minimize the deviation when changing and the clock can be caught again. You also need a synchronized system clock, because when writing to buffers or hardware it is important to avoid overflows or underflows. Even if you try very hard, the hardware will not meet these requirements.

Am 10.03.2021 um 10:09 schrieb cfdez-tech notifications@github.com:

Hi @ghollingworth https://github.com/ghollingworth, I've been working on the ptp 1588 driver for the bcm54210pe, at the moment I've got the full skeleton implemented and the kernel can check configuration available for the PTP and PHC:

root@rpi-cm4:~# ethtool -T eth0 Time sta[ 64.191025] DEBUG: Passed bcm54210pe_ts_info 258 mping parameters for eth0: Capabilities: hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE) hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE) hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE) PTP Hardware Clock: 0 Hardware Transmit Timestamp Modes: off (HWTSTAMP_TX_OFF) on (HWTSTAMP_TX_ON) one-step-sync (HWTSTAMP_TX_ONESTEP_SYNC) Hardware Receive Filter Modes: none

Also I can get and set the PHC using phc_ctl tool:

root@rpi-cm4:~# date Fri Mar 9 13:03:13 UTC 2018

root@rpi-cm4:~# phc_ctl /dev/ptp0 get [ 174.782856] DEBUG: Passed ptp_clock_gettime 109 [ 174.787636] DEBUG: Passed ptp_clock_gettime 114 [ 174.792269] DEBUG: bcm54210pe_gettime 176 [ 174.796722] DEBUG: Get time 4386 13124 21862 0 0 [ 174.801471] DEBUG: Current PCH time 860116326.0 [ 174.806145] DEBUG: Passed ptp_clock_adjtime 126 [ 174.810797] DEBUG: Passed ptp_clock_gettime 109 [ 174.815454] DEBUG: Passed ptp_clock_gettime 114 [ 174.820073] DEBUG: bcm54210pe_gettime 176 [ 174.824450] DEBUG: Get time 4386 13124 21862 0 0 [ 174.829163] DEBUG: Current PCH time 860116326.0 phc_ctl[174.832]: clock time is 1234549248713053542.000000000 or (null) root@rpi-cm4:~# phc_ctl /dev/ptp0 set [ 176.947297] DEBUG: Passed ptp_clock_gettime 109 [ 176.951944] DEBUG: Passed ptp_clock_gettime 114 [ 176.956614] DEBUG: bcm54210pe_gettime 176 [ 176.961005] DEBUG: Get time 4386 13124 21862 0 0 [ 176.965790] DEBUG: Current PCH time 860116326.0 [ 176.970450] DEBUG: Passed ptp_clock_adjtime 126 [ 176.975122] DEBUG: Passed ptp_clock_settime 98 [ 176.979653] DEBUG: Original time 1520600486 102643396 [ 176.984846] DEBUG: Set time 0 23202 34214 1552 14020 phc_ctl[176.988]: set clock time to 1520600486.102643396 or Fri Mar 9 13:01:26 2018

root@rpi-cm4:~# phc_ctl /dev/ptp0 get

[ 179.635848] DEBUG: Passed ptp_clock_gettime 109 [ 179.640502] DEBUG: Passed ptp_clock_gettime 114 [ 179.645127] DEBUG: bcm54210pe_gettime 176 [ 179.649521] DEBUG: Get time 0 23202 34214 1552 14020 [ 179.654601] DEBUG: Current PCH time 1520600486.101725892 [ 179.660022] DEBUG: Passed ptp_clock_adjtime 126 [ 179.664650] DEBUG: Passed ptp_clock_gettime 109 [ 179.669267] DEBUG: Passed ptp_clock_gettime 114 [ 179.673883] DEBUG: bcm54210pe_gettime 176 [ 179.678254] DEBUG: Get time 0 23202 34214 1552 14020 [ 179.683331] DEBUG: Current PCH time 1520600486.101725892 phc_ctl[179.687]: clock time is 1520600486.101725892 or Fri Mar 9 13:01:26 2018

The current blocking point is that, despite reading and setting the PHC using phc_ctl set/get tool, this counter is not increasing its value, so it's not counting. Has any of the contributors in here got a contact in Broadcom in order to know whats happening with the register configuration? I've checked several times and I think it's correct. I cannot get any more ideas from the datasheet.

In parallel, I'm currently developing the time stamping routines in order to get the info from the Broadcom internal FIFO. I think I will have the first draft maybe during next week.

I'll be glad to contribute to this issue. Maybe we can create an branch in order to start collaborating here.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/raspberrypi/linux/issues/4151#issuecomment-795123362, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYOHWPQN75QC7K4U5XUMGTTC4ZKZANCNFSM4XWOSD6Q.

[wowczarek]

@marcohabeck this thread is about delivering PTP drivers for the onboard PHY in the CM4.

[wowczarek]

@ghollingworth, @cfdez-tech if you don't mind me asking, since you have access to the documents - from the ETHTOOL_TS_INFO output above I only see one timestamping mode for RX and TX - is this because this is only the first cut of the implementation, or does it actually support timestamping all packets?

[ghollingworth]

It only supports the timestamping of IEEE1588 Sync and Delay Request packets (on both TX and RX) (also supports peer delay request and response)

[cfdez-tech]

@wowczarek The reference to TS mode for RX and TX it's just the first draft of my implementation (It's not even working yet). As @ghollingworth said, it supports IEEE1588 Sync, Delay and response.

[wowczarek]

@ghollingworth @cfdez-tech thank you both.

[ghollingworth]

So far I have the following functionality working:

1) TX and RX timestamping of Sync and Delay Request packets 2) Reading and writing the time counter 3) Interrupts

So I've pretty much got everything required to at least get the simple PTP implementation working. Just trying to wire up interrupts at the moment and then will need to implement the hwtstamp functions etc to actually apply the timestamps to the packets. From there it should be working to generate / recreate the master clock.

Turns out the best use of my time was to implement a debugfs interface to the registers so I could read / write them from the command line! From there I was able to reverse engineer the missing documentation (i.e. the description of actually how you use the registers to do things!)

[wowczarek]

Great news - the key item for slave / sync will be a) the PTP_SYS_OFFSET ioctl and then b) frequency adjustment.

For the latter - do we know if the counter is always incrementing by 1 i.e. you need to write an abstraction for a multiplier to control the apparent time delta per tick, or is there an actual register for adjusting the increment?

Also how is the timestamp register organised and expressed, is it just a tick counter or is there an actual seconds/nanoseconds representation?

My guess is a tick counter with no adjustment facility, but my hope is for Intel-like, proper seconds/nanoseconds with adjustment.

[ghollingworth]

The counter is incremented in ps (the standard setting is 8000), and is incremented every tick of the 125MHz oscillator. This can be updated to increase/decrease the frequency. We can also update the counter with an offset. The counters are 80bit and are stored as 32bit seconds 32bit nanoseconds

[wowczarek]

Thanks - this is perfect. Makes implementation easier without tracking rollovers and those dreadful cycle counters :) Offset adjustment is good, this is a straight plug for ADJ_SETOFFSET so another win.

Two more questions or five and I'll shut up for now and order some CM4s instead ;)

1. Timestamper matchers / filters, do you know if they match on PTP payload (message type) only, or also/instead on UDP port and/or multicast MAC? Trying to establish if this thing can timestamp IP/unicast PTP and any encapsulations, namely dot1q. Also what about IPv6?
2. What is captured in the TS FIFO entry apart from the timestamp? I assume message type and seq id, but what else? Is VLAN number or PTP domain number also captured?

Thanks!

[marcohabeck]

@wowczarek but to be able to use the PHY sensibly, it takes a little more. EE 802.1Qav traffic shaper VLAN support, and several queues. The driver and of course the hardware must support this. In order to be able to implement real-time capable fieldbuses at all. The board is supposed to be intended for the industry, isn't it? I am honest and say that at the price it is questionable to develop all of this without help. That everything is under NDA makes it very difficult. There are alternatives ...

Am 11.03.2021 um 01:40 schrieb Wojciech Owczarek @.***>:

@marcohabeck https://github.com/marcohabeck this thread is about delivering PTP drivers for the onboard PHY in the CM4.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/raspberrypi/linux/issues/4151#issuecomment-796326767, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYOHWM3JBTNTV3WAEW4FXLTDAGOXANCNFSM4XWOSD6Q.

[cfdez-tech]

Great news @ghollingworth!

[neggles]

@marcohabeck: but to be able to use the PHY sensibly, it takes a little more. EE 802.1Qav traffic shaper VLAN support, and several queues. The driver and of course the hardware must support this. In order to be able to implement real-time capable fieldbuses at all. The board is supposed to be intended for the industry, isn't it? I am honest and say that at the price it is questionable to develop all of this without help. That everything is under NDA makes it very difficult. There are alternatives ...

I am struggling to understand the point you are actually trying to make here. Yes, PTPv2 support alone is not enough to add support for EtherCAT, Ethernet AVB and other similar real-time ethernet-based protocols - but they all require PTPv2 be implemented first. There are plenty of uses for PTPv2 outside of synchronous networking, too - so the obvious first step would be to add driver support for the PHY PTP clock/timestamping hardware, which is what this thread is about.

Yes, there's an NDA on some of the hardware documentation, but there are also plenty of people with access to that NDA'd documentation & it's not always needed to figure it out anyway. Many hands make light work, and the low cost of raspberry pi produced devices attracts a lot of people!

Getting back on topic, excellent news indeed! Hopefully there's support for timestamping unicast PTP (and NTP!) packets, since I only have one switch (with a mere four ports) that supports layer-2 PTP...

[rolandash]

@marcohabeck: but to be able to use the PHY sensibly, it takes a little more. EE 802.1Qav traffic shaper VLAN support, and several queues. The driver and of course the hardware must support this. In order to be able to implement real-time capable fieldbuses at all. The board is supposed to be intended for the industry, isn't it? I am honest and say that at the price it is questionable to develop all of this without help. That everything is under NDA makes it very difficult. There are alternatives ...

I am struggling to understand the point you are actually trying to make here. Yes, PTPv2 support alone is not enough to add support for EtherCAT, Ethernet AVB and other similar real-time ethernet-based protocols - but they all require PTPv2 be implemented first. There are plenty of uses for PTPv2 outside of synchronous networking, too - so the obvious first step would be to add driver support for the PHY PTP clock/timestamping hardware, which is what this thread is about.

Yes, there's an NDA on some of the hardware documentation, but there are also plenty of people with access to that NDA'd documentation & it's not always needed to figure it out anyway. Many hands make light work, and the low cost of raspberry pi produced devices attracts a lot of people!

Getting back on topic, excellent news indeed! Hopefully there's support for timestamping unicast PTP (and NTP!) packets, since I only have one switch (with a mere four ports) that supports layer-2 PTP...

I am just curious about what kind switch you got that supports layer-2 PTP? Would you able to share some info about it, like witch brand, model, the chip used inside, etc... I am trying hard to find a decent one of SMB grade.

It seems that Broadcom has its own time sync tech (BroadSync HD), which claims to be 802.1as compliant. But 802.1as is not fully compliant with 1588v2, so this get things even more complex.

thanks!

[neggles]

@rolandash: I am just curious about what kind switch you got that supports layer-2 PTP? Would you able to share some info about it, like witch brand, model, the chip used inside, etc... I am trying hard to find a decent one of SMB grade.

This may be a bit off-topic, but could be useful if someone needs to do interoperability testing with the CM4 and a PTP Transparent Clock. You can also make a PTP Transparent Clock out of a PC with several Intel (or other PTP hardware clock equipped) NICs, using ptp4l, though I've not tried that myself.

It's an MRV OptiSwitch OS904 - not consumer grade at all, but very cheap on the used market. There are quite a few similar devices, originally used by ISPs to terminate customer connections, which have PTP TC support such as the ADVA FSP150CC series (support varies by model). I am not sure what chipset it uses, but it turns out the model I have might not actually support PTP at all - this seems limited to the OS904-MBH, which I do not have.

On that note, you do have to be careful with what you buy - it's usually possible to factory reset / clear configuration from these devices without too much trouble, but not always, and it's often hard to find manuals, so do your research beforehand.

@rolandash: It seems that Broadcom has its own time sync tech (BroadSync HD), which claims to be 802.1as compliant. But 802.1as is not fully compliant with 1588v2, so this get things even more complex.

thanks!

802.1as (TSN), Ethernet AVB, SyncE etc. do add complexity here, but they all make use of the same hardware APIs as PTP - they just make some behavioural tweaks and use different default message rates, and then layer extra things on top. ptp4l supports running a master clock in most of these different sync modes, AFAIK.

[wowczarek]

@neg2led: This may be a bit off-topic, but could be useful if someone needs to do interoperability testing with the CM4 and a PTP Transparent Clock.

As to interop, this is usually pretty simple... With the emphasis on usually - but results tend to be binary. The CM4 itself only provides RX and TX timestamps, the rest is down to software, as in transmitting the PTP messages is done with regular sockets. The TC then matches, pairs up and corrects those, so there is little effect the CM4 will have here. Not so much down to interop but rather feature support, things like:

• What encapsulations, if any, the TC will correct? None, 802.1Q, QinQ? Will it correct frames sprayed over link aggregation? Etc.
• Does the TC support PTP over IPv4/v6/Ethernet and then multicast only or also unicast? And then L2 or also with L3 forwarding / routing?
• Is the TC one-step (on the fly corrections for Sync and Delay Request) or two-step (measuring sync and placing corrections in the corresponding follow-up, same with delay req + resp), and then if it's two-step, how does it deal with one-step PTP - like IEEE 1588 says, that is turning it into two-step, or like some vendors, not working with one-step period.

The end node will have little effect on this. The only "real" interop issues I've seen were with FPGAs or otherwise reprogrammable NIC that do much more than PTP. Things like padding frames with dummy trailers - either intentionally (there are some clever tricks to influence queuing on the way), or unintentionally. Some TCs used to trip over this.

@neg2led: ADVA FSP150CC series (support varies by model). I am not sure what chipset it uses, OS904-MBH

Good finds! Need to hit up my Adva and MRV reps at work for some more details. List prices for these things tend to be so obscene for a personal purchase that one tends to forget about eBay and that this kit surfaces through those channels eventually as it gets refreshed/decommed in production. Yes, mobile backhaul / metro type switches tend to have TC support. My current pick and the only product line I'm more or less up to date with "street" prices is Juniper ACX - there are quite a few models, most of them available on eBay and most have at least TC support, some even have a GM with a GPS receiver - although the lower-end Gigabit models easily go into £k / $k, there's the occasional "I'm selling this, don't know what it is" bargain. Junipers are typically Broadcom-based and at least for TC operation, having used them in production extensively, I can say that they are good. Far from the first "half-software" TCs (old Hirschmanns and whatnot) that were two-step with very long processing latencies for PTP. They corrected them fine, but Broadcoms do it all in hardware and can pretty much hit line rate of PTP - not that anyone would ever see this oudside of a lab.

[maxlem]

Just found out this thread. Am-I reading it correctly if I conclude that hooking a gps pps signal to the sync in pin we could (once the software us sorted out) make a ptp GM out of an rpi4 + gps?

That would be fantastic!

I used ptp a lot at my previous job to synchronize sensors, such as flir blakfly s cameras. What we ended up doing, to validate the whole stack, was to flip a pin once per second on the synchronized device (from an rtos) and measure the delay with a good scope. Clunky I know but we only needed sub 500us precision.

I guess with the sync out pin on another rpi4 (or maybe an intel nic), you could at least use a scope and a gps to check both are in sync (skipping the switch)

Anyway, just wanted to cheer you up!

[pelwell]

This thread is specifically about the Ethernet PHY on the CM4, which is not the same as that on the regular Pi 4B.

[maxlem]

Oh! Thanks for pointing out, I assumed they shared hardware specs, anyway, it's still great

[maxlem]

I did check out, and the RPi4 uses the BCM54213PE phy It's not listed here: https://www.broadcom.com/products/ethernet-connectivity/phy-and-poe/optical/bcm5421xe It's the same family of chips, I wonder if work done here couldn't at least enable HARDWARE timestamping