CM4 is missing IEEE1588-2008 support through BCM54210PE

[tschiemer]

Is this the right place for my bug report? Yes: I asked on the forum, I asked the raspberry foundation through the contact form but didn't get any response - it is very much kernel related as is explained in the forum post.

Describe the bug The CM4 has an undocumented inofficial PHY (BCM54210PE) that supports hardware timestamping (and a hardware clock, actually??), but there seems to be no driver supporting these features.

So CM4 does actually not provide hardware based IEEE1588-2008 support as opposed to what the raspberry pi foundation actually communicates in the CM4 datasheet.

To reproduce sudo ethtool -T eth0 only shows software timestamping capabilities.

Expected behaviour Depends - the raspberry pi foundation never communicated what IEEE1588-2008 features are supported such that is is unclear what exactly can be expected; also the PHY documentation is not available such that possibilities (ie optimal expected behaviour) can not be described.

Expected (according to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-using_ptp):

• eth support for:
  • SOF_TIMESTAMPING_TX_HARDWARE
  • SOF_TIMESTAMPING_RX_HARDWARE
  • (possibly) SOF_TIMESTAMPING_RAW_HARDWARE
• shows PTP Hardware Clock (because BCM54210PE has on-chip clock)
• Hardware Transmit Timestamp Modes ??
• Hardware Receive Filter Modes ??

The CM4 datasheet lists SYNC_IN, SYNC_OUT pins ment for IEEE1588-2008 support but has not further specified this. It actually is a question wether these pins are needed in the first place - because the PHY would seem to have a hardware-clock itself. Further it must be clear wether the PHY's internal clock (if any) is routed to a CM4 pin in any form and if not this should be fixed in future revisions because custom boards for the CM4 might require tight synchronization to a hardware clock.

Actual behaviour sudo ethtool -T eth0 only shows software timestamping capabilities. There is no (documented) synchronizable IEEE1588-2008 clock signal going from the CM4.

System Raspberry Pi Compute Module 4

Raspberry Pi OS Lite, January 11th 2021, Kernel version: 5.4 Ubuntu Server 20.04.2 LTS,

Logs none

Additional context BCM54210PE documentation is required to implement to see actual features provided and configuration options - IEEE802.3-2018 seems to define said capabilities etc. Possibly also required are CM4 schematics and BCM2711 documentation. Also see IEEE802.3-2018 Section 90. Ethernet support for time synchronization protocols

I can try to take care of the implementation, but I would like to have the mentioned documents.

[KyleJudd]

I'm reconfiguring my network now to implement IP4/IP6 timestamping. The plan is to use a desktop PC with an i210 and ptp4l as the master so I can listen for packets on the CM4.

What profiles do people want to see working? ptp4l contains a folder of cfg files for me to pick from - what is the most common profile/cfg that people use? Or if it is best to specify parameters on the command line, can someone please suggest basic settings that people use for IP4/IP6?

[mlichvar]

From the hardware and driver point of view, I think it's only the transport that really matters: L2, multicast IPv4/IPv6, and unicast IPv4/IPv6.

For the L2 and multicast IPv4/6 transports you don't need any configuration files. You can run them as:

ptp4l -m -q -i eth0 -2
ptp4l -m -q -i eth0 -4
ptp4l -m -q -i eth0 -6

on both computers. Add -s to the one that you want to be the client if you don't want them to select it automatically by the MAC address.

For unicast you can use the example UNICAST-MASTER.cfg and UNICAST-SLAVE.cfg config files modified for your network.

[jdimpson]

What profiles do people want to see working? ptp4l contains a folder of cfg files for me to pick from - what is the most common profile/cfg that people use? Or if it is best to specify parameters on the command line, can someone please suggest basic settings that people use for IP4/IP6?

As I understand it, the gPTP configuration file puts ptp4l into conformance with the IEEE 802.1AS profile. 802.1AS is the profile required for other TSN (and AVB?) functions. (Although I believe many implementations are more flexible.)

[KyleJudd]

I did get the IP4/IP6 stuff working ... as well as Layer 2 at least. I will probably post the code tomorrow evening (EST) with instructions on how to set things up. As mentioned before, there are still issues with steering the clock (AdjFine) that we are working to resolve.

[KyleJudd]

leragequit - The process you mentioned is correct, but the answer is “not yet”

1. The most recent changes to the code aren’t posted yet, that will take another day or two. When I do, I will try to update the “readme.txt” with step-by-step instructions.
2. Timestamping does work, but there are still some issues with clock steering that we are trying to work out. That should be resolved sooner than later, but I can’t say when.
3. You should be able to use IEEE1588v2, but v2.1 messages aren’t currently timestamped by the hardware. Much of the existing ptp apps just send out 2.0 messages, so you might have to find a way to reconfigure your software to use 2.0 only. Again, the hardware issue may or may not be resolved soon, but I can’t say when.

We could definitely use your help testing and confirming that things work, once these issues above are resolved.

Thanks, Kyle Judd

From: leragequit

Hello there, I have been lurking for years and I am very happy to see so much progress.

Am I right with the assumption that I can just clone and compile git clone -b cm4-ptp-kylejudd https://github.com/KyleJudd/linux.git , replace my rpi kernel and I should be able to use IEEE1588v2?

I am currently thinking on how to confirm the accuracy, would two rpi based stratum 1 clocks do the trick?

Best regards,

Sascha

[pason234]

sorry for asking, I am wonder that is it possible to CM4 perform wireless PTP.

[ahmadexp]

sorry for asking, I am wonder that is it possible to CM4 perform wireless PTP.

Short answer is no. Reason for that is, PTP is based on Hardware Time Stamping. The eth on the CM4 is the first Rpi capable to do that and therefore it is capable to to PTP. There is no wifi controller in the market yet (let alone, being on the Pi) that is capable to do hardware time stamping, aside from other RF related issues. We have been investigating the usage of UWB to do this which you can find the slide about it (Getting Precision Time Synchronization to End Users) on https://www.opencompute.org/wiki/Time_Appliances_Project

[eric957]

Hi all, So the question for Broadcom is to open or provide access to the documentation of bcm54210pe, is that correct ? I may have an opportunity to ask them shortly. Thx. Eric

[lasselj]

Yes. That is correct.

-- Best regards,

Lasse L. Johnsen

On 15 Feb 2022, at 12:15, Eric Dujardin @.***> wrote:

 Hi all, So the question for Broadcom is to open or provide access to the documentation of bcm54210pe, is that correct ? I may have an opportunity to ask them shortly. Thx. Eric

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[KyleJudd]

That would be a big help.

[KyleJudd]

I you do get a chance to talk to them, please let them know that the driver is mostly working, we just need to check a few details about registers and then we will be finished. They might say no if they think this is a new project and they will have to answer a lot of questions and hold our hands, etc.

[ghollingworth]

I'm currently discussing with Broadcom whether someone from this list would be able to access their documentation under NDA.

[KyleJudd]

I just posted new code.

The driver now does IP4/IP6 timestamping, but issues still remain that prevent it from working in ptp apps like ptp4l and others. Please be patient with that. As a reminder, the purpose of my repository is to share code with other developers that may be able to help finish the project. The driver is not ready for regular use and should only be downloaded by those who intend to debug it.

Changes: 1) Implemented IP4/IP6 timestamping 2) Refactored buffers used for timestamping to make them more generic. 3) Refactored code to help me debug 4) Added README page with some "How To Build" instructions.

To see the README page in GitHub, make sure you switch to the cm4-ptp-kylejudd in the top left corner. I haven't tested the instructions yet, but they should be accurate enough for people use.

[cogleym]

I stumbled across this forum and was a little confused.

The driver you are working on does what exactly with the time stamping function on the nic phy.

Such as does the driver sync time between the phy and the host cpu? or does it allow syncing time over its sync in input?

I would think the sync input pin would provide better accuracy.

[wowczarek]

@cogleym the driver implements the Linux PHC API, allowing to run hardware-assisted PTP (IEEE 1588), and no, the driver does not sync time, software like ptp4l syncs time, this driver only allows a PTP stack to use h/w timestamping.

Re. Sync input pin: are you questioning the point of running PTP at all? IEEE 1588 was invented precisely so that you don't have to run sync cables to every device that is already on a network. Otherwise yes, a 1PPS input will likely provide the best precision (accuracy will be as good as your cable delay calibration), but this is beside the point. The CM4 spec sheet advertises PTP support and this work is to turn this statement into reality.

[cogleym]

I didn't really understand the standard but it makes sense to reduce all the head ache as much as possible. I wonder why they added the physical pins. Just optional I guess.

[lasselj]

I will add that following exchange of emails yesterday and calls earlier in the week, Kyle and I are pausing work for a week or so in the hopes that Broadcom will agree to release documentation required to continue the work. Until such a time where we understand what registers are required to implement the adjfine function, there just isn’t much point in continuing. On the upside most other issues are resolved and other residual issues outstanding seem entirely manageable.

[neggles]

Something that has been concerning me; the driver code Kyle et al. have been working on seems to be effectively re-implementing the kernel's software timestamping of packets, except it's using the PHC as the clock source for the timestamps?

That is not at all how this is supposed to work; when the PHY is configured appropriately, the packets are automatically/transparently timestamped by the PHY as the packet passes through it - that's the entire point of PHY timestamping...

Having spent quite a bit of time lately digging through the sources and reading docs to understand how this all plugs together, the majority of the code to enable/make use of HW timestamping & the PTP hardware clock on a Broadcom PHY already exists in the kernel Broadcom PHY driver code, it's just missing the model-specific register definitions/callbacks used to enable/disable TX/RX timestamping, adjust the clock frequency, configure 1PPS in/out, etc...

I suspect there is not a huge amount of code that actually needs to be written to make this work, but it's all dependent on this particular PHY sub-model's register map.

[Edit]: The above was written in response to someone else's comment, which I had misread, and following the replies below I have looked a bit closer and I was mistaken. Apologies. Something still doesn't seem quite right, though.

I didn't really understand the standard but it makes sense to reduce all the head ache as much as possible. I wonder why they added the physical pins. Just optional I guess.

The sync input pin is used for a 1PPS (pulse-per-second) input from a reference/master clock (e.g. a GPS module or atomic clock) - the PHY provides the timestamp of each pulse & the system can use those timestamps to calibrate the PHC (PTP Hardware Clock) inside the PHY to match the reference input. This is used when you want the device in question to be a master PTP clock for the network (as well as a few other modes)

Likewise, the sync output can be used to generate a 1PPS signal to feed into other equipment - if you have a system which is synchronized to a PTP grandmaster over the network, the accuracy of the 1PPS output from that system's PHC will be nearly identical to the accuracy of the input feeding the grandmaster.

[lasselj]

Hi Andrew,

Something that has been concerning me; the driver code Kyle et al. have been working on seems to be effectively re-implementing the kernel's software timestamping of packets, except it's using the PHC as the clock source for the timestamps?

No, you have that wrong.

That is not at all how this is supposed to work; when the PHY is configured appropriately, the packets are automatically/transparently timestamped by the PHY as the packet passes through it - that's the entire point of PHY timestamping...

The timestamps are from the PHY I assure you. As the rest of your message suppose they are not, I will stop the response here.

All the best,

Lasse

[ghollingworth]

I think it's actually a lot simpler than that, depending on how you implement it. To explain a little about how I understand the BRCM phy implementation.

In the PHY there is a hardware time counter that counts at 125MHz, around that counter is a PID loop that monitors incoming timestamps on PTPv1 or v2 packets and adjusts the time counter automatically to track an external time source. There is no need for anything in Linux to make this time source work or correct it. Other than running an application like ptp4l which sends and receives the correct packets such that the PHY can read/write the timestamps.

What ptp4l, therefore, needs to do is just read the current time from the hardware and set its clock based on that (or just use the timestamps on incoming packets to do that). SYNC input and output should work correctly, but as previously mentioned SYNC input is only useful when the unit is the master clock and you want to synchronize with an external time source (like a GPS source).

I'm still in discussion with BRCM concerning the best way forward, but it's likely to be contracting someone to do the work for us and therefore get a Raspberry Pi email address etc.

So any contracting engineers who can show experience with IEEE1588 time sync (internal and external) and have the hardware to hand to test it. Or at least an engineer who can convince me that they have the knowledge and experience to go and learn enough to write and support this code... Please email me... 'gordon' 'curley a symbol' raspberrypi fullstop 'shortened version of company'

[lasselj]

Hi Gordon,

I think it's actually a lot simpler than that, depending on how you implement it. To explain a little about how I understand the BRCM phy implementation.

In the PHY there is a hardware time counter that counts at 125MHz, around that counter is a PID loop that monitors incoming timestamps on PTPv1 or v2 packets and adjusts the time counter automatically to track an external time source. There is no need for anything in Linux to make this time source work or correct it. Other than running an application like ptp4l which sends and receives the correct packets such that the PHY can read/write the timestamps.

I suspect that is "probably" not correct Gordon. Also, 1ns resolution (as oppose to 8ns) is supported btw.

So suppose I was following 5 upstream PTP sources how would that work if they were each 2 years apart? And the PID "loop" would need to take into account packet rates etc. Those algos are complicated. Implementing them in hardware would be difficult and unnecessary. It would require the hardware to keep state in respect of T3 / T4 etc.etc. The PHC does not implement all that functionality.

What ptp4l, therefore, needs to do is just read the current time from the hardware and set its clock based on that (or just use the timestamps on incoming packets to do that). SYNC input and output should work correctly, but as previously mentioned SYNC input is only useful when the unit is the master clock and you want to synchronize with an external time source (like a GPS source).

There is a defined set of APIs for all of this in Linux. We already know that BCM has implemented support for 95% of this. We've tested it. It works. We just need the last 5% in respect of finely trimming the PHC. It's just a few registers that are missing at this point. Timestamping of packets ingress and egress on both layer 2 and 4 already work flawlessly.

I'm still in discussion with BRCM concerning the best way forward, but it's likely to be contracting someone to do the work for us and therefore get a Raspberry Pi email address etc.

So any contracting engineers who can show experience with IEEE1588 time sync (internal and external) and have the hardware to hand to test it. Or at least an engineer who can convince me that they have the knowledge and experience to go and learn enough to write and support this code... Please email me... 'gordon' 'curley a symbol' raspberrypi fullstop 'shortened version of company'

All the best,

Lasse

[callumbrieske]

There seems to be a lot of conjecture going on from people who clearly have a poor understanding of how PTP typically operates.

I have never come across a PTP solution that implements the PID servo in hardware outside of application specific FPGA implementations. Furthermore, it is would be completely unnecessary to have the PID loop in hardware.

Ptp4l (And pretty much every other IEEE 1588 implementation.) implements the PID Servo (or alternatives e.g. linear regression.) in software. The entire PTP implementation CAN be done in software, but for decent accuracy a hardware clock in the MAC/PHY is required. We need to be able to 'trim' this clock with our servo, hence the significance of the work getting adjfine working correctly.

There is also no hard requirement for the MAC to insert timestamps into outgoing packets, but we need a mechanism to sample the time that a packet passes through the MAC. This is the distinction between 1 Step and 2 Step PTP, and some hardware is not capable of 1 Step operation.

Lasse & Kyle clearly have significant real world experience with IEEE 1588, and its really unfortunate to see their knowledge and experience being disregarded.

[neggles]

There seems to be a lot of conjecture going on from people who clearly have a poor understanding of how PTP typically operates.

I have never come across a PTP solution that implements the PID servo in hardware outside of application specific FPGA implementations. Furthermore, it is would be completely unnecessary to have the PID loop in hardware.

There are quite a few FPGA implementations out and about, but yes, even the DP86340 doesn't implement the PID servo in hardware. The PHC is essentially a counter whose frequency is tuned by the PTP software.

There is also no hard requirement for the MAC to insert timestamps into outgoing packets, but we need a mechanism to sample the time that a packet passes through the MAC. This is the distinction between 1 Step and 2 Step PTP, and some hardware is not capable of 1 Step operation.

This applies when the timestamping is happening in the MAC, which (as the name implies) is not the case with PHY timestamping. For example, here are two other PHYs with timestamping capability that already have functional mainline kernel drivers (and public datasheets conveniently listing all of the registers and their functions (hey, Broadcom, are you listening? 😜):

• the TI/National Instrument DP86340 PHYTER
• and the QCA AR8031

Both these chips can add timestamps to both inbound and outbound frames transparently (see pg. 92 of the AR8031 datasheet and pg. 104 of the DP86340 datasheet), because adding/checking these timestamps in hardware at the PHY is the simplest and most straightforward way to achieve nanosecond-level accuracy.

I admit I am assuming the BCM54210PE's PTP feature set is at least on par with the AR8031's, but it would be rather disappointing if it wasn't...

As an aside, the majority of SoMs/SBCs on the market with 'hardware timestamping capability' - save for those which are explicitly geared towards TSN/RAN/AVB/etc purposes - do not use PHY timestamping, but instead use a SoC with timestamping hardware built into the MAC (e.g. the STM32MP157 and a number of NXP's SoCs). There's nothing wrong with doing it that way, but it requires more effort to achieve the same level of accuracy you can achieve with a HW-timestamping PHY (since you have to calibrate for PHY delay as well, amongst other things).

This makes the CM4 rather exceptional in this area, assuming the drivers ever materialize.

Lasse & Kyle clearly have significant real world experience with IEEE 1588, and its really unfortunate to see their knowledge and experience being disregarded.

Please do not mistake my unfamiliarity with Linux kernel PHY driver code/structure/layout for a lack of 'real world experience' with 1588v2. This isn't a particularly productive path to head down, though, so I'll leave it alone.

[callumbrieske]

@neg2led Apologies, my comments weren't directed at you.

For many applications you need to be able to support 2 step ptp, so having the MAC or PHY insert timestamps is not always useful, especially if there is no way of reading back transmitted timestamps to allow followup messages to be sent.

[ghollingworth]

I'm sorry @callumbrieske I clearly don't meet your definition of a person of use in this conversation.

I'll bow out of this and leave you to sort it out then

[neggles]

@callumbrieske ah, no problem, sorry for jumping to conclusions!

You do have a point on 2-step, however that's covered by some PHY registers which contain the sequenceId / sourcePortIdentity / timestamp of the last transmitted timestamped packet. Those timestamps would need to be pulled from the PHY shortly after the packet is sent, then correlated with the original packet; the code which does that is part of what confused me earlier.

I do still think this code should be a part of the main Broadcom PHY driver code - as it is with the DP86340, AR8031, e1000e, etc - since it's not likely to be upstreamable otherwise and I suspect it would significantly reduce how much code needs to be written (there are already routines for timestamping/PHC use on other broadcom PHYs), but I suppose that's down to developer preferences.

(one other note - it looks like AR8031 support is not actually properly/fully implemented in the kernel which would explain some things i've been seeing on another device - though an attempt to upstream support was made in 2020. Might work on that myself, could be quite useful.)

@ghollingworth apologies if I have added to your frustration with this, not my intent.

[mlichvar]

If everything is working except the PHC frequency adjustments, on the latest Linux kernels (5.17+ recommended) you can create a virtual clock on top of the PHC:

echo 1 > /sys/class/ptp/ptp0/n_vclocks

The virtual /dev/ptp1 will use a timecounter cyclecounter to allow the frequency to be adjusted in software. This can work only with two-step sync, but it could be useful for testing until the driver can be finished. If the driver didn't need to support one-step sync and PPS out, the cyclecounter-based solution could be in the driver itself. Some existing drivers (e.g. mlx4) do that.

You will need to patch your PTP software to bind its sockets to the virtual clock if it doesn't already support it. Here is an example: https://github.com/torvalds/linux/blob/master/tools/testing/selftests/net/timestamping.c

For linuxptp, there were patches submitted on the upstream development list.

[lasselj]

@mlichvar Thank you for this comment. In fact we discussed previously virtualising the PHC by applying a coefficient to the values read from the physical PHC. I was not aware of the virtual PHC support in the kernel you make reference to. I guess if we are left with no other choice we will select either this approach or similar. Thanks again.

[KyleJudd]

I've been on vacation for the past few days, and thus absent from this discussion. My favorite quote is "Having spent quite a bit of time lately digging through the sources and reading docs ..." - Thank you neg2led for taking the time to do that. There really are aspects of the hardware and driver that are a little counter-intuitive, and I will try to explain in another technical post how & why my code works the way it does. I communicate with my friends and co-workers through "lines of C & C++ code that I write". It's much easier to understand what I am thinking by working through the code. It's difficult for me to explain things to people that haven't worked with the code.

Second favorite is "Lasse & Kyle clearly have significant real world experience with IEEE 1588..." - Thanks for the support, and this is correct, but I also support Gordon's effort to find the best person for the job. Gordon knows that my expertise is limited to 802.1 AS / AVB, etc., and that is only a small subset of what this Raspberry Pi could be used for. It may be that Lasse and I are the only people available right now, but I'd be happy if Gordon can find someone that can do this work so that I don't have to :)

Lastly, the only remaining issues I see are

1) Adjusting the clock - It seems like we are trying to set register 0x1234 when we need to be setting register 0x4321, and the documentation is the best source for this info. Lasse's Timebeat software has great graphs that will illustrate whether the clock control is working correctly. For my 802.1 AS needs, I don't need to steer a clock, so I probably will implement a pseudo-clock in my own software, but I would also like to see the AdjFine functions work for those who need them - if possible. I'm sure Gordon is doing what he can to help with this.

2) Interrupt driven timestamping - This will improve efficiency, but not really that much anyways. We can only poll the PHY so often anyways, so this is not a huge issue (see tech post to follow). It occurs to me that the bcmgenet driver must be communicating with the PHY to receive "link status" and "speed" changes, so I will either mimick what they are doing or piggy-back on their interupt code if necessary.

Thanks again - stay tuned for updates in real time :)

[KyleJudd]

I think sharing my knowledge of AVB/TSN 802.1 AS will help answer some of the questions asked above and explain why, for me, my driver code is nearly 100% complete, even if it doesn't work with ptp4l or Timebeat ... yet. 802.1 AS is a profile or subset of 1588 which is commonly used for audio/video or automotive purposes, and is designed to provide sub-microsecond accuracy. The BCM54210PE only claims to support 1588-2008 and not 802.1 AS. Supporting 802.1 AS using the BCM54210PE MAY be possible, and that is what I am working through right now. In my testing, the BCM54210PE does support sub-microsecond timestamping, and that would not be possible if it were using any sort of software logic. I've also confirmed that the timestamping registers we read in the driver are not a running clock, rather a value locked-in when the timestamp is generated. In that way, it is 802.1 AS compliant - so far so good.

Let me compare the BCM54210PE to a chip that does claim 802.1 AS support, the Intel i210. It is an unfair comparison because they were designed for different purposes, but the differences will help illustrate how the BCM54210PE does work, and how the driver code is written to work with the BCM54210PE.

In Intel's product briefs for the i210, they boast 2 things - 1) Integrated PHY, 2) 802.1 AS support. The BCM54210PE PHY is NOT integrated into the MAC, and thus requires a separate driver. Non-integrated PHY's are common and traditional, and this is a good strategy to support multiple configurations. The problem is that traditionally these non-integrated PHYs exist on what is known as the MDIO bus, not the PCI bus, and the MDIO bus is significantly slower than the PCI bus. Traditionally, it was only used to communicate info like "link status" and "link speed" and those features are not necessarily time-critical. In modern PTP implementations, time-critical features are required of the PHY, and communicating through the MDIO bus can limit what the PHY can do. Thus, having an integrated PHY on the chip, on the PCI bus, is desirable when implementing more time-critical features.

Understanding this, the driver I have written communicates with the BCM54210PE entirely through the MDIO bus, and uses standard Linux provided functions to do so. Carlos did a great job building that infrastructure, and I simply tweaked a few things to get his code to work with my 802.1 AS gear. So far so good.

The BCM54210PE driver in its current state reads timestamp data through a single set of registers for both TX & RX timestamps. One register indicates whether a timestamp is available, four registers contain the timestamp value, and 2 other registers are read to get necessary meta-data info about the timestamp. Both TX and RX timestamps are read through the same 4 timestamp registers, and we need the meta-data to differentiate between TX and RX timestamps. We also need data (Sequence ID and Message Type) to help us match the timestamp values in the BCM54210PE and packets received in the bcmgenet driver. So we end up reading 7 registers on a slow bus to get 1 single timestamp, and this process almost requires us to store timestamps and match them later in worker threads.

If you look at the driver for the i210, they read TX timestamps from one register, RX from another register, and they don't really have a need to read other registers to get the meta-data, and they don't need to store the timestamps and match them later. When their MAC driver receives a packet, they can simply read the RX buffer and assume that there is a timestamp and that the timestamp matches the message received. After they send a message, they can read the TX buffer, and they might have to spin-wait a little, but they can assume the TX timestamp matches the packet that was just sent. This is a huge advantage because it is much more efficient, and without the need for storing and matching timestamps, there is a lot less that can go wrong.

But the BCM54210PE doesn't have that advantage, and even when we poll the PHY through the MDIO, we can only read about 20-30 times per second, because we are really reading 7 different individual registers. For proper 802.1 AS, we need to be able to read about 10-15 timestamps per second. So the BCM54210PE can do what 802.1 AS requires, but not much more. Using Interrupts to limit reads from 20-30 per sec to 10-15 may be possible, but it doesn't look like it will provide any serious performance boost.

The BCM54210PE driver does receive and match the 10-15 timestamps per second properly. There may be bugs to work out, but for the most part this feature is working correctly, and there already are limits to how much we can improve this. The BCM54210PE and MDIO bus setup does create problems where some of the timestamps may take more than 1 ms to flow from PHY, to driver buffer, to kernel buffer, to user buffer. Programs like ptp4l are hard coded to only wait 1 ms for the timestamp, and this may not be possible. But source code is provided for ptp4l and users can change this value and recompile the software if this is a problem. Strangely enough, 802.1 AS with sub-microsecond precision only requires that messages be sent/received with 10 ms turnaround time, so 1 ms response from the driver/kernel isn't even required for some of the most precise measurements.

What about the IO pins?

I know that the i210 was designed with a set of I/O pins that can be used to regulate the clock of an AD/DA converter. For 802.1 AS audio projects, the real purpose of using PTP is to measure peer delay and match the clock frequency of an AD/DA converter on one device with the frequency of another AD/DA converter on another device. This is done by matching the frequency of NIC clocks to a common master or leader clock using PTP, then syncing the AD/DA to the NIC frequency --OR-- vice versa, sync NIC frequency to AD/DA. I am no expert in this because I am a "software-only" guy, I do not have AD/DA converters to worry about. But i do know that this is a common use of IO pins in regards to PTP - match one hardware clock to a distant hardware clock by syncing to NIC clocks that are already in sync.

So, without AD/DA converters to worry about, the only clock-frequency sync I have to do is typically done with what someone referred to as "Pseudo-Clock". Linux seems to provide a few different ways to do this, but this is really something done outside the PHY driver. That being said, I do have a clever idea that I have used in the past that I can try in the PHY driver. By multiplying all clock values by a ratio (1.000001 or 0.999998), you should get the same result as if you were able to adjust the frequency of the hardware clock. Purists will argue with me if this is truly the right thing to do, or if it is 802.1 AS compliant, but I have read the 802.1 AS specs a few times, and I can't find anything that explicitly prohibits this. In fact, the use of frequency ratios is required in peer-delay calculations, and I interpret this as an acknowledgement that Pseudo-clocking may be necessary in certain 802.1 AS projects.

The stock i210 igb driver is only one of several i210 drivers people use to implement 802.1 AS AVB. Igb_avb is an Intel provided tuning, avb4linux contains another igb driver tunned differently. I'd imagine real-world manufacturers modify the igb driver as much as they need to get the i210 to do what they want.

So producing a single BCM54210PE driver that does every variation of PTP with every variation of external hardware or software pseudo-clock may not be possible. More realistically, anyone that wants to develop a serious product with the BCM54210PE would have to tweak the driver I wrote, tweak the ptp stack (ptp4l, etc), and tweak the software to take advantage of what the BCM54210PE CAN do. But Linux is good for that kind of stuff :)

If the BCM54210PE, or my driver, can't do what someone requires, Rasperry Pi has provided PCI slots so you can use an i210 with the CM4. CM4+i210 has been tested and it does work. There are other PTP options as well. But to use any of these options will likely require further tinkering with drivers, ptp stacks, software etc. Hopefully my explanation of how the BCM54210PE works through the MDIO bus will help others understand how my driver was written SO THAT THEY CAN TWEAK IT for their own purposes.

Thanks for your support.

[wowczarek]

@KyleJudd Great writeup. Reminds me of me repeatedly preaching to vendors when I was actively developing ptpd.

Programs like ptp4l are hard coded to only wait 1 ms for the timestamp, and this may not be possible.

Ha, lookey here:

ptpd -O | grep tx
libcck:linuxts_tx_backoff = 10
libcck:linuxts_tx_timeout = 1500
libcck:linuxts_tx_retries = 6
libcck:linuxts_tx_multiplier = 2.000000

Delayed TX timestamps (either occasional or constant) have been an issue for ages with many drivers, igb and others. What ptp4l does is more of a hard limit rather than an assumption, it's more of a "if it's been delayed so much, it's probably broken or not worth it" kind of thing, but if every TX timestamp is delayed, it's a different story.

It's been a while since I've looked into it, but I can vaguely remember that (not 100% sure if with h/w timestamps), a packet looped back with IP_MULTICAST_LOOP still contained the TX timestamp, so you could move on and get the timestamp when it was ready. PTPd was always designed to be able to consume its own, looped back messages as this was the only way to get TX timestamps in the early days. Can't remember now if this was reliable for h/w timestamps though, so I either never commited this or eventually removed it.

Edit: still trying to find the source, but IIRC the errqueue message that will eventually arrive at the socket with the late TX timestamp, I think has, or can have, the original packet attached to it. So I used to pick those up and feed back into PTPd that would recognise them as its own previously transmitted messages.

[mlichvar]

The TX timestamp timeout in ptp4l can be configured with the tx_timestamp_timeout option, no need for patching. The default value in the development code was recently changed to 10 milliseconds to better work with some existing hardware.

If the BCM can provide only about 10 timestamps per second, that will certainly limit the use cases, but I think it will still be a great feature for the CM4.

[KyleJudd]

Its good to hear that some of these ptp apps can be configured. I just meant that each has its own quirks and you have to figure out how to use the command line options or recompile the program to make it work. The differences in 1588 profiles and NIC/PHY hardware make it difficult for things to work out-of-the-box.

The BCM driver can handle probably 20-30 timestamps per second, which should be fine for most profiles/applications. Sub-microsecond 802.1 AS profile uses about 12 messages per second. I wish I could write driver code to handle error cases where driver is flooded with 100 or 1000 messages / sec, but I'm not sure there is anything I can do to prevent problems like that.

[lasselj]

Yeah, so Kyle and I caught up again this afternoon and the status is still the same I'm afraid. We do not have the information required about the Broadcom chip to proceed with the last outstanding work on the driver. So long as this continues to be the case, there is nothing we can do. I guess at this point, it just sort of is what it is... :-)

[JohnnyBBravo]

Wow, thanks everyone for putting in your time and effort in getting this to work! It's really unfortunate to see that development is now grinding to a halt because of a lack of documentation on the Broadcom chip.

@ghollingworth are you in a position to get the right documentation to the right people? It would be such a shame if the development of this driver strands at this point, having almost reached the finish line...

Thanks again everyone for your efforts!

[lasselj]

Hello,

A quick update to say that Timebeat/I am working for Raspberry Pi to resolve this issue. Today, I got the documents required and have now implemented a working adjfine function in the driver. The results are not bad and I'm pretty sure I can improve them further. Here is a screenshot from the Timebeat front-end showing the 54210PE on a CM4 keeping time to within +/-50ns.

I shall provide further updates in due course.

All the best,

Lasse

[nullr0ute]

A quick update to say that Timebeat/I am working for Raspberry Pi to resolve this issue. Today, I got the documents required and have now implemented a working adjfine function in the driver. The results are not bad and I'm pretty sure I can improve them further. Here is a screenshot from the Timebeat front-end showing the 54210PE on a CM4 keeping time to within +/-50ns.

Will these changes be submitted to the upstream linux kernel for review?

[lasselj]

Will these changes be submitted to the upstream linux kernel for review?

Yes

[callumbrieske]

This is excellent news.

[jwbowen]

@lasselj, can you post a link to the mailing list thread when you send the patches upstream?

[lasselj]

Here is a picture from earlier today of the CM4 on an IO board doing PPS out on pin 9 on the J2 jumper (as described on page 7 here) - which is really the SYNC_OUT from the 54210PE. I guess this could be handy and for sure I want to add support for the PTP_ENABLE_PPS and PTP_PF_PEROUT ioctls time permitting.

[wowczarek]

Here is a picture from earlier today of the CM4 on an IO board doing PPS out on pin 9 on the J2 jumper

Fantastic. Even without a working SYNC_IN this can be fed to an external TIC to measure and sync the board to an external reference, albeit in a roundabout way. But with a working SYNC_IN, either as a PPS device or PHC API external events, the CM4 becomes a time nut's time machine. Looking forward to this and I hope the input+output sawtooth / resolutions are decent.

[lasselj]

Fantastic. Even without a working SYNC_IN this can be fed to an external TIC to measure and sync the board to an external reference, albeit in a roundabout way. But with a working SYNC_IN, either as a PPS device or PHC API external events, the CM4 becomes a time nut's time machine. Looking forward to this and I hope the input+output sawtooth / resolutions are decent.

Am working on SYNC_IN and the more general "per_out" in addition to PPS... :-)

[jonathankrs]

Wow some improvement on this. Is the files already uploaded?

[ahmadexp]

Still some stubborn bugs, but overall things look promising.

Sent from my iPhone

On Apr 9, 2022, at 4:03 PM, jonathankrs @.***> wrote:

 Wow some improvement on this. Is the files already uploaded?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[lasselj]

Hi,

Yeah, so I've been working with @KyleJudd, @ahmadexp , @ghollingworth and Broadcom trying to make some progress. So far I am not all that happy about things, but maybe that's just me. I'd really like a bit more responsiveness from Broadcom.

The code is here if you want to contribute, review, comment.... :

https://github.com/timebeat-app/linux/tree/bcm54210-1588

On the upside:

• gettime, settime, adjfine & friends all work fine now so you can sync very accurately the phc
• gettimex is implemented but because of the mac<=>phy mdio bus separated nature of the bcm54210 no spin locks are possible, so the best I can get the system clock to is +/- 1-2us which is 2 orders of magnitude above my preference, but on the other hand in my experience quite a common number of a BCM LoM controller for instance.

(btw, pro-tip, if you want accurate sync between any phc and the system clock, then bump the cpu to max with the cpupower tool or similar. At lower p-states the sync is not good otherwise).

The downside:

• I am really struggling with the PPS OUT (perout) stuff. (sync out pin on the CM4 IO board. The documented behaviour in the BCM data sheet simply does not match what the controller does. I have documented this carefully, sent it to BCM, asked them to comment and so far I have not received a response (or an acknowledgement on my emails for that matter...)
• No PPS IN (extts) yet. I suspect this could be easier than the PPS OUT stuff though
• No one-step yet. It's cool. Must have one-step :-)
• Behaviour not interrupt driven yet. Still using poll.

For me the sync in/out stuff is quite important. For Raspberry PI projects one should be able to connect a GNSS receiver (@ahmadexp ideally on a Timecard :-)) or generate a stable frequency for some peripheral, so I'll keep at this until it is fixed.

Having said all this, PPS out is actually working, but not "consistently well" let's say (notice the strangeness of the pulse from the BCM chip.... this surely is a hardware bug....) :

https://drive.google.com/file/d/10z-qdgkZdX_4qUNrwKZ5sPCyKjL_3JaK/view?usp=sharing

I am working on perout, extts and one_step functionality. @KyleJudd is looking at different things including interrupt driven behaviour. @ahmadexp is testing, reviewing, commenting. If you want to join the effort let me know. There's still lots of stuff to be done.

Lastly, I guess it works now "somewhat"(tm). This is due to the significant work of @cfdez-tech, @KyleJudd and @ghollingworth. I was just the guy who dotted the Is and and crossed the Ts in the end. All credit goes to those guys.

All the best,

Lasse

[chronosfin]

Would be interesting to see other peoples views on this : https://github.com/timebeat-app/linux/tree/bcm54210-1588 if anyone has the ability to test?

[geerlingguy]

@chronosfin - Just a note (would post an issue but the repo there doesn't allow it)—cross-compiling can bring that time down from hour(s) on a Pi to 5-10 minutes on a modern desktop.

Here's how I cross-compile in Docker on my Mac/Windows/Linux machines: https://github.com/geerlingguy/raspberry-pi-pcie-devices/tree/master/extras/cross-compile

Also, here's a listing of just the changes (https://github.com/KyleJudd/linux/compare/rpi-5.10.y...timebeat-app:bcm54210-1588) — add .patch to the end of that URL to download a patch file that should be able to be applied to local checkouts of the rpi-5.10.y branch.

[lasselj]

Hello,

It's beginning to come together. The latest commit is very solid. I've just pushed it to the public repo (https://github.com/timebeat-app/linux/tree/bcm54210-1588). I am getting close to upstream PR. The polling thread of doom is gone, the performance is excellent. This is what I am getting at interval -7 (128 transactions per second). Definitely in the single nanos now :

/Lasse

[652436962]

Hello,

It's beginning to come together. The latest commit is very solid. I've just pushed it to the public repo (https://github.com/timebeat-app/linux/tree/bcm54210-1588). I am getting close to upstream PR. The polling thread of doom is gone, the performance is excellent. This is what I am getting at interval -7 (128 transactions per second). Definitely in the single nanos now :

/Lasse I install ubuntu 22.04 and build this(https://github.com/timebeat-app/linux/tree/bcm54210-1588), but show it ( $ ethtool -T eth0 Time stamping parameters for eth0: Capabilities: software-transmit
software-receive
software-system-clock PTP Hardware Clock: none Hardware Transmit Timestamp Modes: none Hardware Receive Filter Modes: none）

something wrong ?