how to improve the precision of phc2sys?

[RuiWang1340]

Hello I have a board with hardware like this. CPU: IMX6. swith: MV88e6390. This board act as TC. After synchronizing to the GMC, the OFFSET value between GMC and hardware Clock(/dev/ptp2) in MV88e6390 is about dozens of nano secend. phc2sys was used to synchronize the CLOCK_REALTIME to /dev/ptp2(hardware clock of MV88e6390). But the OFFSET value between CLOCK_REALTIME and /dev/ptp2 is up to 800 microsecond. Affter parameter of "-d /dev/pps2" is used, the OFFSET value, between CLOCK_REALTIME and /dev/ptp2, is about several microsecond. PPS is generate by MV88e6390. It is send to CPU through a GPIO. See pps-gpio.c in kernel.

The PPS improve the precision of phc2sys. but the precision is microsecond.
How can I inprove the precision of phc2sys?

1. The CPU access registers in MV88e6390 through MDIO bus;

2. All of registers in MV88e6390 share one lock.

   I have performed an experiment： setting a GPIO to high, as soon as the function pps_gpio_irq_handler was executed. the offset between PPS and the GPIO is 8 microsecond to 14 microsecond. So the jitter of interrupt is about 6 microsecond.

   again, How can I inprove the precision of phc2sys in this situation?

[vladimiroltean]

So you have tried two ways to synchronize the system clock to the Marvell switch:

1. Through MDIO

The jitter in read latency between one MDIO read and another is causing a measurement error of up to 800us. Is the MDIO bus shared with other devices (PHYs)? Can you disable/move those?

Actually as per commit net: dsa: mv88e6xxx: Longer timeout for PTP TX timestamp it appears that latencies as high as 40 ms are to be expected for reading timestamps/PTP time through MDIO.

In fact any MDIO access will make the latency unpredictable and hence throw off the time. There have been some patches to improve this: net: dsa: mv88e6xxx: Release mutex between each ATU read, net: dsa: mv88e6xxx: Release mutex between each statistics read.

One other trick to smoothen/average out hardware jitter (from man phc2sys):

       -N phc-num
              Specify  the number of master clock readings per one slave clock update. Only the fastest reading is used to update the slave clock, this is useful to minimize the error caused by ran‐     
              dom delays in scheduling and bus utilization.  The default is 5.

If these still do not help, I would suggest reaching out to the mv88e6xxx maintainers for advice:

$ ./scripts/get_maintainer.pl drivers/net/dsa/mv88e6xxx/ptp.c 
Andrew Lunn <andrew@lunn.ch> (maintainer:MARVELL 88E6XXX ETHERNET SWITCH FABRIC DRIVER)
netdev@vger.kernel.org (open list:MARVELL 88E6XXX ETHERNET SWITCH FABRIC DRIVER)

2. Through GPIO and PPS

Actually PPS input is best paired to a hardware block that takes timestamps on input compare events (as seen in the extts support for the ptp_qoriq driver). It doesn't look like the i.MX6 supports that. That way the timestamps are taken by hardware and not by the driver, and hence not subject to latency/jitter. Otherwise, as you noticed, the interrupt servicing latency is quite variable.

Miroslav Lichvar is keeping a fork of the pps-gpio-poll kernel module from openwrt-stratum1. The idea is that polling at the right time for pin change events yields a lower jitter than servicing interrupts. As an experiment could you try to port it over to the GPIO controller of the i.MX6?

May I ask what is the relation between your question and OpenIL?

[RuiWang1340]

hello Vladimir

Thank you for your advice. I'm so sorry i didn't make myself clear. my question should be "how to improve the precision of phc2sys on my board?"

Is the MDIO bus shared with other devices (PHYs)? Can you disable/move those? no, the MDIO bus is not shared with other devices. But driver read a register in MV88e6390 every 16 sec. and I can not disable this function.

-N phc-num was set to 2 in my experiment. As you said, the latencies for reading PTP time through MDIO is TOO high. phc2sys gives Warning of "PPS is not in sync with PHC" again and again, When the default value (5) was used. the fastest reading might be happend in fourth or in fifth, the corresponding value of “phc_ts % NS_PER_SEC” is bigger than PHC_PPS_OFFSET_LIMIT. Therefore the phc2sys does not work at all when -N =5 on my board. I have tried three ways to synchronize the system clock to the Marvell switch:

1. phc2sys -s /dev/ptp2 -c CLOCK_REALTIME -O 0 -m /dev/ptp2 is the hardware clock of MV88e6390 result: the OFFSET value between CLOCK_REALTIME and /dev/ptp2 is up to 800 microsecond.
2. phc2sys -s /dev/ptp2 -d /dev/pps2 -c CLOCK_REALTIME -O 0 -m after MV88e6390 synchronize the GMC, it output a PPS. A GPIO was set to be a PPS client on CPU. /dev/pps2 is the PPS client result: the OFFSET value between CLOCK_REALTIME and /dev/ptp2 is about several microsecond.

3. phc2sys -d /dev/pps2 -c CLOCK_REALTIME -O 0 -m result: the OFFSET value between CLOCK_REALTIME and /dev/ptp2 is about several microsecond too.
   But the sec is not correct.

   The kernel version is 4.19.2. the code of PPS client for IMX6 is in pps-gpio.c. The function of pps_gpio_irq_handler was changed, when this function was executed, Another GPIO was set to High, then the OFFSET between PPS given by MV88e6390 and this GPIO can be capture by oscilloscope. the offset between PPS and the GPIO is 8 microsecond to 14 microsecond.

   I have another question, How can I prove the OFFSET value between hardware clock and CLOCK_REALTIME, which given by phc2sys, is correct?

[vladimiroltean]

The offset between CLOCK_REALTIME and the PHC can be measured in several ways. It can be done through successive readings (clock_gettime), procedure which is sensitive to jitter. You know this is the case if the "delay" reported by phc2sys in its output is not constant. It can also be done through a new socket ioctl: PTP_SYS_OFFSET. The PTP driver callback is named getcrosststamp and very few PHC drivers implement it currently, so you have to rely upon the first method.

Let's focus on case 1 for a moment (no PPS). You can simplify it as follows:

# ptp4l -i swp1 -m -2 &
# phc2sys -a -r -m
(...)
ptp4l[941.743]: master offset        -17 s2 freq   -4163 path delay      1298
phc2sys[942.156]: CLOCK_REALTIME phc offset       -26 s2 freq   -4175 delay   4000

You can also measure the offset using testptp (provided with the Linux kernel sources, under tools/testing/selftests/ptp/testptp.c:

# testptp -d /dev/ptp0 -k 1
system and phc clock time offset request okay
system time: 1577345598.930027000
phc    time: 1893.829831615
system time: 1577345598.930028640
system/phc clock time offset is 1577343705100196205 ns
system     clock time delay  is 1640 ns

# testptp -d /dev/ptp0 -S
set time okay

# testptp -d /dev/ptp0 -k 1
system and phc clock time offset request okay
system time: 1917.237832815
phc    time: 1917.237839200
system time: 1917.237834855
system/phc clock time offset is -5365 ns
system     clock time delay  is 2040 ns

I would suggest understanding the characteristics of the PHC access latency via MDIO first, with testptp. As you pointed out in the code you linked to, even PPS uses MDIO access to figure out the correct second (PPS only gives phase alignment), so it is unavoidable to ensure that MDIO read latency is consistent.

[RuiWang1340]

hello Vladimir

Thank you for your help.

Is there a direct way to measure the offset between GMC and CLOCK_REALTIME of slave OC?

All that I can come up with a method for measurement, is an indirect way. first, the offset between GMC and slave OC hardware clock can be measured by oscilloscope, as both of them can output PPS signal. Then, the offset between slave OC hardware clock and slave OC CLOCK_REALTIME can be measured several ways, which have been listed on your last letter. Therefore the the offset value between GMC and CLOCK_REALTIME of slave OC can be estimated.

[vladimiroltean]

I don't know of a more direct/precise way either, sorry. Probably the most accurate way of processing PPS events through GPIO is to use poll mode from a hrtimer. That will reduce some/most of the PHC <-> CLOCK_REALTIME jitter, and the rest through clever filtering algorithms. Then you can probably also use a hrtimer to toggle PPS on another GPIO pin based on CLOCK_REALTIME.

[RuiWang1340]

hello Thank you for your help.

I have another question about the implementation of ptp4l in ring network. Chips on each board are like this. CPU: IMX6. swith: MV88e6390. There are six switch's GE port on each board.

I have five boards.
One board is set to be GMC. The second board is set to be OC(slave). The last three borads act as TC. These three TC build a ring network.

Although, RSTP was implemented, swich can discard all frames but management frames. Unfortunately, PTP frames are treated as management frames. CPU can receive PTP frames even though THE port is set to blocking by RSTP. whch means cyclic paths is not removed by RSTP. ptp4l is functioning well under this situation. ptp4l choses a "better" port by comparing the stepsRemoved field of Announce messages.

But, which method is the best(normal, general) way to deal with this situation? discarding ptp frame by RSTP, or just like the ptp4l program does?

[vladimiroltean]

I can't provide a complete answer to this - maybe this question could be better addressed by the linuxptp community. The linuxptp stack can establish a spanning tree through BMCA that is independent of the spanning tree for forwarding. In general this is a good thing, because the spanning tree established by RSTP may not be optimal for clock synchronization to the grandmaster. For the two trees to be independent it is necessary to use PTP over L2 (Ethernet) rather than L4 (UDP) - so that PTP frames may be trapped by the multicast MAC filters of the switches (transparent clocks). Looking at the mv88e6390 driver I do see it does claim L4 PTP support in its RX filter, so if this bothers you, you can give regular ptp4l -4 a try. The power utility profile of PTP (IEC 61850-9-3) does specify how PTP may be transported over redundant (HSR/PRP) paths - I've no insight in it, however.

[chealy]

I too work with i.MX6 and Marvell 88E6390 switch. Recently, I did some investigation into improving MDIO performance for the purposes of large amounts of register access for copper cable testing. Ultimately, I worked with Andrew Lunn and we found three different areas that improve MDIO throughput: 1) switch from interrupt driven MDIO completion to polled MDIO completion 2) Disable 32 bit MDIO preamble when talking to Marvell Switch as preamble is not required by the Marvell Switches 3) Increase MDIO clock speed from 2.5MHz to 12.5MHz as the Marvell Switch supports this. In making these three changes, we went from it taking on average more than 100uS between MDIO operations to under 20uS between MDIO operations. The latency between operations also became more consistent with much less variability. Here's a kernel submission with the referenced changes: https://lists.openwall.net/netdev/2020/04/18/2

I would think the above three items will help for the i.MX6 + Marvell 88E6390 use case.

[vladimiroltean]

Hi Chris, I've been quite actively watching the topic, since it has been of interest to sja1105 synchronization too. Hubert Feurstein had a go at this, too, but instead of improving the MDIO throughput itself, he went for cross-timestamping the buffer containing the PTP time with the system clock time at which it was collected: https://lore.kernel.org/netdev/20190820084833.6019-3-hubert.feurstein@vahle.at/T/ Sadly his patch series never got merged. I do believe that for the best solution, a combination of both approaches might be required. I did "steal" his idea and I added a cross-timestamping mechanism in the SPI subsystem: https://lore.kernel.org/netdev/20190905010114.26718-3-olteanv@gmail.com/T/ So there would exist a "precedent" if somebody were to add a similar mechanism in the MDIO layer.

[mksyahoo]

Hello I have a board with hardware like this. CPU: IMX6. swith: MV88e6390. This board act as TC. After synchronizing to the GMC, the OFFSET value between GMC and hardware Clock(/dev/ptp2) in MV88e6390 is about dozens of nano secend. phc2sys was used to synchronize the CLOCK_REALTIME to /dev/ptp2(hardware clock of MV88e6390). But the OFFSET value between CLOCK_REALTIME and /dev/ptp2 is up to 800 microsecond. Affter parameter of "-d /dev/pps2" is used, the OFFSET value, between CLOCK_REALTIME and /dev/ptp2, is about several microsecond. PPS is generate by MV88e6390. It is send to CPU through a GPIO. See pps-gpio.c in kernel.

The PPS improve the precision of phc2sys. but the precision is microsecond. How can I inprove the precision of phc2sys?

1. The CPU access registers in MV88e6390 through MDIO bus;
2. All of registers in MV88e6390 share one lock.

I have performed an experiment： setting a GPIO to high, as soon as the function pps_gpio_irq_handler was executed. the offset between PPS and the GPIO is 8 microsecond to 14 microsecond. So the jitter of interrupt is about 6 microsecond.

again, How can I inprove the precision of phc2sys in this situation?

[RuiWang1340] We are also using MV88e6390 and would like to use PTP. How are you synchronizing the MV88e6390 clock with the master? Can you please provide some guidance?

[RuiWang1340]

Hi mksyahoo，    Is there any problem with your experiment？ yes，the MV88e6390 clock synchronizing  with the master in my test.

---Original--- From: @.> Date: Fri, Nov 24, 2023 07:38 AM To: @.>; Cc: @.**@.>; Subject: Re: [nxp-archive/openil_linuxptp] how to improve the precision ofphc2sys? (#5)

Hello I have a board with hardware like this. CPU: IMX6. swith: MV88e6390. This board act as TC. After synchronizing to the GMC, the OFFSET value between GMC and hardware Clock(/dev/ptp2) in MV88e6390 is about dozens of nano secend. phc2sys was used to synchronize the CLOCK_REALTIME to /dev/ptp2(hardware clock of MV88e6390). But the OFFSET value between CLOCK_REALTIME and /dev/ptp2 is up to 800 microsecond. Affter parameter of "-d /dev/pps2" is used, the OFFSET value, between CLOCK_REALTIME and /dev/ptp2, is about several microsecond. PPS is generate by MV88e6390. It is send to CPU through a GPIO. See pps-gpio.c in kernel.

The PPS improve the precision of phc2sys. but the precision is microsecond. How can I inprove the precision of phc2sys?

The CPU access registers in MV88e6390 through MDIO bus;

All of registers in MV88e6390 share one lock.

I have performed an experiment： setting a GPIO to high, as soon as the function pps_gpio_irq_handler was executed. the offset between PPS and the GPIO is 8 microsecond to 14 microsecond. So the jitter of interrupt is about 6 microsecond.

again, How can I inprove the precision of phc2sys in this situation?

[RuiWang1340] We are also using MV88e6390 and would like to use PTP. How are you synchronizing the MV88e6390 clock with the master? Can you please provide some guidance?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[mksyahoo]

Hi mksyahoo，    Is there any problem with your experiment？ yes，the MV88e6390 clock synchronizing  with the master in my test. … ---Original--- From: @.> Date: Fri, Nov 24, 2023 07:38 AM To: @.>; Cc: @.**@.>; Subject: Re: [nxp-archive/openil_linuxptp] how to improve the precision ofphc2sys? (#5) Hello I have a board with hardware like this. CPU: IMX6. swith: MV88e6390. This board act as TC. After synchronizing to the GMC, the OFFSET value between GMC and hardware Clock(/dev/ptp2) in MV88e6390 is about dozens of nano secend. phc2sys was used to synchronize the CLOCK_REALTIME to /dev/ptp2(hardware clock of MV88e6390). But the OFFSET value between CLOCK_REALTIME and /dev/ptp2 is up to 800 microsecond. Affter parameter of "-d /dev/pps2" is used, the OFFSET value, between CLOCK_REALTIME and /dev/ptp2, is about several microsecond. PPS is generate by MV88e6390. It is send to CPU through a GPIO. See pps-gpio.c in kernel. The PPS improve the precision of phc2sys. but the precision is microsecond. How can I inprove the precision of phc2sys? The CPU access registers in MV88e6390 through MDIO bus; All of registers in MV88e6390 share one lock. I have performed an experiment： setting a GPIO to high, as soon as the function pps_gpio_irq_handler was executed. the offset between PPS and the GPIO is 8 microsecond to 14 microsecond. So the jitter of interrupt is about 6 microsecond. again, How can I inprove the precision of phc2sys in this situation? [RuiWang1340] We are also using MV88e6390 and would like to use PTP. How are you synchronizing the MV88e6390 clock with the master? Can you please provide some guidance? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

Thank you for the reply

[mksyahoo]

RuiWang1340

Hi RuiWang1340,

In your case, the switch 88e6390 is configured as DSA? If so are you running the ptp4l on one of the switch's port?

[RuiWang1340]

Hi mksyahoo    Yes, 88e6390 is  configured as DSA. And the device with 88e6390 work as TC mode.

---Original--- From: @.> Date: Sun, Dec 3, 2023 03:09 AM To: @.>; Cc: @.**@.>; Subject: Re: [nxp-archive/openil_linuxptp] how to improve the precision ofphc2sys? (#5)

RuiWang1340

Hi RuiWang1340,

In your case, the switch 88e6390 is configured as DSA? If so are you running the ptp4l on one of the switch's port?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[msmudide]

Hi mksyahoo    Yes, 88e6390 is  configured as DSA. And the device with 88e6390 work as TC mode. … ---Original--- From: @.> Date: Sun, Dec 3, 2023 03:09 AM To: @.>; Cc: @.**@.>; Subject: Re: [nxp-archive/openil_linuxptp] how to improve the precision ofphc2sys? (#5) RuiWang1340 Hi RuiWang1340, In your case, the switch 88e6390 is configured as DSA? If so are you running the ptp4l on one of the switch's port? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

Hi RuiWang1340,

Are you running the ptp4l? If so on what interface?

[msmudide]

Hello I have a board with hardware like this. CPU: IMX6. swith: MV88e6390. This board act as TC. After synchronizing to the GMC, the OFFSET value between GMC and hardware Clock(/dev/ptp2) in MV88e6390 is about dozens of nano secend. phc2sys was used to synchronize the CLOCK_REALTIME to /dev/ptp2(hardware clock of MV88e6390). But the OFFSET value between CLOCK_REALTIME and /dev/ptp2 is up to 800 microsecond. Affter parameter of "-d /dev/pps2" is used, the OFFSET value, between CLOCK_REALTIME and /dev/ptp2, is about several microsecond. PPS is generate by MV88e6390. It is send to CPU through a GPIO. See pps-gpio.c in kernel.

The PPS improve the precision of phc2sys. but the precision is microsecond. How can I inprove the precision of phc2sys?

1. The CPU access registers in MV88e6390 through MDIO bus;
2. All of registers in MV88e6390 share one lock.

I have performed an experiment： setting a GPIO to high, as soon as the function pps_gpio_irq_handler was executed. the offset between PPS and the GPIO is 8 microsecond to 14 microsecond. So the jitter of interrupt is about 6 microsecond.

again, How can I inprove the precision of phc2sys in this situation?

What kernel version are you using? Does your kernel has pps support?

[mksyahoo]

Hi RuiWang1340,

   Looks like you have the kernel that can program the switch to generate perioding outputs from its gpio pin, which is connected to your processor gpio/pps input. I used mv88e6xxx_config_periodic_trig() and mv88e6xxx_ptp_enable_perout() functions from the following patch - [PATCH net-next RFC 0/9] net: dsa: PTP timestamping for mv88e6xxx (kernel.org), but I do not see periodic output on the switch's gpio pin. Am I missing anything? Can you please point to the kernel or patches that you are using?

On Thursday, February 8, 2024 at 02:50:27 PM EST, msmudide ***@***.***> wrote:  

Hello I have a board with hardware like this. CPU: IMX6. swith: MV88e6390. This board act as TC. After synchronizing to the GMC, the OFFSET value between GMC and hardware Clock(/dev/ptp2) in MV88e6390 is about dozens of nano secend. phc2sys was used to synchronize the CLOCK_REALTIME to /dev/ptp2(hardware clock of MV88e6390). But the OFFSET value between CLOCK_REALTIME and /dev/ptp2 is up to 800 microsecond. Affter parameter of "-d /dev/pps2" is used, the OFFSET value, between CLOCK_REALTIME and /dev/ptp2, is about several microsecond. PPS is generate by MV88e6390. It is send to CPU through a GPIO. See pps-gpio.c in kernel.

The PPS improve the precision of phc2sys. but the precision is microsecond. How can I inprove the precision of phc2sys?

• The CPU access registers in MV88e6390 through MDIO bus;
• All of registers in MV88e6390 share one lock.

I have performed an experiment： setting a GPIO to high, as soon as the function pps_gpio_irq_handler was executed. the offset between PPS and the GPIO is 8 microsecond to 14 microsecond. So the jitter of interrupt is about 6 microsecond.

again, How can I inprove the precision of phc2sys in this situation?

What kernel version are you using? Does your kernel has pps support?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[msmudide]

Hello I have a board with hardware like this. CPU: IMX6. swith: MV88e6390. This board act as TC. After synchronizing to the GMC, the OFFSET value between GMC and hardware Clock(/dev/ptp2) in MV88e6390 is about dozens of nano secend. phc2sys was used to synchronize the CLOCK_REALTIME to /dev/ptp2(hardware clock of MV88e6390). But the OFFSET value between CLOCK_REALTIME and /dev/ptp2 is up to 800 microsecond. Affter parameter of "-d /dev/pps2" is used, the OFFSET value, between CLOCK_REALTIME and /dev/ptp2, is about several microsecond. PPS is generate by MV88e6390. It is send to CPU through a GPIO. See pps-gpio.c in kernel.

The PPS improve the precision of phc2sys. but the precision is microsecond. How can I inprove the precision of phc2sys?

1. The CPU access registers in MV88e6390 through MDIO bus;
2. All of registers in MV88e6390 share one lock.

I have performed an experiment： setting a GPIO to high, as soon as the function pps_gpio_irq_handler was executed. the offset between PPS and the GPIO is 8 microsecond to 14 microsecond. So the jitter of interrupt is about 6 microsecond.

again, How can I inprove the precision of phc2sys in this situation?

Hi RuiWang1340,

 Can you please share how you made pps come out of the MV88e6390? I meant to ask what kernel changes are needed.