GigE synchronization problem: Multiple cameras assigned as Master

[kostastsing]

I have two a2A1920-51gcPRO cameras that are connected on the same Premium Basler GigE Interface Card connected to an Ubuntu server via PCIe. I am trying to perform synchronized image acquisition from the two cameras using the Periodic Signal feature. To initialize the PTP synchronization I am using the same code as here: https://github.com/basler/pypylon/issues/535 but I am setting the Priority1 of the first camera to 255 and of the other to 0 in order to use the first one as Master. However, after configuring PTP synchronization both cameras are assigned as Master. What may the problem be? I have also uploaded a screenshot of the network configuration.

[thiesmoeller]

If you use a two or four port card, best is to use the Ubuntu PTP daemon as grandmaster clock and tell the daemon to sync all interfaces.

Less performant but also working is to bridge the interfaces.

[kostastsing]

Is there a guide how to setup an Ubuntu PTP daemon as grandmaster clock? I have followed the instructions mentioned here https://github.com/basler/pypylon/issues/603 for running ptp4l with ptp4l -i enp3s0f1 -i enp3s0f3 -m --priority1 1 -l 7 --logSyncInterval -1 but I haven't managed to get it working.

[thiesmoeller]

Leave the camera PTP settings as default, as you want to be ptp4l as master

[Nailtha]

Thank you @thiesmoeller for the support. I am coming back to this as I'm working on the same issue with @kostastsing and have not gotten it to work so far.

When I run the following command for ptp4l: sudo ptp4l -i eth0 -i eth1 -m , ptp4l fails to create a clock, as seen in the error below:

ptp4l[675586.772]: selected /dev/ptp0 as PTP clock
ptp4l[675586.772]: port 2 (eth1): PHC device mismatch
ptp4l[675586.772]: port 2 (eth1): /dev/ptp0 requested, ptp1 attached
ptp4l[675586.772]: failed to open port eth1
failed to create a clock

When I try and run ptp4l for each port in a separate terminal (i.e. sudo ptp4l -i eth0 -m and sudo ptp4l -i eth1 -m), I get the following outputs:

for eth0

ptp4l[675747.970]: port 1 (eth0): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[675747.971]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[675747.971]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[675749.647]: port 1 (eth0): new foreign master 003053.fffe.36326d-1
ptp4l[675753.211]: selected best master clock 003053.fffe.36326d
ptp4l[675753.211]: foreign master not using PTP timescale
ptp4l[675753.211]: port 1 (eth0): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[675754.252]: master offset     -20586 s0 freq  +29916 path delay       527
ptp4l[675755.143]: master offset     -20622 s1 freq  +29879 path delay       530
ptp4l[675756.025]: master offset          5 s2 freq  +29884 path delay       530
ptp4l[675756.026]: port 1 (eth0): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[675756.907]: master offset          7 s2 freq  +29888 path delay       528
ptp4l[675757.843]: master offset          6 s2 freq  +29889 path delay       527

for eth1, another master clock is selected:

ptp4l[675812.357]: selected /dev/ptp1 as PTP clock
ptp4l[675812.359]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[675812.359]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[675812.359]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[675813.328]: port 1 (eth1): new foreign master 003053.fffe.36326e-1
ptp4l[675816.892]: selected best master clock 003053.fffe.36326e
ptp4l[675816.892]: foreign master not using PTP timescale
ptp4l[675816.893]: port 1 (eth1): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[675818.015]: master offset         29 s0 freq  +24582 path delay       521
ptp4l[675819.007]: master offset         28 s2 freq  +24581 path delay       522
ptp4l[675819.007]: port 1 (eth1): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED

Running phc2sys (i.e. phc2sys -a -rr -m) I get the following:

phc2sys[675875.526]: reconfiguring after port state change
phc2sys[675875.527]: selecting eth1 as domain source clock
phc2sys[675875.527]: selecting CLOCK_REALTIME for synchronization
phc2sys[675875.527]: CLOCK_REALTIME phc offset 1713263380160434468 s0 freq +100000000 delay   1901
phc2sys[675876.527]: failed to step clock: Invalid argument
phc2sys[675877.527]: CLOCK_REALTIME phc offset 1713263379938126418 s0 freq  -11948 delay   1988
phc2sys[675878.528]: CLOCK_REALTIME phc offset 1713263379826953752 s2 freq -100000000 delay   1872
phc2sys[675879.528]: CLOCK_REALTIME phc offset 1713263379917879311 s2 freq +100000000 delay   2429
phc2sys[675880.529]: CLOCK_REALTIME phc offset 1713263379806735088 s2 freq +100000000 delay   1872
phc2sys[675881.529]: clockcheck: clock jumped forward or running faster than expected!
phc2sys[675881.529]: CLOCK_REALTIME phc offset 1713263383076319518 s0 freq +100000000 delay   1987
phc2sys[675882.529]: CLOCK_REALTIME phc offset 1713263382965157290 s2 freq -100000000 delay   1987
phc2sys[675883.530]: CLOCK_REALTIME phc offset 1713263383056081119 s2 freq +100000000 delay   2464

And somehow, the clocks do not seem to be synchronized. Any idea what I'm missing here? Thanks!

[Nailtha]

As an additional input that might be of help, currently I am trying to run the following: ptp4l for eth0 and eth1, independently, i.e. as described above.

At the same time, again as described above, I am running phc2sys in a different terminal window which, as I understand it, is synchronizing the system clock to the selected master clock (eth0 in my case).

I notice however the following behavior: Even though phc2sys is trying to minimize clock offfset, when the offset actually becomes small, the clock jumps, increasing the offset, as seen below.

phc2sys[921687.524]: CLOCK_REALTIME phc offset -14181233 s2 freq -831849 delay   2182
phc2sys[921688.524]: CLOCK_REALTIME phc offset -13362220 s2 freq -4267206 delay   2210
phc2sys[921689.524]: CLOCK_REALTIME phc offset  -9122829 s2 freq -4036481 delay   2186
phc2sys[921690.525]: CLOCK_REALTIME phc offset  -5112227 s2 freq -2762728 delay   2217
phc2sys[921691.525]: CLOCK_REALTIME phc offset  -2367425 s2 freq -1551594 delay   2214
phc2sys[921692.526]: CLOCK_REALTIME phc offset   -828851 s2 freq -723247 delay   2211
phc2sys[921693.526]: CLOCK_REALTIME phc offset   -117239 s2 freq -260291 delay   2209
phc2sys[921694.526]: CLOCK_REALTIME phc offset    131750 s2 freq  -46473 delay   2240
phc2sys[921695.527]: clockcheck: clock jumped forward or running faster than expected!
phc2sys[921695.527]: CLOCK_REALTIME phc offset 2250596511 s0 freq  -46473 delay   2208
phc2sys[921696.527]: CLOCK_REALTIME phc offset 2250631625 s2 freq -103594 delay   2176
phc2sys[921697.528]: CLOCK_REALTIME phc offset 2250723832 s2 freq +100000000 delay   2240
phc2sys[921698.528]: CLOCK_REALTIME phc offset 2139574893 s2 freq +100000000 delay   2016
phc2sys[921699.528]: CLOCK_REALTIME phc offset 2028415098 s2 freq +100000000 delay   1987
phc2sys[921700.529]: CLOCK_REALTIME phc offset 1917257360 s2 freq +100000000 delay   1988
phc2sys[921701.529]: CLOCK_REALTIME phc offset 1806099089 s2 freq +100000000 delay   1958

I cannot explain this but I have a feeling it affects camera synchronization. Any thoughts? Thanks!