Configured TAPRIO - ptp4l do not sync

[rezabfil-sec]

Hi, I am having a simple three node setup formed in a triangle:

source-->hop-->sink source-->sink

All of the connections are done using the Intel i210 cards. After I configured the hop using TAPRIO (w/ flags 0x1 and also w/o), the ptp4l stops to be in sync with other hops and also stops to exchange messages. When I try to setup ETF, CBS, or not qdisc on the hop, everything works as expected and clocks are in sync.

Are you aware of similar issues by the TAPRIO qdisc?

On the source is configured ETF qdisc as follows:

qdisc replace dev enp3s0 parent root handle 100 mqprio        \
      num_tc 3                                        \
      map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
      queues 1@0 1@1 2@2 \
      hw 0

qdisc replace dev enp3s0 parent 100:1 etf \
    clockid CLOCK_TAI delta 500000 offload  skip_sock_check

qdisc replace dev enp3s0 parent 100:2 etf \
    clockid CLOCK_TAI delta 500000 offload deadline_mode skip_sock_check

The hop has a following TAPRIO + ETF configuration:

qdisc replace dev enp5s0 parent root handle 100 taprio  \
  num_tc 3                                  \ 
  map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \  
  queues 1@0 1@1 2@2 \
  base-time 1605802187000000000          \ 
  flags 0x1                                                       \
  txtime-delay 400000 \
  sched-entry S 01 250000 \
  sched-entry S 00 50000 \ 
  sched-entry S 02 250000 \
  sched-entry S 00 50000 \ 
  sched-entry S 04 350000 \
  sched-entry S 00 50000 \ 
  clockid CLOCK_TAI                                                                                                                                                                                                                         

  qdisc replace dev enp5s0 parent 100:1 etf \
    clockid CLOCK_TAI delta 500000 offload  skip_sock_check                                                                                                                                                                                     
  qdisc replace dev enp5s0 parent 100:2 etf \   
     clockid CLOCK_TAI delta 500000 offload deadline_mode skip_sock_check           

I am collecting traffic on the sink using TCPDUMP.

ptp4l gPTP.cfg on the hop, sink and source follow same logic :

[global]
gmCapable               1
priority1               248
priority2               248
logAnnounceInterval     2
logSyncInterval         2
logMinDelayReqInterval  2
logMinPdelayReqInterval 2
syncReceiptTimeout      5
announceReceiptTimeout  5
neighborPropDelayThresh 1000
min_neighbor_prop_delay -20000000
assume_two_step         1
path_trace_enabled      1
follow_up_info          1
transportSpecific       0x1
ptp_dst_mac             01:80:C2:00:00:0E
network_transport       L2

[enp3s0]
boundary_clock_jbod     1
[enp4s0]
boundary_clock_jbod     1

Executing following ptp4lcommand:

ptp4l -f gPTP.cfg --step_threshold=1 --socket_priority 1

What have I tried so far:

• Changing various parameters regarding the delays, retransmissions, (logMinDelayReqInterval, neighborPropDelayThresh, announceReceiptTimeout,...), but it did not help.
• Various socket_priority values
• With and without flags 0x1 on the TAPRIO parent qdisc.
• Various configs on ETF child qdisc on the hop.
• Changing SW/HW timestamping.

To note, I also opened an issue on linuxptp, but after looking more into, it seems like the issue is caused by the TAPRIO itself, as other qdiscs do not introduce this problem... https://github.com/openil/linuxptp/issues/19

Any pointers on how to solve this issue would be appreciated! Thanks!

[vcgomes]

The first thing that comes to mind is that as you are using a cycle time of 1ms, it could be that when ptp4l sends a packet, that packet can be delayed for up to 1ms, ptp4l has a default timeout for the retrieval of transmission timestamps of 1ms, it could be that you are hitting this timeout.

You could try setting the tx_timestamp_timeout ptp4l config option to something like 10 or 100 and see if it helps.

Could you also paste the ptp4l log so I can see if this is indeed the case?

[rezabfil-sec]

Thanks, I tried it and here is the output:

ptp4l -f /data/gPTP.cfg --step_threshold=1 -H -m -l 7

ptp4l[5745.041]: config item (null).assume_two_step is 1
ptp4l[5745.041]: config item (null).check_fup_sync is 0
ptp4l[5745.041]: config item (null).tx_timestamp_timeout is 100
ptp4l[5745.041]: config item (null).hwts_filter is 0
ptp4l[5745.041]: config item (null).clock_servo is 0
ptp4l[5745.042]: config item (null).clock_type is 32768
ptp4l[5745.042]: config item (null).clock_servo is 0
ptp4l[5745.042]: config item (null).clockClass is 248
ptp4l[5745.042]: config item (null).clockAccuracy is 254
ptp4l[5745.042]: config item (null).offsetScaledLogVariance is 65535
ptp4l[5745.042]: config item (null).productDescription is ';;'
ptp4l[5745.042]: config item (null).revisionData is ';;'
ptp4l[5745.042]: config item (null).userDescription is ''
ptp4l[5745.042]: config item (null).manufacturerIdentity is '00:00:00'
ptp4l[5745.042]: config item (null).domainNumber is 0
ptp4l[5745.042]: config item (null).slaveOnly is 0
ptp4l[5745.042]: config item (null).gmCapable is 1
ptp4l[5745.042]: config item (null).gmCapable is 1
ptp4l[5745.042]: config item (null).G.8275.defaultDS.localPriority is 128
ptp4l[5745.042]: config item (null).maxStepsRemoved is 255
ptp4l[5745.042]: config item (null).time_stamping is 1
ptp4l[5745.042]: config item (null).twoStepFlag is 1
ptp4l[5745.042]: config item (null).twoStepFlag is 1
ptp4l[5745.042]: config item (null).time_stamping is 1
ptp4l[5745.042]: config item (null).priority1 is 248
ptp4l[5745.042]: config item (null).priority2 is 248
ptp4l[5745.042]: interface index 5 is up
ptp4l[5745.042]: interface index 6 is up
ptp4l[5745.042]: config item (null).free_running is 0
ptp4l[5745.042]: selected /dev/ptp2 as PTP clock
ptp4l[5745.043]: config item (null).clockIdentity is '000000.0000.000000'
ptp4l[5745.043]: config item (null).uds_address is '/var/run/ptp4l'
ptp4l[5745.043]: section item /var/run/ptp4l.announceReceiptTimeout now 0
ptp4l[5745.043]: section item /var/run/ptp4l.delay_mechanism now 0
ptp4l[5745.043]: section item /var/run/ptp4l.network_transport now 0
ptp4l[5745.043]: section item /var/run/ptp4l.delay_filter_length now 1
ptp4l[5745.043]: config item (null).free_running is 0
ptp4l[5745.043]: config item (null).freq_est_interval is 1
ptp4l[5745.043]: config item (null).write_phase_mode is 0
ptp4l[5745.043]: config item (null).gmCapable is 1
ptp4l[5745.043]: config item (null).kernel_leap is 1
ptp4l[5745.043]: config item (null).utc_offset is 37
ptp4l[5745.043]: config item (null).timeSource is 160
ptp4l[5745.043]: config item (null).pi_proportional_const is 0.000000
ptp4l[5745.043]: config item (null).pi_integral_const is 0.000000
ptp4l[5745.043]: config item (null).pi_proportional_scale is 0.000000
ptp4l[5745.043]: config item (null).pi_proportional_exponent is -0.300000
ptp4l[5745.043]: config item (null).pi_proportional_norm_max is 0.700000
ptp4l[5745.043]: config item (null).pi_integral_scale is 0.000000
ptp4l[5745.043]: config item (null).pi_integral_exponent is 0.400000
ptp4l[5745.043]: config item (null).pi_integral_norm_max is 0.300000
ptp4l[5745.043]: config item (null).step_threshold is 1.000000
ptp4l[5745.043]: config item (null).first_step_threshold is 0.000020
ptp4l[5745.043]: config item (null).max_frequency is 900000000
ptp4l[5745.043]: config item (null).servo_offset_threshold is 0
ptp4l[5745.043]: config item (null).servo_num_offset_values is 10
ptp4l[5745.043]: config item (null).dataset_comparison is 0
ptp4l[5745.043]: config item (null).delay_filter_length is 10
ptp4l[5745.043]: config item (null).delay_filter is 1
ptp4l[5745.043]: config item (null).tsproc_mode is 0
ptp4l[5745.043]: config item (null).initial_delay is 0
ptp4l[5745.043]: config item (null).summary_interval is 0
ptp4l[5745.043]: config item (null).sanity_freq_limit is 200000000
ptp4l[5745.043]: PI servo: sync interval 1.000 kp 0.700 ki 0.300000
ptp4l[5745.043]: config item /var/run/ptp4l.boundary_clock_jbod is 0
ptp4l[5745.043]: config item /var/run/ptp4l.network_transport is 0
ptp4l[5745.043]: config item /var/run/ptp4l.masterOnly is 0
ptp4l[5745.043]: config item /var/run/ptp4l.BMCA is 0
ptp4l[5745.043]: config item /var/run/ptp4l.delayAsymmetry is 0
ptp4l[5745.043]: config item /var/run/ptp4l.follow_up_info is 1
ptp4l[5745.043]: config item /var/run/ptp4l.freq_est_interval is 1
ptp4l[5745.043]: config item /var/run/ptp4l.msg_interval_request is 0
ptp4l[5745.043]: config item /var/run/ptp4l.net_sync_monitor is 0
ptp4l[5745.044]: config item /var/run/ptp4l.path_trace_enabled is 1
ptp4l[5745.044]: config item /var/run/ptp4l.tc_spanning_tree is 0
ptp4l[5745.044]: config item /var/run/ptp4l.ingressLatency is 0
ptp4l[5745.044]: config item /var/run/ptp4l.egressLatency is 0
ptp4l[5745.044]: config item /var/run/ptp4l.delay_mechanism is 0
ptp4l[5745.044]: config item /var/run/ptp4l.hybrid_e2e is 0
ptp4l[5745.044]: config item /var/run/ptp4l.fault_badpeernet_interval is 16
ptp4l[5745.044]: config item /var/run/ptp4l.fault_reset_interval is 4
ptp4l[5745.044]: config item /var/run/ptp4l.delay_filter_length is 1
ptp4l[5745.044]: config item /var/run/ptp4l.delay_filter is 1
ptp4l[5745.044]: config item /var/run/ptp4l.tsproc_mode is 0
ptp4l[5745.044]: config item (null).slave_event_monitor is ''
ptp4l[5745.044]: config item enp5s0.boundary_clock_jbod is 1
ptp4l[5745.044]: config item enp5s0.network_transport is 3
ptp4l[5745.044]: config item enp5s0.masterOnly is 0
ptp4l[5745.044]: config item enp5s0.BMCA is 0
ptp4l[5745.044]: config item enp5s0.delayAsymmetry is 0
ptp4l[5745.044]: config item enp5s0.follow_up_info is 1
ptp4l[5745.044]: config item enp5s0.freq_est_interval is 1
ptp4l[5745.044]: config item enp5s0.msg_interval_request is 0
ptp4l[5745.044]: config item enp5s0.net_sync_monitor is 0
ptp4l[5745.044]: config item enp5s0.path_trace_enabled is 1
ptp4l[5745.044]: config item enp5s0.tc_spanning_tree is 0
ptp4l[5745.044]: config item enp5s0.ingressLatency is 0
ptp4l[5745.044]: config item enp5s0.egressLatency is 0
ptp4l[5745.044]: config item enp5s0.delay_mechanism is 2
ptp4l[5745.044]: config item enp5s0.unicast_master_table is 0
ptp4l[5745.044]: config item enp5s0.unicast_listen is 0
ptp4l[5745.044]: config item enp5s0.hybrid_e2e is 0
ptp4l[5745.044]: config item enp5s0.fault_badpeernet_interval is 16
ptp4l[5745.044]: config item enp5s0.fault_reset_interval is 4
ptp4l[5745.044]: config item enp5s0.delay_filter_length is 10
ptp4l[5745.044]: config item enp5s0.delay_filter is 1
ptp4l[5745.044]: config item enp5s0.tsproc_mode is 0
ptp4l[5745.044]: config item enp6s0.boundary_clock_jbod is 1
ptp4l[5745.044]: config item enp6s0.network_transport is 3
ptp4l[5745.044]: config item enp6s0.masterOnly is 0
ptp4l[5745.044]: config item enp6s0.BMCA is 0
ptp4l[5745.044]: port 2: just a bunch of devices
ptp4l[5745.044]: config item enp6s0.delayAsymmetry is 0
ptp4l[5745.044]: config item enp6s0.follow_up_info is 1
ptp4l[5745.044]: config item enp6s0.freq_est_interval is 1
ptp4l[5745.044]: config item enp6s0.msg_interval_request is 0
ptp4l[5745.044]: config item enp6s0.net_sync_monitor is 0
ptp4l[5745.044]: config item enp6s0.path_trace_enabled is 1
ptp4l[5745.044]: config item enp6s0.tc_spanning_tree is 0
ptp4l[5745.044]: config item enp6s0.ingressLatency is 0
ptp4l[5745.044]: config item enp6s0.egressLatency is 0
ptp4l[5745.044]: config item enp6s0.delay_mechanism is 2
ptp4l[5745.044]: config item enp6s0.unicast_master_table is 0
ptp4l[5745.044]: config item enp6s0.unicast_listen is 0
ptp4l[5745.044]: config item enp6s0.hybrid_e2e is 0
ptp4l[5745.044]: config item enp6s0.fault_badpeernet_interval is 16
ptp4l[5745.044]: config item enp6s0.fault_reset_interval is 4
ptp4l[5745.044]: config item enp6s0.delay_filter_length is 10
ptp4l[5745.044]: config item enp6s0.delay_filter is 1
ptp4l[5745.044]: config item enp6s0.tsproc_mode is 0
ptp4l[5745.044]: config item enp5s0.logMinDelayReqInterval is 0
ptp4l[5745.044]: config item enp5s0.logAnnounceInterval is 0
ptp4l[5745.045]: config item enp5s0.inhibit_announce is 0
ptp4l[5745.045]: config item enp5s0.ignore_source_id is 0
ptp4l[5745.045]: config item enp5s0.announceReceiptTimeout is 3
ptp4l[5745.045]: config item enp5s0.syncReceiptTimeout is 3
ptp4l[5745.045]: config item enp5s0.transportSpecific is 1
ptp4l[5745.045]: config item enp5s0.ignore_transport_specific is 0
ptp4l[5745.045]: config item enp5s0.G.8275.portDS.localPriority is 128
ptp4l[5745.045]: config item enp5s0.logSyncInterval is -3
ptp4l[5745.045]: config item enp5s0.operLogSyncInterval is 0
ptp4l[5745.045]: config item enp5s0.logMinPdelayReqInterval is 0
ptp4l[5745.045]: config item enp5s0.operLogPdelayReqInterval is 0
ptp4l[5745.045]: config item enp5s0.neighborPropDelayThresh is 800
ptp4l[5745.045]: config item enp5s0.min_neighbor_prop_delay is -20000000
ptp4l[5745.045]: config item enp5s0.asCapable is 1
ptp4l[5745.045]: config item enp5s0.inhibit_delay_req is 0
ptp4l[5745.045]: config item enp5s0.ptp_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5745.045]: config item enp5s0.p2p_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5745.045]: config item global.socket_priority is 0
ptp4l[5745.102]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[5745.102]: config item enp6s0.logMinDelayReqInterval is 0
ptp4l[5745.102]: config item enp6s0.logAnnounceInterval is 0
ptp4l[5745.102]: config item enp6s0.inhibit_announce is 0
ptp4l[5745.102]: config item enp6s0.ignore_source_id is 0
ptp4l[5745.102]: config item enp6s0.announceReceiptTimeout is 3
ptp4l[5745.102]: config item enp6s0.syncReceiptTimeout is 3
ptp4l[5745.102]: config item enp6s0.transportSpecific is 1
ptp4l[5745.102]: config item enp6s0.ignore_transport_specific is 0
ptp4l[5745.102]: config item enp6s0.G.8275.portDS.localPriority is 128
ptp4l[5745.102]: config item enp6s0.logSyncInterval is -3
ptp4l[5745.102]: config item enp6s0.operLogSyncInterval is 0
ptp4l[5745.102]: config item enp6s0.logMinPdelayReqInterval is 0
ptp4l[5745.102]: config item enp6s0.operLogPdelayReqInterval is 0
ptp4l[5745.102]: config item enp6s0.neighborPropDelayThresh is 800
ptp4l[5745.102]: config item enp6s0.min_neighbor_prop_delay is -20000000
ptp4l[5745.102]: config item enp6s0.asCapable is 1
ptp4l[5745.102]: config item enp6s0.inhibit_delay_req is 0
ptp4l[5745.103]: config item enp6s0.ptp_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5745.103]: config item enp6s0.p2p_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5745.103]: config item global.socket_priority is 0
ptp4l[5745.142]: port 2: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[5745.142]: config item /var/run/ptp4l.logMinDelayReqInterval is 0
ptp4l[5745.142]: config item /var/run/ptp4l.logAnnounceInterval is 0
ptp4l[5745.142]: config item /var/run/ptp4l.inhibit_announce is 0
ptp4l[5745.142]: config item /var/run/ptp4l.ignore_source_id is 0
ptp4l[5745.142]: config item /var/run/ptp4l.announceReceiptTimeout is 0
ptp4l[5745.142]: config item /var/run/ptp4l.syncReceiptTimeout is 3
ptp4l[5745.142]: config item /var/run/ptp4l.transportSpecific is 1
ptp4l[5745.142]: config item /var/run/ptp4l.ignore_transport_specific is 0
ptp4l[5745.142]: config item /var/run/ptp4l.G.8275.portDS.localPriority is 128
ptp4l[5745.142]: config item /var/run/ptp4l.logSyncInterval is -3
ptp4l[5745.142]: config item /var/run/ptp4l.operLogSyncInterval is 0
ptp4l[5745.142]: config item /var/run/ptp4l.logMinPdelayReqInterval is 0
ptp4l[5745.142]: config item /var/run/ptp4l.operLogPdelayReqInterval is 0
ptp4l[5745.142]: config item /var/run/ptp4l.neighborPropDelayThresh is 800
ptp4l[5745.142]: config item /var/run/ptp4l.min_neighbor_prop_delay is -20000000
ptp4l[5745.142]: config item /var/run/ptp4l.asCapable is 1
ptp4l[5745.142]: config item /var/run/ptp4l.inhibit_delay_req is 0
ptp4l[5745.142]: config item (null).uds_address is '/var/run/ptp4l'
ptp4l[5745.142]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[5745.142]: port 1: received link status notification
ptp4l[5745.142]: interface index 5 is up
ptp4l[5745.142]: port 2: received link status notification
ptp4l[5745.143]: interface index 6 is up
ptp4l[5745.265]: port 1: peer port id set to 6cb311.fffe.52545e-1
ptp4l[5745.266]: missing timestamp on transmitted peer delay response
ptp4l[5745.266]: port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[5745.306]: waiting 2^{4} seconds to clear fault on port 1
ptp4l[5745.938]: port 2: peer port id set to 6cb311.fffe.525468-2
ptp4l[5745.939]: missing timestamp on transmitted peer delay response
ptp4l[5745.939]: port 2: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[5745.974]: waiting 2^{4} seconds to clear fault on port 2
ptp4l[5761.306]: clearing fault on port 1
ptp4l[5761.306]: config item enp5s0.logMinDelayReqInterval is 0
ptp4l[5761.306]: config item enp5s0.logAnnounceInterval is 0
ptp4l[5761.306]: config item enp5s0.inhibit_announce is 0
ptp4l[5761.306]: config item enp5s0.ignore_source_id is 0
ptp4l[5761.306]: config item enp5s0.announceReceiptTimeout is 3
ptp4l[5761.306]: config item enp5s0.syncReceiptTimeout is 3
ptp4l[5761.306]: config item enp5s0.transportSpecific is 1
ptp4l[5761.306]: config item enp5s0.ignore_transport_specific is 0
ptp4l[5761.306]: config item enp5s0.G.8275.portDS.localPriority is 128
ptp4l[5761.306]: config item enp5s0.logSyncInterval is -3
ptp4l[5761.306]: config item enp5s0.operLogSyncInterval is 0
ptp4l[5761.306]: config item enp5s0.logMinPdelayReqInterval is 0
ptp4l[5761.306]: config item enp5s0.operLogPdelayReqInterval is 0
ptp4l[5761.306]: config item enp5s0.neighborPropDelayThresh is 800
ptp4l[5761.306]: config item enp5s0.min_neighbor_prop_delay is -20000000
ptp4l[5761.306]: config item enp5s0.asCapable is 1
ptp4l[5761.306]: config item enp5s0.inhibit_delay_req is 0
ptp4l[5761.306]: config item enp5s0.ptp_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5761.306]: config item enp5s0.p2p_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5761.306]: config item global.socket_priority is 0
ptp4l[5761.338]: port 1: FAULTY to LISTENING on INIT_COMPLETE
ptp4l[5761.338]: port 1: ignoring message
ptp4l[5761.338]: port 1: received link status notification
ptp4l[5761.338]: interface index 5 is up
ptp4l[5761.974]: clearing fault on port 2
ptp4l[5761.974]: config item enp6s0.logMinDelayReqInterval is 0
ptp4l[5761.974]: config item enp6s0.logAnnounceInterval is 0
ptp4l[5761.974]: config item enp6s0.inhibit_announce is 0
ptp4l[5761.974]: config item enp6s0.ignore_source_id is 0
ptp4l[5761.974]: config item enp6s0.announceReceiptTimeout is 3
ptp4l[5761.974]: config item enp6s0.syncReceiptTimeout is 3
ptp4l[5761.974]: config item enp6s0.transportSpecific is 1
ptp4l[5761.974]: config item enp6s0.ignore_transport_specific is 0
ptp4l[5761.974]: config item enp6s0.G.8275.portDS.localPriority is 128
ptp4l[5761.974]: config item enp6s0.logSyncInterval is -3
ptp4l[5761.974]: config item enp6s0.operLogSyncInterval is 0
ptp4l[5761.974]: config item enp6s0.logMinPdelayReqInterval is 0
ptp4l[5761.974]: config item enp6s0.operLogPdelayReqInterval is 0
ptp4l[5761.974]: config item enp6s0.neighborPropDelayThresh is 800
ptp4l[5761.974]: config item enp6s0.min_neighbor_prop_delay is -20000000
ptp4l[5761.974]: config item enp6s0.asCapable is 1
ptp4l[5761.974]: config item enp6s0.inhibit_delay_req is 0
ptp4l[5761.974]: config item enp6s0.ptp_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5761.974]: config item enp6s0.p2p_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5761.974]: config item global.socket_priority is 0
ptp4l[5762.014]: port 2: FAULTY to LISTENING on INIT_COMPLETE
ptp4l[5762.014]: port 2: ignoring message
ptp4l[5762.014]: port 2: received link status notification
ptp4l[5762.014]: interface index 6 is up
ptp4l[5762.266]: port 1: peer port id set to 6cb311.fffe.52545e-1
ptp4l[5762.266]: missing timestamp on transmitted peer delay response
ptp4l[5762.266]: port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[5762.302]: waiting 2^{4} seconds to clear fault on port 1
ptp4l[5763.003]: port 2: peer port id set to 6cb311.fffe.525468-2
ptp4l[5763.003]: missing timestamp on transmitted peer delay response
ptp4l[5763.003]: port 2: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[5763.042]: waiting 2^{4} seconds to clear fault on port 2
ptp4l[5778.302]: clearing fault on port 1
ptp4l[5778.302]: config item enp5s0.logMinDelayReqInterval is 0
ptp4l[5778.302]: config item enp5s0.logAnnounceInterval is 0
ptp4l[5778.302]: config item enp5s0.inhibit_announce is 0
ptp4l[5778.302]: config item enp5s0.ignore_source_id is 0
ptp4l[5778.302]: config item enp5s0.announceReceiptTimeout is 3
ptp4l[5778.302]: config item enp5s0.syncReceiptTimeout is 3
ptp4l[5778.302]: config item enp5s0.transportSpecific is 1
ptp4l[5778.302]: config item enp5s0.ignore_transport_specific is 0
ptp4l[5778.302]: config item enp5s0.G.8275.portDS.localPriority is 128
ptp4l[5778.302]: config item enp5s0.logSyncInterval is -3
ptp4l[5778.302]: config item enp5s0.operLogSyncInterval is 0
ptp4l[5778.302]: config item enp5s0.logMinPdelayReqInterval is 0
ptp4l[5778.302]: config item enp5s0.operLogPdelayReqInterval is 0
ptp4l[5778.302]: config item enp5s0.neighborPropDelayThresh is 800
ptp4l[5778.302]: config item enp5s0.min_neighbor_prop_delay is -20000000
ptp4l[5778.302]: config item enp5s0.asCapable is 1
ptp4l[5778.302]: config item enp5s0.inhibit_delay_req is 0
ptp4l[5778.302]: config item enp5s0.ptp_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5778.302]: config item enp5s0.p2p_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5778.302]: config item global.socket_priority is 0
ptp4l[5778.342]: port 1: FAULTY to LISTENING on INIT_COMPLETE
ptp4l[5778.342]: port 1: received link status notification
ptp4l[5778.342]: interface index 5 is up
ptp4l[5779.042]: clearing fault on port 2
ptp4l[5779.042]: config item enp6s0.logMinDelayReqInterval is 0
ptp4l[5779.042]: config item enp6s0.logAnnounceInterval is 0
ptp4l[5779.042]: config item enp6s0.inhibit_announce is 0
ptp4l[5779.042]: config item enp6s0.ignore_source_id is 0
ptp4l[5779.042]: config item enp6s0.announceReceiptTimeout is 3
ptp4l[5779.042]: config item enp6s0.syncReceiptTimeout is 3
ptp4l[5779.042]: config item enp6s0.transportSpecific is 1
ptp4l[5779.042]: config item enp6s0.ignore_transport_specific is 0
ptp4l[5779.042]: config item enp6s0.G.8275.portDS.localPriority is 128
ptp4l[5779.042]: config item enp6s0.logSyncInterval is -3
ptp4l[5779.042]: config item enp6s0.operLogSyncInterval is 0
ptp4l[5779.042]: config item enp6s0.logMinPdelayReqInterval is 0
ptp4l[5779.042]: config item enp6s0.operLogPdelayReqInterval is 0
ptp4l[5779.042]: config item enp6s0.neighborPropDelayThresh is 800
ptp4l[5779.042]: config item enp6s0.min_neighbor_prop_delay is -20000000
ptp4l[5779.042]: config item enp6s0.asCapable is 1
ptp4l[5779.042]: config item enp6s0.inhibit_delay_req is 0
ptp4l[5779.042]: config item enp6s0.ptp_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5779.042]: config item enp6s0.p2p_dst_mac is '01:80:C2:00:00:0E'
ptp4l[5779.042]: config item global.socket_priority is 0
ptp4l[5779.078]: port 2: FAULTY to LISTENING on INIT_COMPLETE
ptp4l[5779.078]: port 2: received link status notification
ptp4l[5779.078]: interface index 6 is up
ptp4l[5779.267]: port 1: peer port id set to 6cb311.fffe.52545e-1
ptp4l[5779.267]: missing timestamp on transmitted peer delay response
ptp4l[5779.267]: port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[5779.298]: waiting 2^{4} seconds to clear fault on port 1
ptp4l[5780.078]: port 2: delay timeout
ptp4l[5780.078]: missing timestamp on transmitted peer delay request
ptp4l[5780.078]: port 2: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[5780.118]: waiting 2^{4} seconds to clear fault on port 2

I also modified the config to make it simpler.

 [global]
gmCapable               1
priority1               248
priority2               248
logAnnounceInterval     0
logSyncInterval         -3
syncReceiptTimeout      3
neighborPropDelayThresh 800
min_neighbor_prop_delay -20000000
assume_two_step         1
path_trace_enabled      1
follow_up_info          1
transportSpecific       0x1
ptp_dst_mac             01:80:C2:00:00:0E
network_transport       L2
delay_mechanism         P2P
tx_timestamp_timeout    100

[enp5s0]
network_transport       L2
boundary_clock_jbod     1
delay_mechanism         P2P
[enp6s0]
network_transport       L2
boundary_clock_jbod     1
delay_mechanism         P2P

Also, in comparison to the previous config, I started to see a traffic on the interfaces, so it might be a progress in the right direction. Thanks!

[vcgomes]

This is interesting, what kernel version are you running? I am thinking that you are hitting the bug that this patch solves: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1e08511d5d01884a3c9070afd52a47799312074a

[rezabfil-sec]

It is Ubuntu 20.04 LTS. cat /proc/version Linux version 5.4.0-45-generic (buildd@lgw01-amd64-033) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #49-Ubuntu SMP Wed Aug 26 13:38:52 UTC 2020

[rezabfil-sec]

Also, I was checking the TAPRIO man page and found the flags 0x2 option. https://man7.org/linux/man-pages/man8/tc-taprio.8.html

Which NICs currently available in market are supporting it? I tried it with the i210, but it does not work. Based on its specification it should support 802.1Qbv..

[rezabfil-sec]

Giving an update. I tried to see the behavior on a simpler setup with just two nodes and followed exactly the same configuration as provided in https://tsn.readthedocs.io/qdiscs.html#configuring-taprio-qdisc and https://tsn.readthedocs.io/timesync.html.

Unfortunately, even for that scenario the issue persists.

What worked is to not run the TAPRIO using the txtime-assist mode, which is not suitable for our scenario.

Are you able to reproduce this issue? Thank you!

[rezabfil-sec]

Hi, just an update from our side available on https://github.com/openil/linuxptp/issues/19

The issue is reproduced also on another setup (question from tribela).

We used i225 and it works with the flags 0x2option. Unfortunately, does not help with the PTP issue.

If there are any ideas on how to debug the issue, I am open to testing it! After trying different configs, as shown in previous posts, I ran out of ideas.

[albymor]

Same issue here. Any update on this?

[rezabfil-sec]

Unfortunately, does not look like - we solved by creating a PTP "overlay", where we use additional links for ptp4l sync between different nodes and synchronize later on the physical hardware clocks using phc2sys. Hopefully, you have a spare links for that.

[mremzero]

@rezabfil-sec

Update: Reread the full chain and realized I missed the fact that you tried setting socket-priority via the command line. In either case, the following does work for me on I210 NICs on Linux 5.15.79 with the RT PREEMPT patch.

Original: Please try to add the following in the ptp config file

socket_priority         2 

The issue is that the default priority is set to 0. Your priority map maps it to TC 2, the lowest priority traffic. Alternatively, this should also work:

qdisc replace dev enp5s0 parent root handle 100 taprio  \
  num_tc 3                                  \ 
  map 1 1 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \  
  queues 1@0 1@1 2@2 \
  base-time 1605802187000000000          \ 
  flags 0x1                                                       \
  txtime-delay 400000 \
  sched-entry S 01 250000 \
  sched-entry S 02 250000 \
  sched-entry S 04 350000 \
  clockid CLOCK_TAI     

[rezabfil-sec]

@mremzero Thanks for the comment. We have tried this approach in meanwhile and you are right, it helps, but unfortunately some of the challenges still persist:

• The socket_priority set to 2 or 3 makes it work, but only with ETF in offload. Otherwise, ETF without offload results in the same error

  ptp4l[5780.078]: port 2: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
  ptp4l[5780.118]: waiting 2^{4} seconds to clear fault on port 2

• Even though we could use it in offload, some NICs (e.g., the Intel i210) has only two HW queues fitting for the offload and those are to be used for higher priority traffic, where low latency and jitter is required.

So overall, I would still expect it to work for any other queue as a BE traffic, but definitely this approach can work for some scenarios. :)

[njavro]

Hi, has there been any progress with this issue?