OpenVisualCloud / Media-Transport-Library

A real-time media transport(DPDK, AF_XDP, RDMA) stack for both raw and compressed video based on COTS hardware.
BSD 3-Clause "New" or "Revised" License
171 stars 55 forks source link

Long-running code will cause epoch drop. #937

Closed blaine0611 closed 1 month ago

blaine0611 commented 4 months ago

Hello,

Thanks for sharing your great work. I've been using your library for a few months. However, running the code for a long time may cause epoch drop.

Code running conditions: MTL Version: 23.12.0 Wed Apr 10 05:59:11 2024 091158a4 gcc-13.2.0 DPDK Version: DPDK 23.07.0 CPU: Intel(R) Xeon(R) Gold 6430 Memory: Samsung M321R2GA3BB6-CQKVG DDR5 4800MHz 16G x 8 Network Card: Ethernet Controller E810-C for QSFP 1592 Operating platform: Ubuntu 23.10

Running the MTL-library code about 14 hours with eight TX and four RX. There were three epoch drops in total. The first epoch drop occurred after six hours. The second epoch drop occurred after nine hours. The third epoch drop occurred after twelve hours.

I was wondering if you could provide some insight into how to solve my situation. Thank you.

After six hours:

MT: 2024-07-30 03:05:28, M T D E V S T A T E MT: 2024-07-30 03:05:28, DEV(0): Avr rate, tx: 70525.475414 Mb/s, rx: 0.020678 Mb/s, pkts, tx: 66650456, rx: 281 MT: 2024-07-30 03:05:28, M T D E V S T A T E MT: 2024-07-30 03:05:28, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-07-30 03:05:28, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, CNI(0): eth_rx_rate 0.020678 Mb/s, eth_rx_cnt 281 MT: 2024-07-30 03:05:28, PTP(0): time 1722280122451157100, 2024-07-30 03:08:05 MT: 2024-07-30 03:05:28, PTP(0): delta avg 1007, min 1002, max 1037, cnt 80 MT: 2024-07-30 03:05:28, PTP(0): correct_delta avg 62, min 54, max 110, cnt 80 MT: 2024-07-30 03:05:28, PTP(0): path_delay avg 10, min 5, max 15, cnt 80 MT: 2024-07-30 03:05:28, PTP(0): mode l4, sync cnt 80, expect avg 1007:65@0.125742s MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 50.000110, frame 500 pkts 8229932:8228929 inflight 504215:506452 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0): throughput 8707.439192 Mb/s: 0.000000 Mb/s, cpu busy 75.461411 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 16 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0:app_tx_video_1): fps 49.999980, frame 500 pkts 8229956:8228958 inflight 504444:506454 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0): throughput 8707.447263 Mb/s: 0.000000 Mb/s, cpu busy 75.481621 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, DEV(0): Avr rate, tx: 70525.475414 Mb/s, rx: 0.020678 Mb/s, pkts, tx: 66650456, rx: 281 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0): transmitter recalculate warmup 13 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 49.999854, frame 500 pkts 8229972:8228973 inflight 505020:506469 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0): throughput 8707.441229 Mb/s: 0.000000 Mb/s, cpu busy 75.420784 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 11 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 49.999752, frame 500 pkts 8229988:8228985 inflight 503868:506449 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0): throughput 8707.436183 Mb/s: 0.000000 Mb/s, cpu busy 75.572968 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 9 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 49.999668, frame 500 pkts 8230004:8229004 inflight 505476:506422 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0): throughput 8707.441703 Mb/s: 0.000000 Mb/s, cpu busy 75.444397 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 12 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 49.999569, frame 500 pkts 8230004:8229005 inflight 503944:506474 MT: 2024-07-30 03:05:28, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0): throughput 8707.425511 Mb/s: 0.000000 Mb/s, cpu busy 75.574135 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 25 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.999473, frame 500 pkts 8230040:8229041 inflight 504697:506485 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0): throughput 8707.446898 Mb/s: 0.000000 Mb/s, cpu busy 75.455093 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 16 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.999382, frame 500 pkts 8230048:8229050 inflight 505154:506514 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0): throughput 8707.440538 Mb/s: 0.000000 Mb/s, cpu busy 75.432060 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 10 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 03:05:28, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 03:05:28, E N D S T A T E MT: 2024-07-30 03:05:28, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-07-30 03:05:28, CNI(0): eth_rx_rate 0.020678 Mb/s, eth_rx_cnt 281 MT: 2024-07-30 03:05:28, PTP(0): time 1722280122451157100, 2024-07-30 03:08:05 MT: 2024-07-30 03:05:28, PTP(0): delta avg 1007, min 1002, max 1037, cnt 80 MT: 2024-07-30 03:05:28, PTP(0): correct_delta avg 62, min 54, max 110, cnt 80 MT: 2024-07-30 03:05:28, PTP(0): path_delay avg 10, min 5, max 15, cnt 80 MT: 2024-07-30 03:05:28, PTP(0): mode l4, sync cnt 80, expect avg 1007:65@0.125742s MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 50.000110, frame 500 pkts 8229932:8228929 inflight 504215:506452 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0): throughput 8707.439192 Mb/s: 0.000000 Mb/s, cpu busy 75.461411 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 16 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0:app_tx_video_1): fps 49.999980, frame 500 pkts 8229956:8228958 inflight 504444:506454 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0): throughput 8707.447263 Mb/s: 0.000000 Mb/s, cpu busy 75.481621 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0): transmitter recalculate warmup 13 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(1,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 49.999854, frame 500 pkts 8229972:8228973 inflight 505020:506469 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0): throughput 8707.441229 Mb/s: 0.000000 Mb/s, cpu busy 75.420784 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 11 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 49.999752, frame 500 pkts 8229988:8228985 inflight 503868:506449 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0): throughput 8707.436183 Mb/s: 0.000000 Mb/s, cpu busy 75.572968 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 9 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 49.999668, frame 500 pkts 8230004:8229004 inflight 505476:506422 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0): throughput 8707.441703 Mb/s: 0.000000 Mb/s, cpu busy 75.444397 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 12 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 49.999569, frame 500 pkts 8230004:8229005 inflight 503944:506474 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0): throughput 8707.425511 Mb/s: 0.000000 Mb/s, cpu busy 75.574135 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 25 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.999473, frame 500 pkts 8230040:8229041 inflight 504697:506485 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0): throughput 8707.446898 Mb/s: 0.000000 Mb/s, cpu busy 75.455093 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 16 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.999382, frame 500 pkts 8230048:8229050 inflight 505154:506514 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0): throughput 8707.440538 Mb/s: 0.000000 Mb/s, cpu busy 75.432060 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 10 MT: 2024-07-30 03:05:28, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 03:05:28, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 03:05:28, E N D S T A T E

After nine hours:

MT: 2024-07-30 06:14:18, M T D E V S T A T E MT: 2024-07-30 06:14:18, DEV(0): Avr rate, tx: 70524.853245 Mb/s, rx: 0.020544 Mb/s, pkts, tx: 66649866, rx: 280 MT: 2024-07-30 06:14:18, M T D E V S T A T E MT: 2024-07-30 06:14:18, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-07-30 06:14:18, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, CNI(0): eth_rx_rate 0.020544 Mb/s, eth_rx_cnt 280 MT: 2024-07-30 06:14:18, PTP(0): time 1722291452439242945, 2024-07-30 06:16:55 MT: 2024-07-30 06:14:18, PTP(0): delta avg 1029, min 1018, max 1067, cnt 80 MT: 2024-07-30 06:14:18, PTP(0): correct_delta avg 366, min 220, max 445, cnt 80 MT: 2024-07-30 06:14:18, PTP(0): path_delay avg 11, min 5, max 20, cnt 80 MT: 2024-07-30 06:14:18, PTP(0): mode l4, sync cnt 80, expect avg 1032:404@0.125512s MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 50.000501, frame 500 pkts 8229860:8228859 inflight 504037:506455 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0): throughput 8707.433238 Mb/s: 0.000000 Mb/s, cpu busy 75.556839 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 11 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0:app_tx_video_0): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0:app_tx_video_1): fps 50.000415, frame 500 pkts 8229864:8228863 inflight 503819:506464 MT: 2024-07-30 06:14:18, DEV(0): Avr rate, tx: 70524.853245 Mb/s, rx: 0.020544 Mb/s, pkts, tx: 66649866, rx: 280 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0): throughput 8707.422523 Mb/s: 0.000000 Mb/s, cpu busy 75.550911 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0): transmitter recalculate warmup 14 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0:app_tx_video_1): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 50.000301, frame 500 pkts 8229896:8228893 inflight 504224:506447 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0): throughput 8707.434416 Mb/s: 0.000000 Mb/s, cpu busy 75.486870 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 26 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0:app_tx_video_2): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 50.000216, frame 500 pkts 8229912:8228911 inflight 506051:506473 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0): throughput 8707.438686 Mb/s: 0.000000 Mb/s, cpu busy 75.427589 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 15 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0:app_tx_video_3): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 50.000133, frame 500 pkts 8229964:8228969 inflight 504017:506409 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0): throughput 8707.486395 Mb/s: 0.000000 Mb/s, cpu busy 75.525932 MT: 2024-07-30 06:14:18, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0): dummy pkts 1000, burst 998 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 15 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0:app_tx_video_4): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 50.000027, frame 500 pkts 8230000:8229000 inflight 503945:506470 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0): throughput 8707.499874 Mb/s: 0.000000 Mb/s, cpu busy 75.528267 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 19 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0:app_tx_video_5): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.999951, frame 500 pkts 8230000:8229000 inflight 504119:506462 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0): throughput 8707.486634 Mb/s: 0.000000 Mb/s, cpu busy 75.494156 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 17 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0:app_tx_video_6): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.999805, frame 500 pkts 8230000:8229000 inflight 504387:506504 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0): throughput 8707.461156 Mb/s: 0.000000 Mb/s, cpu busy 75.516022 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 14 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0:app_tx_video_7): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 06:14:18, E N D S T A T E MT: 2024-07-30 06:14:18, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-07-30 06:14:18, CNI(0): eth_rx_rate 0.020544 Mb/s, eth_rx_cnt 280 MT: 2024-07-30 06:14:18, PTP(0): time 1722291452439242945, 2024-07-30 06:16:55 MT: 2024-07-30 06:14:18, PTP(0): delta avg 1029, min 1018, max 1067, cnt 80 MT: 2024-07-30 06:14:18, PTP(0): correct_delta avg 366, min 220, max 445, cnt 80 MT: 2024-07-30 06:14:18, PTP(0): path_delay avg 11, min 5, max 20, cnt 80 MT: 2024-07-30 06:14:18, PTP(0): mode l4, sync cnt 80, expect avg 1032:404@0.125512s MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 50.000501, frame 500 pkts 8229860:8228859 inflight 504037:506455 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0): throughput 8707.433238 Mb/s: 0.000000 Mb/s, cpu busy 75.556839 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 11 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(0,0:app_tx_video_0): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0:app_tx_video_1): fps 50.000415, frame 500 pkts 8229864:8228863 inflight 503819:506464 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0): throughput 8707.422523 Mb/s: 0.000000 Mb/s, cpu busy 75.550911 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0): transmitter recalculate warmup 14 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(1,0:app_tx_video_1): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 50.000301, frame 500 pkts 8229896:8228893 inflight 504224:506447 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0): throughput 8707.434416 Mb/s: 0.000000 Mb/s, cpu busy 75.486870 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 26 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(2,0:app_tx_video_2): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 50.000216, frame 500 pkts 8229912:8228911 inflight 506051:506473 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0): throughput 8707.438686 Mb/s: 0.000000 Mb/s, cpu busy 75.427589 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 15 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(3,0:app_tx_video_3): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 50.000133, frame 500 pkts 8229964:8228969 inflight 504017:506409 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0): throughput 8707.486395 Mb/s: 0.000000 Mb/s, cpu busy 75.525932 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0): dummy pkts 1000, burst 998 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 15 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(4,0:app_tx_video_4): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 50.000027, frame 500 pkts 8230000:8229000 inflight 503945:506470 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0): throughput 8707.499874 Mb/s: 0.000000 Mb/s, cpu busy 75.528267 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 19 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(5,0:app_tx_video_5): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.999951, frame 500 pkts 8230000:8229000 inflight 504119:506462 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0): throughput 8707.486634 Mb/s: 0.000000 Mb/s, cpu busy 75.494156 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 17 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(6,0:app_tx_video_6): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.999805, frame 500 pkts 8230000:8229000 inflight 504387:506504 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0): throughput 8707.461156 Mb/s: 0.000000 Mb/s, cpu busy 75.516022 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0): dummy pkts 1000, burst 1000 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 14 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-07-30 06:14:18, TX_VIDEO_SESSION(7,0:app_tx_video_7): 2 frames are in trans, total 2 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 0.000000 frames 0 pkts 0 MT: 2024-07-30 06:14:18, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-07-30 06:14:18, E N D S T A T E

After twelve hours:

MT: 2024-07-30 09:26:48, M T D E V S T A T E MT: 2024-07-30 09:26:48, M T D E V S T A T E MT: 2024-07-30 09:26:48, DEV(0): Avr rate, tx: 70437.717104 Mb/s, rx: 34891.614668 Mb/s, pkts, tx: 66567514, rx: 34560119 MT: 2024-07-30 09:26:48, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-07-30 09:26:48, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, CNI(0): eth_rx_rate 0.020463 Mb/s, eth_rx_cnt 278 MT: 2024-07-30 09:26:48, PTP(0): time 1722303002417834486, 2024-07-30 09:29:25 MT: 2024-07-30 09:26:48, PTP(0): delta avg 816, min 790, max 870, cnt 79 MT: 2024-07-30 09:26:48, PTP(0): correct_delta avg 315, min 302, max 362, cnt 79 MT: 2024-07-30 09:26:48, PTP(0): path_delay avg 11, min 7, max 35, cnt 79 MT: 2024-07-30 09:26:48, PTP(0): mode l4, sync cnt 79, expect avg 815:333@0.124979s MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 49.899301, frame 499 pkts 8219740:8218740 inflight 503396:505823 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0): throughput 8696.517132 Mb/s: 0.000000 Mb/s, cpu busy 75.497162 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 22 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0:app_tx_video_1): fps 49.899209, frame 499 pkts 8219748:8218753 inflight 502812:505847 MT: 2024-07-30 09:26:48, DEV(0): Avr rate, tx: 70437.717104 Mb/s, rx: 34891.614668 Mb/s, pkts, tx: 66567514, rx: 34560119 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0): throughput 8696.514823 Mb/s: 0.000000 Mb/s, cpu busy 75.523598 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0): transmitter recalculate warmup 16 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 49.899126, frame 499 pkts 8219768:8218772 inflight 504146:505850 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0): throughput 8696.520421 Mb/s: 0.000000 Mb/s, cpu busy 75.482819 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 20 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 49.899035, frame 499 pkts 8219792:8218792 inflight 502976:505836 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0): throughput 8696.525801 Mb/s: 0.000000 Mb/s, cpu busy 75.540085 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 22 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 49.898945, frame 499 pkts 8219784:8218784 inflight 502411:505780 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0): throughput 8696.501572 Mb/s: 0.000000 Mb/s, cpu busy 75.563988 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 14 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-07-30 09:26:48, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 49.898825, frame 499 pkts 8219816:8218819 inflight 503223:505839 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0): throughput 8696.517802 Mb/s: 0.000000 Mb/s, cpu busy 75.548080 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 19 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.898737, frame 499 pkts 8219836:8218838 inflight 503453:505859 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0): throughput 8696.522435 Mb/s: 0.000000 Mb/s, cpu busy 75.527794 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 21 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.898635, frame 499 pkts 8219840:8218844 inflight 502924:505843 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0): throughput 8696.511154 Mb/s: 0.000000 Mb/s, cpu busy 75.518234 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 13 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 49.998545 frames 500 pkts 8640207 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 8723.152987 Mb/s, cpu busy 1.539625 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 49.998544 frames 500 pkts 8640204 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 8723.149958 Mb/s, cpu busy 1.443185 MT: 2024-07-30 09:26:48, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 49.998545 frames 500 pkts 8640205 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 8723.150968 Mb/s, cpu busy 1.438705 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 49.998543 frames 500 pkts 8640207 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 8723.152987 Mb/s, cpu busy 1.408602 MT: 2024-07-30 09:26:48, E N D S T A T E MT: 2024-07-30 09:26:48, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-07-30 09:26:48, CNI(0): eth_rx_rate 0.020463 Mb/s, eth_rx_cnt 278 MT: 2024-07-30 09:26:48, PTP(0): time 1722303002417834486, 2024-07-30 09:29:25 MT: 2024-07-30 09:26:48, PTP(0): delta avg 816, min 790, max 870, cnt 79 MT: 2024-07-30 09:26:48, PTP(0): correct_delta avg 315, min 302, max 362, cnt 79 MT: 2024-07-30 09:26:48, PTP(0): path_delay avg 11, min 7, max 35, cnt 79 MT: 2024-07-30 09:26:48, PTP(0): mode l4, sync cnt 79, expect avg 815:333@0.124979s MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 49.899301, frame 499 pkts 8219740:8218740 inflight 503396:505823 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0): throughput 8696.517132 Mb/s: 0.000000 Mb/s, cpu busy 75.497162 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 22 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0:app_tx_video_1): fps 49.899209, frame 499 pkts 8219748:8218753 inflight 502812:505847 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0): throughput 8696.514823 Mb/s: 0.000000 Mb/s, cpu busy 75.523598 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0): transmitter recalculate warmup 16 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(1,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 49.899126, frame 499 pkts 8219768:8218772 inflight 504146:505850 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0): throughput 8696.520421 Mb/s: 0.000000 Mb/s, cpu busy 75.482819 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 20 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 49.899035, frame 499 pkts 8219792:8218792 inflight 502976:505836 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0): throughput 8696.525801 Mb/s: 0.000000 Mb/s, cpu busy 75.540085 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 22 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 49.898945, frame 499 pkts 8219784:8218784 inflight 502411:505780 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0): throughput 8696.501572 Mb/s: 0.000000 Mb/s, cpu busy 75.563988 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 14 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 49.898825, frame 499 pkts 8219816:8218819 inflight 503223:505839 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0): throughput 8696.517802 Mb/s: 0.000000 Mb/s, cpu busy 75.548080 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 19 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.898737, frame 499 pkts 8219836:8218838 inflight 503453:505859 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0): throughput 8696.522435 Mb/s: 0.000000 Mb/s, cpu busy 75.527794 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 21 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.898635, frame 499 pkts 8219840:8218844 inflight 502924:505843 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0): throughput 8696.511154 Mb/s: 0.000000 Mb/s, cpu busy 75.518234 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0): dummy pkts 998, burst 998 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 13 MT: 2024-07-30 09:26:48, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 49.998545 frames 500 pkts 8640207 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 8723.152987 Mb/s, cpu busy 1.539625 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 49.998544 frames 500 pkts 8640204 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 8723.149958 Mb/s, cpu busy 1.443185 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 49.998545 frames 500 pkts 8640205 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 8723.150968 Mb/s, cpu busy 1.438705 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 49.998543 frames 500 pkts 8640207 MT: 2024-07-30 09:26:48, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 8723.152987 Mb/s, cpu busy 1.408602 MT: 2024-07-30 09:26:48, E N D S T A T E

Configuration:

{ "interfaces": [ { "name": "0000:be:00.0", "ip": "192.168.10.107" } ], "tx_sessions": [ { "dip": [ "239.6.10.111" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "packing": "BPM", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "video_url": "/home/seeder/dev/ys1_4k_yuv422be10.yuv" } ] }, { "dip": [ "239.6.10.112" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "packing": "BPM", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "video_url": "/home/seeder/dev/ys1_4k_yuv422be10.yuv" } ] }, { "dip": [ "239.6.10.113" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "packing": "BPM", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "video_url": "/home/seeder/dev/ys1_4k_yuv422be10.yuv" } ] }, { "dip": [ "239.6.10.114" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "packing": "BPM", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "video_url": "/home/seeder/dev/ys1_4k_yuv422be10.yuv" } ] }, { "dip": [ "239.6.10.115" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "packing": "BPM", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "video_url": "/home/seeder/dev/ys1_4k_yuv422be10.yuv" } ] }, { "dip": [ "239.6.10.116" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "packing": "BPM", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "video_url": "/home/seeder/dev/ys1_4k_yuv422be10.yuv" } ] }, { "dip": [ "239.6.10.117" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "packing": "BPM", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "video_url": "/home/seeder/dev/ys1_4k_yuv422be10.yuv" } ] }, { "dip": [ "239.6.10.118" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "packing": "BPM", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "video_url": "/home/seeder/dev/ys1_4k_yuv422be10.yuv" } ] } ], "rx_sessions": [ { "ip": [ "239.3.20.1" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "display": false } ] }, { "ip": [ "239.3.20.2" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "display": false } ] }, { "ip": [ "239.3.20.3" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "display": false } ] }, { "ip": [ "239.3.20.4" ], "interface": [ 0 ], "video": [ { "replicas": 1, "type": "frame", "pacing": "gap", "start_port": 20000, "payload_type": 96, "tr_offset": "default", "video_format": "i2160p50", "pg_format": "YUV_422_10bit", "display": false } ] } ] }

blaine0611 commented 4 months ago

./Media-Transport-Library/build/app/RxTxApp --config_file RxTxApp.json --ptp --sch_session_quota 4

frankdjx commented 4 months ago

What I observed is all sessions has mismatch epoch troffset 1 during the 10s report period, I could guess all MLT scheduler are interrupted by other program.

MTL scheduler is just a pinned busy loop thread in user space, if kernel schedule the pinned CPU to other task for a long time, it might see such log as MTL scheduler doesn't has enough time to run the session TX jobs.

Can you try to use the kernel isocpus options? Isolating CPUs can help ensure that these MTL tasks are not preempted by other processes, thus maintaining a high level of reliability. See also https://github.com/OpenVisualCloud/Media-Transport-Library/blob/main/doc/design.md#27-manual-assigned-lcores

blaine0611 commented 4 months ago

Hi @frankdjx ,

Thank you very much for your reply!

I have re-run the code according to your previous reply,But the results seem to be the same.I use the kernel isocpus option to ensure that these MTL tasks are not preempted by other processes.

Configuration:

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-6.5.0-44-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro intel_iommu=on iommu=pt isolcpus=8-28

./Media-Transport-Library/build/app/RxTxApp --config_file RxTxApp.json --lcores 8-28 --ptp --nb_tx_desc 4096 --nb_rx_desc 4096 --sch_session_quota 4

$ taskset -cp 1 pid 1's current affinity list: 0-7,29-63

Log1:

MT: 2024-08-01 03:40:07, M T D E V S T A T E MT: 2024-08-01 03:40:07, M T D E V S T A T E MT: 2024-08-01 03:40:07, DEV(0): Avr rate, tx: 70504.394341 Mb/s, rx: 0.020544 Mb/s, pkts, tx: 66630520, rx: 280 MT: 2024-08-01 03:40:07, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-08-01 03:40:07, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, CNI(0): eth_rx_rate 0.020544 Mb/s, eth_rx_cnt 280 MT: 2024-08-01 03:40:07, PTP(0): time 1722455001677135143, 2024-08-01 03:42:44 MT: 2024-08-01 03:40:07, PTP(0): delta avg 982, min 976, max 991, cnt 80 MT: 2024-08-01 03:40:07, PTP(0): correct_delta avg 638, min 607, max 782, cnt 80 MT: 2024-08-01 03:40:07, PTP(0): path_delay avg 10, min 7, max 17, cnt 80 MT: 2024-08-01 03:40:07, PTP(0): mode l4, sync cnt 80, expect avg 982:742@0.124979s MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 49.999982, frame 500 pkts 8229940:8228943 inflight 437141:436161 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0): throughput 8707.431713 Mb/s: 0.000000 Mb/s, cpu busy 78.773712 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 15 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0:app_tx_video_0): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, DEV(0): Avr rate, tx: 70504.394341 Mb/s, rx: 0.020544 Mb/s, pkts, tx: 66630520, rx: 280 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0:app_tx_video_1): fps 49.999910, frame 500 pkts 8229944:8228942 inflight 437655:436763 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0): throughput 8707.418140 Mb/s: 0.000000 Mb/s, cpu busy 78.749390 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0): transmitter recalculate warmup 13 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0:app_tx_video_1): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 49.999788, frame 500 pkts 8229972:8228973 inflight 437616:436716 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0): throughput 8707.429687 Mb/s: 0.000000 Mb/s, cpu busy 78.752411 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 13 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0:app_tx_video_2): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 49.999684, frame 500 pkts 8229988:8228997 inflight 437579:436664 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0): throughput 8707.436960 Mb/s: 0.000000 Mb/s, cpu busy 78.752502 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 20 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0:app_tx_video_3): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 49.999576, frame 500 pkts 8230012:8229012 inflight 437579:436659 MT: 2024-08-01 03:40:07, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0): throughput 8707.434068 Mb/s: 0.000000 Mb/s, cpu busy 78.753189 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 18 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0:app_tx_video_4): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 49.999464, frame 500 pkts 8230000:8229000 inflight 437651:436757 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0): throughput 8707.401807 Mb/s: 0.000000 Mb/s, cpu busy 78.748520 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 9 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0:app_tx_video_5): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.999372, frame 500 pkts 8230000:8229000 inflight 437598:436700 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0): throughput 8707.385850 Mb/s: 0.000000 Mb/s, cpu busy 78.752792 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 9 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0:app_tx_video_6): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.999257, frame 500 pkts 8230000:8229000 inflight 437801:436915 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0): throughput 8707.365785 Mb/s: 0.000000 Mb/s, cpu busy 78.742294 MT: 2024-08-01 03:40:07, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 9 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0:app_tx_video_7): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 03:40:07, E N D S T A T E MT: 2024-08-01 03:40:07, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-08-01 03:40:07, CNI(0): eth_rx_rate 0.020544 Mb/s, eth_rx_cnt 280 MT: 2024-08-01 03:40:07, PTP(0): time 1722455001677135143, 2024-08-01 03:42:44 MT: 2024-08-01 03:40:07, PTP(0): delta avg 982, min 976, max 991, cnt 80 MT: 2024-08-01 03:40:07, PTP(0): correct_delta avg 638, min 607, max 782, cnt 80 MT: 2024-08-01 03:40:07, PTP(0): path_delay avg 10, min 7, max 17, cnt 80 MT: 2024-08-01 03:40:07, PTP(0): mode l4, sync cnt 80, expect avg 982:742@0.124979s MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 49.999982, frame 500 pkts 8229940:8228943 inflight 437141:436161 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0): throughput 8707.431713 Mb/s: 0.000000 Mb/s, cpu busy 78.773712 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 15 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(0,0:app_tx_video_0): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0:app_tx_video_1): fps 49.999910, frame 500 pkts 8229944:8228942 inflight 437655:436763 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0): throughput 8707.418140 Mb/s: 0.000000 Mb/s, cpu busy 78.749390 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0): transmitter recalculate warmup 13 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(1,0:app_tx_video_1): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 49.999788, frame 500 pkts 8229972:8228973 inflight 437616:436716 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0): throughput 8707.429687 Mb/s: 0.000000 Mb/s, cpu busy 78.752411 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 13 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(2,0:app_tx_video_2): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 49.999684, frame 500 pkts 8229988:8228997 inflight 437579:436664 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0): throughput 8707.436960 Mb/s: 0.000000 Mb/s, cpu busy 78.752502 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 20 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(3,0:app_tx_video_3): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 49.999576, frame 500 pkts 8230012:8229012 inflight 437579:436659 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0): throughput 8707.434068 Mb/s: 0.000000 Mb/s, cpu busy 78.753189 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 18 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(4,0:app_tx_video_4): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 49.999464, frame 500 pkts 8230000:8229000 inflight 437651:436757 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0): throughput 8707.401807 Mb/s: 0.000000 Mb/s, cpu busy 78.748520 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 9 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(5,0:app_tx_video_5): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.999372, frame 500 pkts 8230000:8229000 inflight 437598:436700 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0): throughput 8707.385850 Mb/s: 0.000000 Mb/s, cpu busy 78.752792 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 9 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(6,0:app_tx_video_6): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.999257, frame 500 pkts 8230000:8229000 inflight 437801:436915 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0): throughput 8707.365785 Mb/s: 0.000000 Mb/s, cpu busy 78.742294 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 9 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-08-01 03:40:07, TX_VIDEO_SESSION(7,0:app_tx_video_7): 2 frames are in trans, total 2 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 03:40:07, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 03:40:07, E N D S T A T E

Log2:

MT: 2024-08-01 04:41:17, M T D E V S T A T E MT: 2024-08-01 04:41:17, M T D E V S T A T E MT: 2024-08-01 04:41:17, DEV(0): Avr rate, tx: 70504.680061 Mb/s, rx: 0.020544 Mb/s, pkts, tx: 66630790, rx: 280 MT: 2024-08-01 04:41:17, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-08-01 04:41:17, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, CNI(0): eth_rx_rate 0.020544 Mb/s, eth_rx_cnt 280 MT: 2024-08-01 04:41:17, PTP(0): time 1722458671687286551, 2024-08-01 04:43:54 MT: 2024-08-01 04:41:17, PTP(0): delta avg 986, min 979, max 1018, cnt 80 MT: 2024-08-01 04:41:17, PTP(0): correct_delta avg 846, min 769, max 987, cnt 80 MT: 2024-08-01 04:41:17, PTP(0): path_delay avg 10, min 6, max 14, cnt 80 MT: 2024-08-01 04:41:17, PTP(0): mode l4, sync cnt 80, expect avg 994:946@0.125968s MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 49.999917, frame 500 pkts 8229952:8228945 inflight 437159:436188 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0): throughput 8707.422611 Mb/s: 0.000000 Mb/s, cpu busy 78.768997 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 13 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 50.000006, frame 500 pkts 8229936:8228923 inflight 437602:436707 MT: 2024-08-01 04:41:17, DEV(0): Avr rate, tx: 70504.680061 Mb/s, rx: 0.020544 Mb/s, pkts, tx: 66630790, rx: 280 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0): throughput 8707.414842 Mb/s: 0.000000 Mb/s, cpu busy 78.747910 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 11 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 49.999900, frame 500 pkts 8229952:8228942 inflight 437606:436715 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0): throughput 8707.416417 Mb/s: 0.000000 Mb/s, cpu busy 78.749069 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 13 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 49.999800, frame 500 pkts 8229964:8228968 inflight 437598:436697 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0): throughput 8707.426489 Mb/s: 0.000000 Mb/s, cpu busy 78.749947 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 13 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 49.999724, frame 500 pkts 8229988:8228987 inflight 437667:436787 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0): throughput 8707.433397 Mb/s: 0.000000 Mb/s, cpu busy 78.743530 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 11 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-08-01 04:41:17, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.999622, frame 500 pkts 8230004:8229006 inflight 437609:436727 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0): throughput 8707.435745 Mb/s: 0.000000 Mb/s, cpu busy 78.748489 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 13 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.999544, frame 500 pkts 8230024:8229026 inflight 437888:437000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0): throughput 8707.443221 Mb/s: 0.000000 Mb/s, cpu busy 78.733910 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 17 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 04:41:17, E N D S T A T E MT: 2024-08-01 04:41:17, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-08-01 04:41:17, CNI(0): eth_rx_rate 0.020544 Mb/s, eth_rx_cnt 280 MT: 2024-08-01 04:41:17, PTP(0): time 1722458671687286551, 2024-08-01 04:43:54 MT: 2024-08-01 04:41:17, PTP(0): delta avg 986, min 979, max 1018, cnt 80 MT: 2024-08-01 04:41:17, PTP(0): correct_delta avg 846, min 769, max 987, cnt 80 MT: 2024-08-01 04:41:17, PTP(0): path_delay avg 10, min 6, max 14, cnt 80 MT: 2024-08-01 04:41:17, PTP(0): mode l4, sync cnt 80, expect avg 994:946@0.125968s MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 49.999917, frame 500 pkts 8229952:8228945 inflight 437159:436188 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0): throughput 8707.422611 Mb/s: 0.000000 Mb/s, cpu busy 78.768997 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 13 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 50.000006, frame 500 pkts 8229936:8228923 inflight 437602:436707 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0): throughput 8707.414842 Mb/s: 0.000000 Mb/s, cpu busy 78.747910 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 11 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 49.999900, frame 500 pkts 8229952:8228942 inflight 437606:436715 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0): throughput 8707.416417 Mb/s: 0.000000 Mb/s, cpu busy 78.749069 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 13 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 49.999800, frame 500 pkts 8229964:8228968 inflight 437598:436697 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0): throughput 8707.426489 Mb/s: 0.000000 Mb/s, cpu busy 78.749947 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 13 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 49.999724, frame 500 pkts 8229988:8228987 inflight 437667:436787 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0): throughput 8707.433397 Mb/s: 0.000000 Mb/s, cpu busy 78.743530 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 11 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.999622, frame 500 pkts 8230004:8229006 inflight 437609:436727 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0): throughput 8707.435745 Mb/s: 0.000000 Mb/s, cpu busy 78.748489 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 13 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.999544, frame 500 pkts 8230024:8229026 inflight 437888:437000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0): throughput 8707.443221 Mb/s: 0.000000 Mb/s, cpu busy 78.733910 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 17 MT: 2024-08-01 04:41:17, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 04:41:17, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 04:41:17, E N D S T A T E

Log3:

MT: 2024-08-01 06:53:47, M T D E V S T A T E MT: 2024-08-01 06:53:47, M T D E V S T A T E MT: 2024-08-01 06:53:47, DEV(0): Avr rate, tx: 70504.973190 Mb/s, rx: 0.020544 Mb/s, pkts, tx: 66631068, rx: 280 MT: 2024-08-01 06:53:47, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-08-01 06:53:47, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, CNI(0): eth_rx_rate 0.020544 Mb/s, eth_rx_cnt 280 MT: 2024-08-01 06:53:47, PTP(0): time 1722466621695738878, 2024-08-01 06:56:24 MT: 2024-08-01 06:53:47, PTP(0): delta avg 996, min 986, max 1040, cnt 80 MT: 2024-08-01 06:53:47, PTP(0): correct_delta avg 1064, min 934, max 1152, cnt 80 MT: 2024-08-01 06:53:47, PTP(0): path_delay avg 11, min 7, max 26, cnt 80 MT: 2024-08-01 06:53:47, PTP(0): mode l4, sync cnt 80, expect avg 996:1100@0.125068s MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 49.999678, frame 500 pkts 8229980:8228982 inflight 437092:436079 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0): throughput 8707.420071 Mb/s: 0.000000 Mb/s, cpu busy 78.773758 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 22 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0:app_tx_video_1): fps 49.999581, frame 500 pkts 8230004:8229006 inflight 437632:436714 MT: 2024-08-01 06:53:47, DEV(0): Avr rate, tx: 70504.973190 Mb/s, rx: 0.020544 Mb/s, pkts, tx: 66631068, rx: 280 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0): throughput 8707.428545 Mb/s: 0.000000 Mb/s, cpu busy 78.746742 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0): transmitter recalculate warmup 13 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 49.999482, frame 500 pkts 8230020:8229017 inflight 437573:436642 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0): throughput 8707.422944 Mb/s: 0.000000 Mb/s, cpu busy 78.749847 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 9 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 49.999376, frame 500 pkts 8230052:8229050 inflight 437542:436607 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0): throughput 8707.439454 Mb/s: 0.000000 Mb/s, cpu busy 78.751305 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 10 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 49.999299, frame 500 pkts 8230056:8229054 inflight 437574:436626 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0): throughput 8707.430212 Mb/s: 0.000000 Mb/s, cpu busy 78.751404 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 13 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-08-01 06:53:47, SCH(0:sch_0): tasklets 3 max idx 3 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 49.999187, frame 500 pkts 8230068:8229068 inflight 437611:436696 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0): throughput 8707.425596 Mb/s: 0.000000 Mb/s, cpu busy 78.747612 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 12 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.999082, frame 500 pkts 8230092:8229092 inflight 437555:436622 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0): throughput 8707.432755 Mb/s: 0.000000 Mb/s, cpu busy 78.751305 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 19 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.998960, frame 500 pkts 8230124:8229126 inflight 437776:436863 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0): throughput 8707.447339 Mb/s: 0.000000 Mb/s, cpu busy 78.739647 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 20 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 06:53:47, SCH(1:sch_1): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 06:53:47, E N D S T A T E MT: 2024-08-01 06:53:47, SCH(2:sch_2): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(3:sch_3): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(4:sch_4): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(5:sch_5): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(6:sch_6): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(7:sch_7): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(8:sch_8): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(9:sch_9): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(10:sch_10): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, SCH(11:sch_11): tasklets 2 max idx 2 MT: 2024-08-01 06:53:47, CNI(0): eth_rx_rate 0.020544 Mb/s, eth_rx_cnt 280 MT: 2024-08-01 06:53:47, PTP(0): time 1722466621695738878, 2024-08-01 06:56:24 MT: 2024-08-01 06:53:47, PTP(0): delta avg 996, min 986, max 1040, cnt 80 MT: 2024-08-01 06:53:47, PTP(0): correct_delta avg 1064, min 934, max 1152, cnt 80 MT: 2024-08-01 06:53:47, PTP(0): path_delay avg 11, min 7, max 26, cnt 80 MT: 2024-08-01 06:53:47, PTP(0): mode l4, sync cnt 80, expect avg 996:1100@0.125068s MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0:app_tx_video_0): fps 49.999678, frame 500 pkts 8229980:8228982 inflight 437092:436079 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0): throughput 8707.420071 Mb/s: 0.000000 Mb/s, cpu busy 78.773758 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0): transmitter recalculate warmup 22 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(0,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0:app_tx_video_1): fps 49.999581, frame 500 pkts 8230004:8229006 inflight 437632:436714 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0): throughput 8707.428545 Mb/s: 0.000000 Mb/s, cpu busy 78.746742 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0): transmitter recalculate warmup 13 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(1,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0:app_tx_video_2): fps 49.999482, frame 500 pkts 8230020:8229017 inflight 437573:436642 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0): throughput 8707.422944 Mb/s: 0.000000 Mb/s, cpu busy 78.749847 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0): transmitter recalculate warmup 9 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(2,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0:app_tx_video_3): fps 49.999376, frame 500 pkts 8230052:8229050 inflight 437542:436607 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0): throughput 8707.439454 Mb/s: 0.000000 Mb/s, cpu busy 78.751305 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0): transmitter recalculate warmup 10 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(3,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0:app_tx_video_4): fps 49.999299, frame 500 pkts 8230056:8229054 inflight 437574:436626 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0): throughput 8707.430212 Mb/s: 0.000000 Mb/s, cpu busy 78.751404 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0): transmitter recalculate warmup 13 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(4,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0:app_tx_video_5): fps 49.999187, frame 500 pkts 8230068:8229068 inflight 437611:436696 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0): throughput 8707.425596 Mb/s: 0.000000 Mb/s, cpu busy 78.747612 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0): transmitter recalculate warmup 12 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(5,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0:app_tx_video_6): fps 49.999082, frame 500 pkts 8230092:8229092 inflight 437555:436622 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0): throughput 8707.432755 Mb/s: 0.000000 Mb/s, cpu busy 78.751305 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0): transmitter recalculate warmup 19 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(6,0): epoch drop 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0:app_tx_video_7): fps 49.998960, frame 500 pkts 8230124:8229126 inflight 437776:436863 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0): throughput 8707.447339 Mb/s: 0.000000 Mb/s, cpu busy 78.739647 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0): dummy pkts 1000, burst 1000 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0): mismatch epoch troffset 1 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0): transmitter recalculate warmup 20 MT: 2024-08-01 06:53:47, TX_VIDEO_SESSION(7,0): epoch drop 1 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(8,0:app_rx_video_0): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(8,0:app_rx_video_0): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(9,0:app_rx_video_1): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(9,0:app_rx_video_1): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(10,0:app_rx_video_2): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(10,0:app_rx_video_2): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(11,0:app_rx_video_3): fps 0.000000 frames 0 pkts 0 MT: 2024-08-01 06:53:47, RX_VIDEO_SESSION(11,0:app_rx_video_3): throughput 0.000000 Mb/s, cpu busy 0.000000 MT: 2024-08-01 06:53:47, E N D S T A T E

frankdjx commented 4 months ago

Not too much ideals.

Can you help to check if this issue still happens if reduce the traffic, ex, run just one TX and RX?

And also please enable --tasklet_time option to the RxTxApp to allow more timing information in the log to help us know more detail info.

blaine0611 commented 3 months ago

I have re-run the code and reduce the traffic, run just one TX. As a result, there seems to be no more epoch drops in the log.

frankdjx commented 3 months ago

I have re-run the code and reduce the traffic, run just one TX. As a result, there seems to be no more epoch drops in the log.

Then can we identify the max traffic for current setup?

And BTW honestly I has to say IMTL is a software based solution, thus only 1 frame drop for 8 hours is a good result per my personal view.