Closed weic6 closed 1 month ago
I see there are some critical errors(queue can't burst any packets to network) in the FFMpeg TX side.
MTL: 2024-07-16 17:06:33, Error: video_trs_burst_fail(0,0), hang duration 1000 ms
MTL: 2024-07-16 17:06:33, mt_ring_dequeue_clean, count 508 for ring TV_M0S0P0
MTL: 2024-07-16 17:06:33, Error: mt_dev_tx_queue_fatal_error(0), q 1 masked as fatal error
Then the TX stop working after many retries.
It's weird that ./build/app/RxTxApp --config_file config/tx_1v.json
is working well, can you paste the working log also?
And can you try start RX first and then TX? Is it the same behavior?
Dear Frank,
Thanks for your reply.
It's weird that
./build/app/RxTxApp --config_file config/tx_1v.json
is working well, can you paste the working log also?
Below are the outputs for RxTxApp.
tx_1v.json
{
"interfaces": [
{
"name": "0000:01:00.0",
"ip": "192.168.30.10"
}
],
"tx_sessions": [
{
"dip": [
"239.1.1.1"
],
"interface": [
0
],
"st20p": [
{
"replicas": 1,
"start_port": 20000,
"payload_type": 112,
"width": 1920,
"height": 1080,
"fps": "p59",
"device": "AUTO",
"input_format": "YUV422PLANAR10LE",
"transport_format": "YUV_422_10bit",
"st20p_url": "./bunny_yuv422p10le_1080p_60fps_60frame.yuv"
}
]
}
]
}
rx_1v.json
{
"interfaces": [
{
"name": "0000:01:00.0",
"ip": "192.168.30.12"
}
],
"rx_sessions": [
{
"ip": [
"239.1.1.1"
],
"interface": [
0
],
"st20p": [
{
"replicas": 1,
"start_port": 20000,
"payload_type": 112,
"width": 1920,
"height": 1080,
"fps": "p59",
"device": "AUTO",
"output_format": "YUV422PLANAR10LE",
"transport_format": "YUV_422_10bit",
"display": false,
"measure_latency": true
}
]
}
]
}
serverB $ ./build/app/RxTxApp --config_file config/tx_1v.json
Tx output Tx output-1.txt
kernel buffer logs for Tx
$ sudo dmesg
[601399.928285] vfio-pci 0000:01:00.0: Masking broken INTx support
[601399.928422] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x1d0
serverA $ ./build/app/RxTxApp --config_file config/rx_1v.json
Rx output Rx output-1.txt
No new kernel buffer logs for Rx
serverA $ ./build/app/RxTxApp --config_file config/rx_1v.json
Rx output Rx output-2.txt
No new kernel buffer logs for Rx
serverB $ ./build/app/RxTxApp --config_file config/tx_1v.json
Tx output Tx output-2.txt
kernel buffer logs for Tx
$ sudo dmesg
[603161.708825] vfio-pci 0000:01:00.0: Masking broken INTx support
[603161.708962] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x1d0
And can you try start RX first and then TX? Is it the same behavior?
Yes, i also see incomplete frames at Rx, and i still see Error: video_trs_burst_fail
at Tx side. Below are the detail outputs.
serverA $ ffmpeg -p_port 0000:01:00.0 -p_sip 192.168.30.12 -p_rx_ip 239.1.1.1 -udp_port 20000 -payload_type 112 -fps 30 -pix_fmt yuv422p10le -video_size 1920x1080 -f mtl_st20p -i "k" -f rawvideo /dev/null -y
Rx output ffmpeg-Rx output.txt
No new kernel buffer logs for Rx
serverB $ ffmpeg -f lavfi -i "smptehdbars=rate=30:size=1920x1080" -f rawvideo -pix_fmt yuv422p10le -p_port 0000:01:00.0 -p_sip 192.168.30.10 -p_tx_ip 239.1.1.1 -udp_port 20000 -payload_type 112 -f mtl_st20p -
Tx output ffmpeg-Tx output.txt
kernel buffer logs for Tx
[603884.234316] vfio-pci 0000:01:00.0: Masking broken INTx support
[603884.234453] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x1d0
Seems the problem is FFMpeg TX is not working well as some unknow reason.
Can you try run FFMpeg RX
on server A and ./build/app/RxTxApp --config_file config/tx_1v.json
on server B?
And I has compared Tx.output-1.txt
and ffmpeg-Tx.output.txt
, the initial log is nearly very similar. ffmpeg-Tx
is sending 127 frames successfully but then suddenly get fatal error.
MTL: 2024-07-17 10:47:05, TX_VIDEO_SESSION(0,0:st20p_ffmpge): fps 29.734454 frames 127 pkts 525553:0 inflight 130777:131292
MTL: 2024-07-17 10:47:05, TX_VIDEO_SESSION(0,0): throughput 1301.332531 Mb/s: 0.000000 Mb/s, cpu busy 0.000000
MTL: 2024-07-17 10:47:05, TX_VIDEO_SESSION(0,0): busy as no ready frame from user 1
MTL: 2024-07-17 10:47:05, TX_st20p(0,st20p_ffmpge), p(1:in_transmitting) c(2:converted)
MTL: 2024-07-17 10:47:05, TX_st20p(0), frame get try 131 succ 130, put 130
MTL: 2024-07-17 10:47:05, * * E N D S T A T E * *
[mtl_st20p @ 0x560c5be9b8c0] mtl_st20p_write_packet(0), st20p_tx_get_frame timeout
[vost#0:0/rawvideo @ 0x560c5be9c680] Error submitting a packet to the muxer: Input/output error
[out#0/mtl_st20p @ 0x560c5be9b440] Error muxing a packet
MTL: 2024-07-17 10:47:12, st20p_tx_free(0), start
MTL: 2024-07-17 10:47:12, Error: video_trs_burst_fail(0,0), hang duration 1000 ms
MTL: 2024-07-17 10:47:12, mt_ring_dequeue_clean, count 508 for ring TV_M0S0P0
Thanks for your reply. I have conducted more tests according to your suggestions. Based on the results, i am more inclined to believe that the whole transmission aborts when Rx, using FFmpeg, sees incomplete frames at the beginning. Maybe this issue is not caused by Tx side?
serverA $ ffmpeg -p_port 0000:01:00.0 -p_sip 192.168.30.12 -p_rx_ip 239.1.1.1 -udp_port 20000 -payload_type 112 -fps 60 -pix_fmt yuv422p10le -video_size 1920x1080 -f mtl_st20p -i "k" -f rawvideo /dev/null -y
ffmpeg version n6.1.1-157-g1b7ddd6747 Copyright (c) 2000-2023 the FFmpeg developers
serverB $ ./build/app/RxTxApp --config_file config/tx_1v.json
serverB $ ./build/app/RxTxApp --config_file config/tx_1v.json
serverA $ ffmpeg -p_port 0000:01:00.0 -p_sip 192.168.30.12 -p_rx_ip 239.1.1.1 -udp_port 20000 -payload_type 112 -fps 60 -pix_fmt yuv422p10le -video_size 1920x1080 -f mtl_st20p -i "k" -f rawvideo /dev/null -y
serverA $ ./build/app/RxTxApp --config_file config/rx_1v.json
serverB $ ffmpeg -f lavfi -i "smptehdbars=rate=60:size=1920x1080" -f rawvideo -pix_fmt yuv422p10le -p_port 0000:01:00.0 -p_sip 192.168.30.10 -p_tx_ip 239.1.1.1 -udp_port 20000 -payload_type 112 -f mtl_st20p -
serverB $ ffmpeg -f lavfi -i "smptehdbars=rate=60:size=1920x1080" -f rawvideo -pix_fmt yuv422p10le -p_port 0000:01:00.0 -p_sip 192.168.30.10 -p_tx_ip 239.1.1.1 -udp_port 20000 -payload_type 112 -f mtl_st20p -
serverA $ ./build/app/RxTxApp --config_file config/rx_1v.json
I see there are multi segments pkts
logs in the FFMpeg RX which are not supported by current MTL RX logical.
MTL: 2024-07-18 14:12:08, RX_VIDEO_SESSION(0,0): multi segments pkts 255068
MT_DRV_IGC
has similar problem, can you change https://github.com/OpenVisualCloud/Media-Transport-Library/blob/main/lib/src/dev/mt_dev.c#L1271
to adding the WA for MT_DRV_IXGBE
also?
/* to avoid igc/igxbe nic split mbuf */
if (inf->drv_info.drv_type == MT_DRV_IGC || inf->drv_info.drv_type == MT_DRV_IXGBE)
Not sure if it will fix the issue or not. But multi segments
is definitely a problem.
Dear Frank,
Thanks for pointing this out. i make relevant modification according to your suggestion. I see that now FFmpeg Rx and FFmpeg Tx is working normally, in either order. i am happy to submit a PR for ixgbe support if you think it is necessary.
I really appreciate your help. Orz.
Best Wei
File attached is a screenshot for "Running FFmpeg Rx first and then FFmpeg Tx"
Great to know it's the root cause.
Appreciate if you could help to upstream the fix. The latter IGXBE people can survive from this kind of weird issue. Thanks.
Dear Frank, i just created a PR #932 . Thank for the help again. I am closing this issue now.
Dear MTL team,
Thanks for sharing your great work. I am trying to set up MTL library in both of my servers. i plan to use the FFmpeg plugin to transport ST2110 media.
I have two servers (A, B), where server A uses Ethernet Controller 10-Gigabit X540-AT2 1528 and server B uses Ethernet Controller 10G X550T 1563. Both NICs use ixgbe driver, which is supported by DPDK. For now, I connected X540-AT2 NIC port to X550 port directly with a cable 5e, just for testing purpose.
To run your application, I have followed the build guide (except that I choose to bind PF to DPDK PMD, not VF) and run guide. I have added
MT_DRV_IXGBE
inenum mt_driver_type
, and I also add the following instruct mt_dev_driver_info dev_drvs
The sample application RxTxApp can run successfully between the two servers using commands.
I can also view the transmitted video at the Rx side using
python3 python/example/st20p_rx.py
.However, when i try to run the MTL-built FFmpeg with the following commands, the Rx side always gets incomplete frames at the beginning of transmission. I wonder if you could provide any insight about how to make ffmpeg work for my case. Thank you.
FFmpeg Commands
Tx command
Tx output
Rx command
Rx output
Logs obtained from
./script/status_report.sh
Rx at serverA (with intel X540-AT2)
Rx kernel_log.txt
Rx system_info.txt Rx_system_info.txt
Tx at serverB (with intel X550-AT2)
Tx kernel_log.txt
Tx system_info.txt Tx_system_info.txt