OpenVisualCloud / Media-Transport-Library

A real-time media transport(DPDK, AF_XDP, RDMA) stack for both raw and compressed video based on COTS hardware.
BSD 3-Clause "New" or "Revised" License
155 stars 47 forks source link

Fix RX side dropping frames at the beginning. #917

Closed DawidWesierski4 closed 1 day ago

DawidWesierski4 commented 1 week ago

Address an issue where the RX is sending ARP replies to the TX while still being completely uninitialized, which leads to the session dropping frames. The problem worsens as the TX outperforms the RX in terms of initialization speed.

Resolve this by halting the FFmpeg plugin initialization and starting the scheduled tasklets because we don't want the CNI thread to respond to the ARP requests sent in the tv_attach function until the RX is ready to receive those packets.

frankdjx commented 1 week ago

Not understand why it need this change. What's the problem if some frames are missing at the initial stage? Please note the typical case in video production is using multicast address, no one will use unicast address. Even with this change, the similar drop for the first frame parsing is still there with multicast mode.

DawidWesierski4 commented 1 week ago

Although the scenario is unlikely, isn't less frame drop better?

frankdjx commented 1 week ago

It doesn't make sense, it has more codes but not fix anything really.

DawidWesierski4 commented 1 week ago

It doesn't make sense, it has more codes but not fix anything really.

In my testing if the TX outperforms the TX the RX Informs that the first received packet does not start with the sequence number 0

With this change the RX side will start responding to ARP messages and will start the initialization of the TX session after the RX session is initialized

Although this only occurs with unicast it makes it so that the RX side receives the first packets after this change

I can provide the logs confirming this

To replicate the issue the easiest way is to have debug log enabled and a very high performance / low latency scenario You will see the first received packet would not be 0 and a few frames dropped at the beginning

The issue is bigger the more TX outperforms the RX (it allows for faster initialization )

DawidWesierski4 commented 1 week ago

The fix works basically via moving the mtl_start that enables the ARP response further down the line of RX initialization in ffmpeg as for now the RX side is not catching up to the TX side

here an example where the first recived packet has the p_counter 189

MTL: 2024-06-17 22:18:08, rv_attach(0), w 1920 h 1080 fmt ST20_FMT_YUV_422_10BIT packing 0 pt 112 flags 0x0 frame time 40.000000ms fps 25.000000
MTL: 2024-06-17 22:18:08, st22_rx_create, succ on sch 0 session 0
MTL: 2024-06-17 22:18:08, rv_slot_by_tmstamp(0): new tmstamp 41793993
MTL: 2024-06-17 22:18:08, rv_get_frame(0), find frame at 0
MTL: 2024-06-17 22:18:08, rv_slot_by_tmstamp(0): assign slot 0 framebuff 0x3205a17000 for tmstamp 41793993
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3227 kmode 0 trans_order 1
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3227 p_counter 187 sep_counter 0
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), get seq_id 3227 tmstamp 41793993, p_counter 187 sep_counter 0, payload_length 1280
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3228 kmode 0 trans_order 1
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3228 p_counter 188 sep_counter 0
MTL: 2024-06-17 22:18:08, st22p_rx_create(0), codestream fmt JPEGXS_CODESTREAM, output fmt: YUV422PLANAR10LE, flags 0x8000
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3229 kmode 0 trans_order 1
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3229 p_counter 189 sep_counter 0

I did not experience this after this change

The video received usually had a few frames cropped from the beginning

frankdjx commented 6 days ago

Please help to fix lint fail.

/github/workspace/ecosystem/ffmpeg_plugin/mtl_st22p_rx.c:318:7: error: code should be clang-formatted [-Wclang-format-violations]
    /* 
      ^
/github/workspace/ecosystem/ffmpeg_plugin/mtl_st22p_rx.c:324:17: error: code should be clang-formatted [-Wclang-format-violations]
      if (frame)

Just run ./foramt_coding.sh