Customer Case- HLS streams do not play for SRT broadcast after 24 hours unless stream is restarted

[Mohit-3196]

When broadcasting SRT streams in a 7x24 environment, the HLS streams don't play continuously and start to stop/buff a lot. The .ts files however are getting generated fine. WebRTC playback seems fine. The pattern repeats every 24-48 hours unless the broadcast is restarted. It behaves in the same way for both the Origin as well as Edge nodes.

Environment

Ant Media Server v2.7.0 AMS Cluster on AWS SRT Publish Encoders: Sencore and PVI

Logs: https://drive.google.com/drive/folders/1ijfXwIJwg-E8mVCBqaiOogruC27xp8Os?usp=sharing

[dmillerhotwire]

On Saturday, we also saw this in the AntMediaLogs, so we disabled audio. The feed for Palm Chase went down again.

2023-12-09 20:04:17,092 [vert.x-worker-thread-72] INFO io.antmedia.muxer.HLSMuxer - cannot write audio frame to muxer(Palm1.m3u8). Error is Invalid argument 2023-12-09 20:04:17,092 [vert.x-worker-thread-72] INFO io.antmedia.muxer.HLSMuxer - cannot write audio frame to muxer(Palm1.m3u8). Error is Invalid argument 2023-12-09 20:04:17,093 [vert.x-worker-thread-72] INFO io.antmedia.muxer.HLSMuxer - cannot write audio frame to muxer(Palm1.m3u8). Error is Invalid argument 2023-12-09 20:04:17,093 [vert.x-worker-thread-72] INFO io.antmedia.muxer.HLSMuxer - cannot write audio frame to muxer(Palm1.m3u8). Error is Invalid argument 2023-12-09 20:04:17,093 [vert.x-worker-thread-72] WARN io.antmedia.muxer.HLSMuxer - cannot write video frame to muxer(Palm1.m3u8) av_bsf_receive_packet. Error is Invalid argument 2023-12-09 20:04:17,098 [vert.x-worker-thread-72] WARN io.antmedia.muxer.HLSMuxer - cannot write video frame to muxer(Palm1.m3u8) av_bsf_receive_packet. Error is Invalid argument 2023-12-09 20:04:17,098 [vert.x-worker-thread-72] WARN io.antmedia.muxer.HLSMuxer - cannot write video frame to muxer(Palm1.m3u8) av_bsf_receive_packet. Error is Invalid argument 2023-12-09 20:04:17,098 [vert.x-worker-thread-72] WARN io.antmedia.muxer.HLSMuxer - cannot write video frame to muxer(Palm1.m3u8) av_bsf_receive_packet. Error is Invalid argument

[lastpeony]

The problem arises due to DTS overflow in SRT packets. After 24+ hours, this overflow manifests, causing incoming packet DTS values to become lower than the last received DTS. Consequently, SRT packets fail to be written to the HLS muxer. For instance, a quick fix trial involving adding +1 to the problematic packet DTS to ensure it exceeds the last received DTS doesn't trigger an error in the HLS muxer. However, the resulting HLS data is corrupted, as anticipated.

I'm currently delving into potential solutions to address this overflow issue, drawing from strategies we've employed previously to tackle overflow issues on the WebRTC side.

[lastpeony]

Fix: https://gitlab.com/Ant-Media/Ant-Media-Enterprise/-/merge_requests/694

[mekya]

Thank you for the fix and collaboration @lastpeony @burak-58

It's merged into the master.

FYI