search

ant-media / Ant-Media-Server

Ant Media Server is a live streaming engine software that provides adaptive, ultra low latency streaming by using WebRTC technology with ~0.5 seconds latency. Ant Media Server is auto-scalable and it can run on-premise or on-cloud.
https://antmedia.io
Other
4.3k stars 636 forks source link

Customer Case- HLS streams do not play for SRT broadcast after 24 hours unless stream is restarted #5913

Open Mohit-3196 opened 11 months ago

Mohit-3196 commented 11 months ago

When broadcasting SRT streams in a 7x24 environment, the HLS streams don't play continuously and start to stop/buff a lot. The .ts files however are getting generated fine. WebRTC playback seems fine. The pattern repeats every 24-48 hours unless the broadcast is restarted. It behaves in the same way for both the Origin as well as Edge nodes.

Environment

Ant Media Server v2.7.0 AMS Cluster on AWS SRT Publish Encoders: Sencore and PVI

Logs: https://drive.google.com/drive/folders/1ijfXwIJwg-E8mVCBqaiOogruC27xp8Os?usp=sharing

dmillerhotwire commented 11 months ago

On Saturday, we also saw this in the AntMediaLogs, so we disabled audio. The feed for Palm Chase went down again.

2023-12-09 20:04:17,092 [vert.x-worker-thread-72] INFO io.antmedia.muxer.HLSMuxer - cannot write audio frame to muxer(Palm1.m3u8). Error is Invalid argument 2023-12-09 20:04:17,092 [vert.x-worker-thread-72] INFO io.antmedia.muxer.HLSMuxer - cannot write audio frame to muxer(Palm1.m3u8). Error is Invalid argument 2023-12-09 20:04:17,093 [vert.x-worker-thread-72] INFO io.antmedia.muxer.HLSMuxer - cannot write audio frame to muxer(Palm1.m3u8). Error is Invalid argument 2023-12-09 20:04:17,093 [vert.x-worker-thread-72] INFO io.antmedia.muxer.HLSMuxer - cannot write audio frame to muxer(Palm1.m3u8). Error is Invalid argument 2023-12-09 20:04:17,093 [vert.x-worker-thread-72] WARN io.antmedia.muxer.HLSMuxer - cannot write video frame to muxer(Palm1.m3u8) av_bsf_receive_packet. Error is Invalid argument 2023-12-09 20:04:17,098 [vert.x-worker-thread-72] WARN io.antmedia.muxer.HLSMuxer - cannot write video frame to muxer(Palm1.m3u8) av_bsf_receive_packet. Error is Invalid argument 2023-12-09 20:04:17,098 [vert.x-worker-thread-72] WARN io.antmedia.muxer.HLSMuxer - cannot write video frame to muxer(Palm1.m3u8) av_bsf_receive_packet. Error is Invalid argument 2023-12-09 20:04:17,098 [vert.x-worker-thread-72] WARN io.antmedia.muxer.HLSMuxer - cannot write video frame to muxer(Palm1.m3u8) av_bsf_receive_packet. Error is Invalid argument

lastpeony commented 6 months ago

The problem arises due to DTS overflow in SRT packets. After 24+ hours, this overflow manifests, causing incoming packet DTS values to become lower than the last received DTS. Consequently, SRT packets fail to be written to the HLS muxer. For instance, a quick fix trial involving adding +1 to the problematic packet DTS to ensure it exceeds the last received DTS doesn't trigger an error in the HLS muxer. However, the resulting HLS data is corrupted, as anticipated.

I'm currently delving into potential solutions to address this overflow issue, drawing from strategies we've employed previously to tackle overflow issues on the WebRTC side.

lastpeony commented 5 months ago

Fix: https://gitlab.com/Ant-Media/Ant-Media-Enterprise/-/merge_requests/694

mekya commented 5 months ago

Thank you for the fix and collaboration @lastpeony @burak-58

It's merged into the master.

FYI