SRT stream closing due to "too many reordered frames"

Build2NU commented 8 months ago

Which version are you using?

v1.5.1

Just tested in version 1.6.0, still happening.

Which operating system are you using?

[x ] Linux amd64 standard
[ ] Linux amd64 Docker
[ ] Linux arm64 standard
[ ] Linux arm64 Docker
[ ] Linux arm7 standard
[ ] Linux arm7 Docker
[ ] Linux arm6 standard
[ ] Linux arm6 Docker
[ ] Windows amd64 standard
[ ] Windows amd64 Docker (WSL backend)
[ ] macOS amd64 standard
[ ] macOS amd64 Docker
[ ] Other (please describe)

Describe the issue

Viewing an SRT stream for an extended period of time, eventually the stream will disconnect with this error:

Mar 01 09:32:17 vultr mediamtx[4887]: 2024/03/01 09:32:17 INF [SRT] [conn X.X.X.X:51922] closed: too many reordered frames (48)

It doesn't seem to be a problem streaming TO the server, just pulling from it. Once it happens once, it will continue to happen over and over in quick succession. In my network dump, it was happening at that time.

Describe how to replicate the issue

I only have SRT and RTSP enabled. No other services are enabled. I'm running the linux binary for amd64 on ubuntu 22.04.4 LTS I'm using a Vultr VPS server. I've had no issues with these servers before. They've always been solid. I also tried compiling from source for my specific linux install, and this exact same error still happened. But my performance was slightly better, so that's nice. For me, I'm streaming with OBS 30.0.2, NVENC encoder, ffmpeg AAC. CBR @ 7mbps, Preset: P5 slow , with 2 B-frames. Start streaming, and then wait 10-80 minutes for the error to appear. Sometimes it takes a long time to happen.

I also tried running 1.6.0 in docker, same issue.

My internet is extremely good, and I've never run into issues streaming SRT on any other software. This issue also happened with another person in a different part of the world on the same server. I've run a SRT stream out of OBS for over a month before without dropping a frame.

This is my publish URL in OBS: srt://X.X.X.X:8890?streamid=publish:stream1_stream&latency=400000&pkt_size=1316

Did you attach the server logs?

Yes, here they are. I tried setting it to debug, but that didn't add anything. I have replaced my IP address with X.X.X.X

2024/03/01 19:11:07 INF [SRT] listener opened on :8890 (UDP) 2024/03/01 19:11:09 INF [SRT] [conn X.X.X.X:49719] opened 2024/03/01 19:11:09 DEB [path stream1_stream] created 2024/03/01 19:11:09 INF [SRT] [conn X.X.X.X:49719] closed: no one is publishing to path 'stream1_stream' 2024/03/01 19:11:09 DEB [path stream1_stream] destroyed: not in use 2024/03/01 19:11:09 INF [SRT] [conn X.X.X.X:49720] opened 2024/03/01 19:11:09 DEB [path stream1_stream] created 2024/03/01 19:11:10 INF [SRT] [conn X.X.X.X:53161] opened 2024/03/01 19:11:10 INF [SRT] [conn X.X.X.X:53161] closed: no one is publishing to path 'stream1_stream' 2024/03/01 19:11:10 INF [SRT] [conn X.X.X.X:49720] is publishing to path 'stream1_stream', 2 tracks (H264, MPEG-4 Audio) 2024/03/01 19:11:11 INF [SRT] [conn X.X.X.X:53162] opened 2024/03/01 19:11:11 INF [SRT] [conn X.X.X.X:53162] is reading from path 'stream1_stream', 2 tracks (H264, MPEG-4 Audio) 2024/03/01 19:31:33 INF [SRT] [conn X.X.X.X:53162] closed: too many reordered frames (11) 2024/03/01 19:31:35 INF [SRT] [conn X.X.X.X:59348] opened 2024/03/01 19:31:35 INF [SRT] [conn X.X.X.X:59348] is reading from path 'stream1_stream', 2 tracks (H264, MPEG-4 Audio) 2024/03/01 19:31:59 INF [SRT] [conn X.X.X.X:59348] closed: too many reordered frames (46) 2024/03/01 19:32:03 INF [SRT] [conn X.X.X.X:50361] opened 2024/03/01 19:32:03 INF [SRT] [conn X.X.X.X:50361] is reading from path 'stream1_stream', 2 tracks (H264, MPEG-4 Audio)

Which just seems like a server-side lost connection.

Did you attach a network dump?

Yes, I have added it as a comment below

I have a friend running a local network setup in 1.5.0 and not having any issues. I'm testing reverting version and I'll update after I finish testing.

EDIT: the error still happens on his setup when I stream to it, and the error happens when he streams to my clout seup.

I do want to add, the only way for me to not encounter the error, is to stream locally to the windows binary. That works perfectly.

Build2NU commented 8 months ago

I did a 60 second network capture while I got the error multiple times. Not sure if this helps, but I got around to doing it: https://drive.google.com/file/d/1txTU1FdkpGHWbulYnWaZaDt06BCQ2UDU/view?usp=sharing

Build2NU commented 8 months ago

@aler9 Do you have any ideas as to why this is happening? I'm unable to use what I'd say is the best streaming video software solution because of this error. Is there any file I can adjust in source to make it just try and continue instead of closing on me? I'm not even remotely a go developer so I don't know. I'm just a video guy trying to deploy a possible software solution.

lfleuter commented 4 months ago

Any updates on this issue? Because I experience the same problem using v1.8.3. For me, it seems like it usually occurs when the network connection has been poor in the meantime so that the reader and publisher connections were closed and reopened. My experience is also that it repeats every 2-3 minutes after the problem first occurs even if the network connection is stable.

Build2NU commented 4 months ago

Any updates on this issue? Because I experience the same problem using v1.8.3. For me, it seems like it usually occurs when the network connection has been poor in the meantime so that the reader and publisher connections were closed and reopened. My experience is also that it repeats every 2-3 minutes after the problem first occurs even if the network connection is stable.

Same here. For me, it occurs even without poor network quality. Sometimes it takes 20 minutes to appear, sometimes a couple hours, but like you said. once it starts, every 2-3 minutes the SRT reader errors out with the reordered frames error, reconnects, errors again, and again, and again.

Build2NU commented 1 month ago

I have a slight workaround / solution. I have noticed that it only happens with B frames are enabled on the source. If you disable B frames, you never get the too many reordered frames error.

B frames also need to be disabled for WebRTC streaming to work correctly, but for SRT > RTMP or HLS there's no reason it shouldn't work.

@aler9 Does this help? Or should I expect to just stream without B frames for the foreseeable future?

But it doesn't explain why it only fails after a while, and then continues to fail, and why it only seems to fail on the reading side, and not the OBS publishing side.

In order to encode B frames, a video encoder must reorder frames, which means that the order in which the frames are emitted and stored (the decode order) is different from the order in which they were presented to the video encoder (the display order).

bluenviron / mediamtx