Live stream has delay between frames and glitches

[don-pironet-hatch]

We are using the library to live stream from a camera. We started from a FFmpeg cli command that works very well:

ffplay -tune zerolatency -flags low_delay -preset ultrafast -fflags nobuffer rtsp://192.168.2.1:8554/video.mp4

We mapped this on the following configuration:

MediaSourceConfig configuration = new MediaSourceConfig()
{
    VideoDecoderMode = VideoDecoderMode.Automatic,
};

configuration.FFmpegOptions.Add("tune", "zerolatency");
configuration.FFmpegOptions.Add("flags", "low_delay");
configuration.FFmpegOptions.Add("preset", "ultrafast");
configuration.FFmpegOptions.Add("fflags", "nobuffer");

_decoder = await FFmpegMediaSource.CreateFromUriAsync(RtspUrl, configuration);
_mediaPlaybackItem = _decoder.CreateMediaPlaybackItem();

_mediaPlayer = new MediaPlayer();
_mediaPlayer.Source = _mediaPlaybackItem;
_mediaPlayer.Play();

mediaPlayerElement.SetMediaPlayer(_mediaPlayer);

This streams the video but with a lot of delay so when you move the camera it takes 1 - 2 seconds until you notice. During the movement a lot of glitches are introduced in the video.

If we run the ffplay command it works like a charm.

What are we doing wrong?

[lukasf]

I never really used the lib for live scenarios. But I remember I also saw glitches when doing streaming tests. Not sure where this comes from. Do you see anything specific in the debug output? You need to register a log provider to see ffmpeg logs, check App.Xaml.cs of your MediaPlayerCS sample. Or just open the stream in our sample app, it already shows debug out.

Two things you can try for latency: First, MediaPlayer has a RealTimePlayback property, second, you can get the MediaStreamSource from the MediaPlaybackItem. It has a IsLive property. Not sure how much this helps, but maybe worth a try.

[don-pironet-hatch]

The RealTimePlayback property doesn't do anything. Log snippet:

[h264 @ 000002447AD67C80] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD68C80] SEI type 5 size 648 truncated at 646
[h264 @ 000002447B0BE480] SEI type 5 size 648 truncated at 646
[h264 @ 000002447B0BDC80] SEI type 5 size 648 truncated at 646
[h264 @ 000002447B0BD880] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD66480] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD66080] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD68880] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD69880] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD68480] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD69480] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD69080] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD67080] SEI type 5 size 648 truncated at 646
[rtsp @ 0000024479A5D940] max delay reached. need to consume packet
[rtsp @ 0000024479A5D940] RTP: missed 4 packets
[h264 @ 000002447B0BDC80] error while decoding MB 32 64, bytestream -7
[h264 @ 000002447B0BDC80] concealing 497 DC, 497 AC, 497 MV errors in P frame
[h264 @ 000002447AD68080] SEI type 5 size 648 truncated at 646
[rtsp @ 0000024479A5D940] max delay reached. need to consume packet
[rtsp @ 0000024479A5D940] RTP: missed 28 packets
[h264 @ 000002447AD69080] error while decoding MB 29 41, bytestream -15
[h264 @ 000002447AD69080] concealing 3260 DC, 3260 AC, 3260 MV errors in P frame
[h264 @ 000002447AD66880] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD66C80] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD67C80] SEI type 5 size 648 truncated at 646
[h264 @ 000002447AD68C80] SEI type 5 size 648 truncated at 646
[h264 @ 000002447B0BE480] SEI type 5 size 648 truncated at 646
[h264 @ 000002447B0BDC80] SEI type 5 size 648 truncated at 646

[lukasf]

Okay this looks bad. I guess this is what is happening: Since you have all those flags (no buffer, zero latency, etc), no or almost no buffering is done inside ffmpeg. But the server keeps sending data at a constant rate. If the MediaPlayer does not query frames fast enough, the buffer will be full and packets are dropped. Then decoding fails of course when next packet is read. At least, that's my theory when looking at the logs.

On the master banch, we have a new read-ahead buffer (Config.ReadAheadBufferEnabled = true). If you build the lib yourself and enable this, we will asynchronously read packets ahead of time, as fast as possible. This should help you get rid of the decoding errors and artifacts. Our samples have a switch to enable ReadAheadBuffer, so you could just open your stream in the sample app, after you toggle the switch.

I am not sure what to do about the latency. Part of the problem could be this: It does take some time to parse the stream and create decoders, then it takes time to create the MediaSource, open it and start playback. When playback starts, we will return the first decoded frame. It has the timestamp from when we first opened the stream, but since then, some time has passed. MediaPlayer will play at a constant rate, so we will probably never pick up to where playback is right now. I guess we could do something about this in the lib, but that would be a difficult task. And I am not sure how much of the latency is in our lib and how much is in the MediaPlayer area. As an experiment, you could try to increase playback speed of the MediaPlayer for a while, to see if you can get it closer to real time position...?

[don-pironet-hatch]

Thanks again for your help.

The Nuget package already has the ReadAheadBufferEnabled flag. I enabled that and increased the playbackRate. For future reference:

            MediaSourceConfig configuration = new MediaSourceConfig()
            {
                VideoDecoderMode = VideoDecoderMode.Automatic,
                ReadAheadBufferEnabled = true,
            };

            configuration.FFmpegOptions.Add("tune", "zerolatency");
            configuration.FFmpegOptions.Add("flags", "low_delay");
            configuration.FFmpegOptions.Add("preset", "ultrafast");
            configuration.FFmpegOptions.Add("fflags", "nobuffer");

            configuration.FFmpegOptions.Add("reconnect", 1);
            configuration.FFmpegOptions.Add("reconnect_streamed", 1);
            configuration.FFmpegOptions.Add("reconnect_on_network_error", 1);

            _decoder = await FFmpegMediaSource.CreateFromUriAsync(Constant.RtspUrl, configuration);
            _mediaPlaybackItem = _decoder.CreateMediaPlaybackItem();

            _mediaPlayer = new MediaPlayer();
            _mediaPlayer.Source = _mediaPlaybackItem;
            _mediaPlayer.PlaybackSession.PlaybackRate = 2;
            _mediaPlayer.Play();

Now it works perfect and it's very fast.

The time to display the first frame could be improved. I'm uncertain whether this issue is associated with the mediaPlayer or a configuration setting.

[brabebhin]

You may want to give frame server mode of MediaPlayer a try, this seems to have somewhat lower latency.