Reduce Latency

[Ultrasic]

Hello, I'm currently making a project in WPF where I need to play a video stream on the network under 200ms latency. I've been able to do it with FFPlay but I'm stuck with your implementation.

The current situation : I'm receiving an H.264 encoded video encapsulated in RTP and streamed through UDP. I've been able to achieve 150ms latency with FFPlay with this command : .\ffplay.exe C:\AvatarStream1.sdp -protocol_whitelist "file,udp,rtp" -fflags nobuffer -flags low_delay -vf setpts=0

And here is the SDP file :

c=IN IP4 192.168.1.12
m=video 20000 RTP/AVP 96
a=rtpmap:96 H264/90000

Now for the C# code, I'm using the 3.7.20 NuGet version and started from one of your samples with MVVM. Reading through other issues about thic topic I've achieved a latency between 200ms and 250ms, but I've been unable to go under that.

public MainWindow()
{
    InitializeComponent();
    // Initializes Engine (Specifies FFmpeg libraries path which is required
    Engine.Start(new EngineConfig()
    {
        FFmpegPath = "c:\\ffmpeg2",
    });

    ToggleDebug = new RelayCommandSimple(new Action(() => { ShowDebug = !ShowDebug; }));

    Config = new Config();

    Config.Player.MaxLatency = TimeSpan.FromMilliseconds(150).Ticks;

    Config.Audio.Enabled = false;
    Config.Subtitles.Enabled = false;

    Config.Demuxer.FormatOpt["fflags"] = "nobuffer";
    Config.Demuxer.FormatOpt["flags"] = "low_delay";
    Config.Demuxer.FormatOpt.Add("protocol_whitelist", "file,udp,rtp");

    // Inform the lib to refresh stats
    Config.Player.Stats = true;

    Player = new Player(Config);

    DataContext = this;

    // Keep track of error messages
    Player.OpenCompleted += (o, e) => { LastError = e.Error; };
    Player.BufferingCompleted += (o, e) => { LastError = e.Error; };
}

Do you have any idea of what I could do to further optimize the latency?

[anguzo]

@Ultrasic from my personal experience, going under 200ms results in poor stream quality and stability. 200ms is already much better than LibVLCSharp performs.

[Ultrasic]

Hello @anguzo, thanks for the quick response!

Yes, I've previously made an application with LibVLCSharp but wasn't able to go under 400ms.

I also understand the problems coming with going under 200ms but I'm doing an application which requires live control from the user (the camera is attached to something like a robot) and from the research we've done, 200ms is the limit for an application to still be controllable from a user standpoint without being too uncomfortable.

[SuRGeoNix]

@Ultrasic I've been able to play low latency if the stream can support it (under 100ms) with Flyleaf. So probably somewhere else is the issue. I suspect that you use tcp (Flyleaf set this by default instead of udp). set rtsp_transport FormatOpt to udp. If you still have issues attach a trace log to see what is going on.

You also don't set the flags properly. Use FormatFlags (eg. FormatFlags |= 0x40; // For AVFMT_FLAG_NOBUFFER)

check https://github.com/SuRGeoNix/Flyleaf/blob/master/FlyleafLib/Engine/Config.cs

For the Stats you will also need Engine's UIRefresh set to true

[Ultrasic]

I've changed the configuration according to what you told me

Config = new Config();

Config.Player.MaxLatency = 150 * 10000;

Config.Audio.Enabled = false;
Config.Subtitles.Enabled = false;

Config.Demuxer.FormatFlags |= 0x0040;

Config.Demuxer.FormatOpt["flags"] = "low_delay";
Config.Demuxer.FormatOpt["rtsp_transport"] = "udp";
Config.Demuxer.FormatOpt.Add("protocol_whitelist", "file,udp,rtp");

And with a timer I'm measuring a latency of 200ms on average

In the same way, I'm able to average 150ms with FFPlay, but what I find weird is that on the logs, the Player reports a latency of 66-67ms on average so I don't know where those 140ms come from, is it only the rendering?

output.txt

[SuRGeoNix]

OK, first of all this is SDP and not RTSP so it will ignore that rtsp_transport option. A trace level log would help more but indeed flyleaf can't see 200ms back so probably something is going on with ffmpeg. Did you try to set maxlatency to 1? (30 fps btw is 33ms per frame so 2 frames have ~66ms that flyleaf reports). You should count also the hardware (decoding the frame) and network delay. You can increase the frame rate from the source and decrease the quality (to avoid hardware delay?)

[Ultrasic]

I've tried with maxlatency at 1 and didn't notice any improvement (it was actually a bit worse but don't take this into account because I haven't done enough testing)

Here's trace log for it : output1.txt

And something else, I've tried with 1 * 10000 (because i wasn't sure if you wanted one tick or one ms of tick), the latency didn't change but the player reported 0 of latency in the log. output2.txt

It can possibly be a problem with ffmpeg, as I said i tried it with FFPlay and got 150ms but the dlls I tried it with are not the same I use for this application so maybe they did optimize something.

Can it be a problem inherently with C#/WPF being slower? The decoding and all is done with ffmpeg sure but the rendering is C# side so maybe it's slower.

And I could change the framerate of my cameras and/or quality if really it is needed but in the end I would still need to stay with the current configurations.

[SuRGeoNix]

@Ultrasic something happens with ffmpeg. The log (especially the output2) shows that when a demuxer receives a packet it will renderer it almost directly (no much delay there, cant be the issue). Generally, I think for your case 40ms would be ideal (40 * 10000).

[Ultrasic]

Thanks for your insight,, do you have an idea on how I could begin to investigate what could be the problem? As you said, I suppose something is going on with ffmpeg as using a newer version of the dlls gives better result, but at the same time it should be possible to go lower as you where able to.

[SuRGeoNix]

First make sure what ffmpeg requires for your case to play as low latency as possible. Check with FFplay to ensure this. Then read and ask me if required how you pass the options/flags that are required. Ensure that the network latency is not the issue (possible use wireshark to see what is going on). Use Flyleaf trace to see when a packet have been demuxed/decoded/processed/renderered. Use FFmpeg trace as well to see more info if required. Closing this but free to open a new issue/question if you need to.