Low latency playback for real-time streaming

[b005t3r]

I noticed that FFmpegFrameRecorder and FFmpegFrameGrabber introduce a significant latency when processing live streams. I tested things with the ffmpeg app and it turns out I can get it to almost no latency when using -fflags nobuffer, but it doesn't have any effect when set used with setOption("fflags", "nobuffer").

Is there anything I can do to decrease the latency? Or am I simply doing something wrong?

[saudet]

Duplicate of https://github.com/bytedeco/javacv/issues/862

[saudet]

If you figure out which option we need to pass, please post of over there! Thanks

[b005t3r]

@saudet

I tried variety of options, on both grabber and recorder, they make NO difference, absolutely nothing.

It looks like a delay of about 3 secs. is always introduced by FFmpegFrameGrabber and/or FFmpegFrameRecorder, but I have no idea where, why and how. I have no such delay, if I just use ffmpeg/ffplay (I can get to unnoticeable delay, probably lower than 0.1 sec.)

Is there anything you do in FFmpegFrameGrabber/FFmpegFrameRecorder which could introduce a delay? Any idea where I should start looking?

[saudet]

Like you said, we need to set that "nobuffer" option! Once that set, it will behave like you want. If it doesn't behave like you want, it simply means that the "nobuffer" option isn't set like it needs to be set. That's it, there is absolutely nothing more to it.

[saudet]

In any case, maybe there something else at play. Look at the debug log in both cases, and try to spot any differences that might explain the issue.

[b005t3r]

If it doesn't behave like you want, it simply means that the "nobuffer" option isn't set like it needs to be set.

Turns out there is something else at play. The "nobuffer" flag doesn't make any difference and was a false lead - I can see you recommended that in almost all issues where delay was mentioned, but it's not the way to go. I tested that with ffmpeg/ffplay alone, no JavaCV, and it does not have any noticeable effect.

There problem is loading dependenciec/classes/etc. when FFmpegFrameGrabber and/or FFmpegFrameRecorderinstances are created for the first time. It takes a few seconds actually to get this going, which is insane. I was able to work around it a bit by dropping frames on my feed until everything loads up, but it's not ideal and I was able to get to maybe 0.6 sec. of delay.

Is there a way to preload all of that? I tried these:

        FFmpegFrameGrabber.tryLoad();
        FFmpegFrameRecorder.tryLoad();

and they help a bit, but not a whole lot.

[saudet]

That's probably caused by the JVM not JIT compiling everything right away and then taking some time when it does decide to compile stuff. You could try to run your application with java -Xint ... and then at least you're not going to get any overhead from the JIT compiler...

[b005t3r]

-Xint is actually a very bad idea, because it switches to interpreter mode and makes things run 10x slower. I tried -Xcomp but no difference, I still get about 0.6-0.8 sec. lag.

It's actually pretty weird, because I can see in the debug output of my app that it started displaying something before I actually see it displayed, which means this lag is on recorder's side, I guess?

[saudet]

You're talking about display latency? You'll probably get less overhead with a proper framework for that like JavaFX.

[b005t3r]

No, I'm not talking about display latency.

This is my setup: ffmpeg (live video feed) -> app (FFmpegFrameGrabber + FFmpegFrameRecorder) -> ffplay or ffmpeg with sdl output (displaying frames)

This works real-time (no frame drops, no slow downs) in this setup, except this 0.6 sec. latency which I don't have, if I stream directly from ffmpeg to ffplay (or another ffmpeg displying to a sdl window).

Makes sense? Also, this is probably not caused by the JIT/loading time anymore (because of my frame drop at feed workaround), but I tried any reasonable combination of flags with this setup and nothing worked.

[saudet]

Ah, I see. If you use a codec like H.264, that has a default GOP size of around 250 frames. You'll probably want to reduce that.

[b005t3r]

I use rawvideo to everything and avi as a stream format. Tried changing GOP size, no difference (but it shouldn't matter for rawvideo anyway).

[saudet]

AVI doesn't support streaming too well. Try something else like MKV or FLV.

[b005t3r]

AVI is probably the simplest, low-overhead format out there. It does work with unnoticeable latency (like 1-2 frames, maybe) when I take JavaCV out of equations and simply do this:

mkfifo /tmp/pipe
ffmpeg ... -vcodec rawvideo -pix_fmt bgr24 -f avi /tmp/pipe
ffmpeg -i /tmp/pipe -f sdl "Preview"

No lag whatsoever with this config. The issue has to be somewhere how grabber and/or recorder are implemented.

[saudet]

Ah, ok, you're not streaming audio, so yes that doesn't matter. Anyway, unless you plan on explaining exactly what you're trying to do, I'll stop trying to guess.

[b005t3r]

It works with audio as well (except I need to switch ffmpeg to ffplay on the receiving end).

I'm working on a real-time processing app (real-time object detection).

I've got a ffmpeg process running acting as a feed:

ffmpeg ... -vcodec rawvideo -pix_fmt bgr24 -f avi /tmp/in_pipie

And a ffplay acting as a preview window on the receiving end:

ffplay /tmp/out_pipie

In the middle, there's my app, based on JavaCV. I have a grabber instance in the app which reads from tmp/in_pipe and a recorder instance which writes to /tmp/out_pipe. All is done in rawvideo, format is set to avi.

Now, without the app, there's a very little latency when I'm sending video from the feed (ffmpeg) to the preview window (ffplay, using a single pipe for that). With the app, I've got 0.6-0.8 sec. of delay.

There's no buffering on my side in the app, I process frames as they are read from the grabber and I send them to the recorder.

That's what I'm doing :)

[saudet]

It still doesn't explain what you're trying to do. If you want the lowest latency possible, you shouldn't be trying to encode images, and just copy everything instead. Why are you trying to encode images?

[b005t3r]

I don't encode anything, I just need to read from a feed, process the video and send it to an output (ffplay in this example). There's no encoding, everything is read as rawvideo (bgr24) and forwarded like that to the output.

[saudet]

Ok, so don't encode your packets! Try something like this: https://github.com/bytedeco/javacv/blob/master/samples/PacketRecorderTest.java

[b005t3r]

Would you mind to elaborate what you mean by "don't encode"? Nothing is encoded, I decode the input to bgr24 and it's just processed like that throughout the whole processing chain, up to the preview.

[saudet]

It sounds like you're trying to do something that it wasn't designed for. Someone will need to figure out how to modify JavaCV so it does what you need it to do, but you should also really try to focus on explaining more clearly exactly what you want it to do.

[b005t3r]

I hope ffmpeg is good enough for real-time processing :)

Please, let me know what exactly is not clear, I'll try to explain things in more details, if needed.

[saudet]

If it actually does what you need, but it only seems slow, use a profiler to figure out where it spends most of the time. Maybe that will help.

[b005t3r]

I think you don't understand. There's no performance issue, everything runs smoothly. The issue is it introduces additional latency when compared to the original input and this latency is much less if JavaCV stuff is taken out of the equation.

[saudet]

Right, but we need to start debugging this somehow. If you have any better ideas, go for it!

[b005t3r]

Actually, I did some test, removing my app from the chain and adding a new instance of ffmpeg in the middle, so instead of:

ffmpeg -> app -> ffplay

I have

ffmpeg -> ffmpeg -> ffplay

And it looks like the latency is the same as with the app :/ So I guess it has to have something to do with sending output through a named pipe.

Anyway, I'm using this crazy combo only because there's no decklink (BlackMagic) support in your ffmpeg implementation, so I needed to use a separate ffmpeg version just to stream the video from. But I understand, it should be fairly easy to build JavaCV, JavaCPP and presets entirely from scratch with this added, is this right? Is there something like a step by step tutorial for this maybe? :)

[saudet]

Didn't you say you were able to build FFmpeg with support for DeckLink? What's missing?

[saudet]

BTW, it looks like Blackmagic supports DirectShow: https://trac.ffmpeg.org/wiki/Capture/Blackmagic That's not enough?