bytedeco / javacpp-presets

The missing Java distribution of native C++ libraries
Other
2.67k stars 741 forks source link

Low latency playback for real-time streaming #789

Closed b005t3r closed 5 years ago

b005t3r commented 5 years ago

I noticed that FFmpegFrameRecorder and FFmpegFrameGrabber introduce a significant latency when processing live streams. I tested things with the ffmpeg app and it turns out I can get it to almost no latency when using -fflags nobuffer, but it doesn't have any effect when set used with setOption("fflags", "nobuffer").

Is there anything I can do to decrease the latency? Or am I simply doing something wrong?

saudet commented 5 years ago

Duplicate of https://github.com/bytedeco/javacv/issues/862

saudet commented 5 years ago

If you figure out which option we need to pass, please post of over there! Thanks

b005t3r commented 5 years ago

@saudet

I tried variety of options, on both grabber and recorder, they make NO difference, absolutely nothing.

It looks like a delay of about 3 secs. is always introduced by FFmpegFrameGrabber and/or FFmpegFrameRecorder, but I have no idea where, why and how. I have no such delay, if I just use ffmpeg/ffplay (I can get to unnoticeable delay, probably lower than 0.1 sec.)

Is there anything you do in FFmpegFrameGrabber/FFmpegFrameRecorder which could introduce a delay? Any idea where I should start looking?

saudet commented 5 years ago

Like you said, we need to set that "nobuffer" option! Once that set, it will behave like you want. If it doesn't behave like you want, it simply means that the "nobuffer" option isn't set like it needs to be set. That's it, there is absolutely nothing more to it.

saudet commented 5 years ago

In any case, maybe there something else at play. Look at the debug log in both cases, and try to spot any differences that might explain the issue.

b005t3r commented 5 years ago

If it doesn't behave like you want, it simply means that the "nobuffer" option isn't set like it needs to be set.

Turns out there is something else at play. The "nobuffer" flag doesn't make any difference and was a false lead - I can see you recommended that in almost all issues where delay was mentioned, but it's not the way to go. I tested that with ffmpeg/ffplay alone, no JavaCV, and it does not have any noticeable effect.

There problem is loading dependenciec/classes/etc. when FFmpegFrameGrabber and/or FFmpegFrameRecorderinstances are created for the first time. It takes a few seconds actually to get this going, which is insane. I was able to work around it a bit by dropping frames on my feed until everything loads up, but it's not ideal and I was able to get to maybe 0.6 sec. of delay.

Is there a way to preload all of that? I tried these:

        FFmpegFrameGrabber.tryLoad();
        FFmpegFrameRecorder.tryLoad();

and they help a bit, but not a whole lot.

saudet commented 5 years ago

That's probably caused by the JVM not JIT compiling everything right away and then taking some time when it does decide to compile stuff. You could try to run your application with java -Xint ... and then at least you're not going to get any overhead from the JIT compiler...

b005t3r commented 5 years ago

-Xint is actually a very bad idea, because it switches to interpreter mode and makes things run 10x slower. I tried -Xcomp but no difference, I still get about 0.6-0.8 sec. lag.

It's actually pretty weird, because I can see in the debug output of my app that it started displaying something before I actually see it displayed, which means this lag is on recorder's side, I guess?

saudet commented 5 years ago

You're talking about display latency? You'll probably get less overhead with a proper framework for that like JavaFX.

b005t3r commented 5 years ago

No, I'm not talking about display latency.

This is my setup: ffmpeg (live video feed) -> app (FFmpegFrameGrabber + FFmpegFrameRecorder) -> ffplay or ffmpeg with sdl output (displaying frames)

This works real-time (no frame drops, no slow downs) in this setup, except this 0.6 sec. latency which I don't have, if I stream directly from ffmpeg to ffplay (or another ffmpeg displying to a sdl window).

Makes sense? Also, this is probably not caused by the JIT/loading time anymore (because of my frame drop at feed workaround), but I tried any reasonable combination of flags with this setup and nothing worked.

saudet commented 5 years ago

Ah, I see. If you use a codec like H.264, that has a default GOP size of around 250 frames. You'll probably want to reduce that.

b005t3r commented 5 years ago

I use rawvideo to everything and avi as a stream format. Tried changing GOP size, no difference (but it shouldn't matter for rawvideo anyway).

saudet commented 5 years ago

AVI doesn't support streaming too well. Try something else like MKV or FLV.

b005t3r commented 5 years ago

AVI is probably the simplest, low-overhead format out there. It does work with unnoticeable latency (like 1-2 frames, maybe) when I take JavaCV out of equations and simply do this:

mkfifo /tmp/pipe
ffmpeg ... -vcodec rawvideo -pix_fmt bgr24 -f avi /tmp/pipe
ffmpeg -i /tmp/pipe -f sdl "Preview"

No lag whatsoever with this config. The issue has to be somewhere how grabber and/or recorder are implemented.

saudet commented 5 years ago

Ah, ok, you're not streaming audio, so yes that doesn't matter. Anyway, unless you plan on explaining exactly what you're trying to do, I'll stop trying to guess.

b005t3r commented 5 years ago

It works with audio as well (except I need to switch ffmpeg to ffplay on the receiving end).

I'm working on a real-time processing app (real-time object detection).

I've got a ffmpeg process running acting as a feed:

ffmpeg ... -vcodec rawvideo -pix_fmt bgr24 -f avi /tmp/in_pipie

And a ffplay acting as a preview window on the receiving end:

ffplay /tmp/out_pipie

In the middle, there's my app, based on JavaCV. I have a grabber instance in the app which reads from tmp/in_pipe and a recorder instance which writes to /tmp/out_pipe. All is done in rawvideo, format is set to avi.

Now, without the app, there's a very little latency when I'm sending video from the feed (ffmpeg) to the preview window (ffplay, using a single pipe for that). With the app, I've got 0.6-0.8 sec. of delay.

There's no buffering on my side in the app, I process frames as they are read from the grabber and I send them to the recorder.

That's what I'm doing :)

saudet commented 5 years ago

It still doesn't explain what you're trying to do. If you want the lowest latency possible, you shouldn't be trying to encode images, and just copy everything instead. Why are you trying to encode images?

b005t3r commented 5 years ago

I don't encode anything, I just need to read from a feed, process the video and send it to an output (ffplay in this example). There's no encoding, everything is read as rawvideo (bgr24) and forwarded like that to the output.

saudet commented 5 years ago

Ok, so don't encode your packets! Try something like this: https://github.com/bytedeco/javacv/blob/master/samples/PacketRecorderTest.java

b005t3r commented 5 years ago

Would you mind to elaborate what you mean by "don't encode"? Nothing is encoded, I decode the input to bgr24 and it's just processed like that throughout the whole processing chain, up to the preview.

saudet commented 5 years ago

It sounds like you're trying to do something that it wasn't designed for. Someone will need to figure out how to modify JavaCV so it does what you need it to do, but you should also really try to focus on explaining more clearly exactly what you want it to do.

b005t3r commented 5 years ago

I hope ffmpeg is good enough for real-time processing :)

Please, let me know what exactly is not clear, I'll try to explain things in more details, if needed.

saudet commented 5 years ago

If it actually does what you need, but it only seems slow, use a profiler to figure out where it spends most of the time. Maybe that will help.

b005t3r commented 5 years ago

I think you don't understand. There's no performance issue, everything runs smoothly. The issue is it introduces additional latency when compared to the original input and this latency is much less if JavaCV stuff is taken out of the equation.

saudet commented 5 years ago

Right, but we need to start debugging this somehow. If you have any better ideas, go for it!

b005t3r commented 5 years ago

Actually, I did some test, removing my app from the chain and adding a new instance of ffmpeg in the middle, so instead of:

ffmpeg -> app -> ffplay

I have

ffmpeg -> ffmpeg -> ffplay

And it looks like the latency is the same as with the app :/ So I guess it has to have something to do with sending output through a named pipe.

Anyway, I'm using this crazy combo only because there's no decklink (BlackMagic) support in your ffmpeg implementation, so I needed to use a separate ffmpeg version just to stream the video from. But I understand, it should be fairly easy to build JavaCV, JavaCPP and presets entirely from scratch with this added, is this right? Is there something like a step by step tutorial for this maybe? :)

saudet commented 5 years ago

Didn't you say you were able to build FFmpeg with support for DeckLink? What's missing?

saudet commented 5 years ago

BTW, it looks like Blackmagic supports DirectShow: https://trac.ffmpeg.org/wiki/Capture/Blackmagic That's not enough?