search

bytedeco / javacv

Java interface to OpenCV, FFmpeg, and more
Other
7.53k stars 1.58k forks source link

V1.5.7 FFmpegFrameRecorder cannot encode video frames #1694

Open luunn54 opened 3 years ago

luunn54 commented 3 years ago

FFmpegFrameRecorder could not encode video frames, it occurs on some specific videos. This issue does not occur on V1.5.4.

Code

FFmpegFrameRecorder recorder = initRecorder();
frame = ... // init frame
recorder.record(frame); ->  throw an exception

error log:

Warning: [swscaler @ 0x7fb2e623a000] deprecated pixel format used, make sure you did set range correctly

org.bytedeco.javacv.FFmpegFrameRecorder$Exception: avcodec_send_frame() error -541478725: Error sending a video frame for encoding. (For more details, make sure FFmpegLogCallback.set() has been called.)
    at org.bytedeco.javacv.FFmpegFrameRecorder.recordImage(FFmpegFrameRecorder.java:1050)
    at org.bytedeco.javacv.FFmpegFrameRecorder.record(FFmpegFrameRecorder.java:955)
    at org.bytedeco.javacv.FFmpegFrameRecorder.record(FFmpegFrameRecorder.java:948)
    at abc.Test.main(Test.java:83)

It seems that the error was thrown https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameRecorder.java#L1050

saudet commented 3 years ago

Yes, that's been fixed. Please give it a try with the snapshots: http://bytedeco.org/builds/

saudet commented 3 years ago

Ah, wait a minute, this is with FFmpegFrameRecorder, not FFmpegFrameGrabber? Could you try to fix this in a similar fashion to FFmpegFrameGrabber and open a pull request with the changes? Thanks!

luunn54 commented 3 years ago

Hi @saudet

please help to review the PR: https://github.com/bytedeco/javacv/pull/1695

thank you

saudet commented 2 years ago

Could you try again with JavaCV 1.5.7? If this issue is caused by data frames, @anotherche fixed a couple of things related to that, so it might have done the trick for this issue as well. If you could take the time to check this out, that would be great! Thanks

mtvgn commented 1 year ago

I actually am getting quite some of these "Exception when recording frame: avcodec_send_frame() error -541478725" today. I have been testing with JavaCV version 1.5.8. for quite a while, but until yesterday I was using a USB cable on a USB 2.0 port and I was, mostly, testing on 800x600 I swapped to USB-c (USB 3) yesterday AND went to higher resolutions, HD and Full HD. That has worked a couple of times before it became unstable. Sometimes recording FullHD still works fine, sometimes my log gets flooded with these errors. I suspect the change to the higher resolution has caused this, but I just wondered if this happens on all recent versions, so I switched from 1.5.8 to 1.5.9 to no avail and now that I've gone (back) to 1.5.7 it works. I just wonder if it stays stable, so I've got some testing ahead.

Looking at the changelog quite some changes have been made between 1.5.7 and 1.5.8, so it might be that I'll have to stick with 1.5.7, or I might find out later today that this version also gives me these errors sometimes and then I think I'll skip Full HD for now as that seems to be causing these problems ...

mtvgn commented 1 year ago

Update, the error happens with 1.5.7 as well :-(

mtvgn commented 1 year ago

Things I did to see if I could find a stable environment:

Looking in the log file to see if there's a similarity for when these exceptions occur I can find one thing that might point in the direction of the problem: the app I'm writing should give the user the option to do multiple audio/video recordings in a row, and after a recording I stop the incoming video frames and reopen /start everything for the next recording. Those -541478725 errors never occur at the first recording after starting the app, but happen at recording 2, 3, 4, 5, 6 or 7 so I think I don't close enough resources when finishing a recording.

Right now I have frameGrabber.stop() and frameGrabber.flush() and in another thread recorder.stop(), maybe the order of those calls matters or I'm missing something. Let's see if I can find the cause

mtvgn commented 1 year ago

The order is always the same, first closing the recorder and then the frameGrabber, seems logical to me

anotherche commented 1 year ago

Error -541478725 means AVERROR_EOF (symbols ' ', 'F', 'O', 'E' are encoded as ascii in the negated error code 541478725, which is 20464F45 in HEX, that is literally 'EOF '. By the way, avutil has some helper function for error codes decoding, but I do not remember if javacv maps that function.) This error means that the encoder entered the flushing mode.

Once flushing mode has been entered, additional flush packets are ignored, and sending frames will return AVERROR_EOF

I do not know by now if that's a bug in FFmpegFrameRecorder or some incorrect usage of it in your code. But formally it means that you are trying to do next recoding with the recorder which is already in the flushing mode (after calling recorder.flush()). This can be the true as the errors appear on the 2nd, 3rd ... records (but never on the 1st?). The explanation of the underlying encoding/decoding process is as follows: send/receive encoding and decoding API overview

mtvgn commented 1 year ago

Of course this was caused by me. I have one thread in one class capturing the frames and one thread in another class recording frames. The capturing thread only sets a frame in the other class when recording has started and when that happens a counter in the recording class goes to 1, the recorder only records when that 'frame delivered' counter is higher than the counter of processed frames, so it processes the frame ans sets the 'processed frame counter' to 1 and then waits for frame 2 being delivered. At a second (or third / fourth) recording it could happen that the previous recording had stopped with those two counters being unequal AND the current frame (still in memory) was sent to the recorder before the capturing thread had posted a frame to it as both the counters didn't get reset. Now they both go back to 0 when a new recording starts and everything works fine recording after recording.

zhang22113 commented 4 weeks ago

Of course this was caused by me. I have one thread in one class capturing the frames and one thread in another class recording frames. The capturing thread only sets a frame in the other class when recording has started and when that happens a counter in the recording class goes to 1, the recorder only records when that 'frame delivered' counter is higher than the counter of processed frames, so it processes the frame ans sets the 'processed frame counter' to 1 and then waits for frame 2 being delivered. At a second (or third / fourth) recording it could happen that the previous recording had stopped with those two counters being unequal AND the current frame (still in memory) was sent to the recorder before the capturing thread had posted a frame to it as both the counters didn't get reset. Now they both go back to 0 when a new recording starts and everything works fine recording after recording.

Hello, I am a beginner in JavaCV and have seen your discussion. I would like to inquire if ffmpegRecorder can encode frames as H264. And the encoding result is the same as the grabber.grabAVPacket result?https://github.com/bytedeco/javacv/issues/2278

saudet commented 4 weeks ago

Sure, we can encode in H.264 like you did in your code. FFmpegFrameGrabber.grabAVPacket() returns data in FFmpeg data structure, it's not a stream, but it will contain H.264 data if you give it such a stream, yes.

zhang22113 commented 4 weeks ago

当然，我们可以像您在代码中所做的那样使用 H.264 进行编码。FFmpegFrameGrabber.grabAVPacket() 在 FFmpeg 数据结构中返回数据，它不是流，但如果您给它这样的流，它将包含 H.264 数据，是的。

But I don't know why the data packets obtained by grabber.grabAVPacket can be played normally when converted into H264 stream, but when I encode them into H264 stream using frame, they cannot be used