search

bytedeco / javacv

Java interface to OpenCV, FFmpeg, and more
Other
7.57k stars 1.58k forks source link

FFmpegFrameRecorder fails to start for large images (width 6000, height 3300 pixels) #2106

Closed VidVartak closed 1 year ago

VidVartak commented 1 year ago

Using org.bytedeco:javacv-platform:1.5.8, I am trying to generate a video by adding png images as frames. For this I create a recorder like this:

        recorder = new FFmpegFrameRecorder( videoFile.getAbsolutePath(), 6000, 3300);
        recorder.setVideoCodec(AV_CODEC_ID_H264);
        recorder.setFormat("mp4");
        recorder.setFrameRate(videoFPS);
        recorder.setPixelFormat(AV_PIX_FMT_YUV420P);

Later I try adding frames like this:

                OpenCVFrameConverter.ToIplImage iplConverter = new OpenCVFrameConverter.ToIplImage();
                Java2DFrameConverter java2dConverter = new Java2DFrameConverter();
                IplImage img = iplConverter.convert(java2dConverter.convert(inputImage));
                Frame frame = grabberConverter.convert(img);
                try {
                    recorder.record(frame, AV_PIX_FMT_RGB32_1);
                } catch (Exception e) {
                    LOGGER.error("Error when recording frame : ", e);
                }

It fails with this error:

org.bytedeco.javacv.FFmpegFrameRecorder$Exception: No video output stream (Is imageWidth > 0 && imageHeight > 0 and has start() been called?) (For more details, make sure FFmpegLogCallback.set() has been called.)
    at org.bytedeco.javacv.FFmpegFrameRecorder.recordImage(FFmpegFrameRecorder.java:984)
    at org.bytedeco.javacv.FFmpegFrameRecorder.record(FFmpegFrameRecorder.java:971)

Digging a bit deeper it appears that avcodec_open2 returns the negative value of -1313558101 from this call:

            if ((ret = avcodec_open2(video_c, video_codec, options)) < 0) {
                releaseUnsafe();
                av_dict_free(options);
                throw new Exception("avcodec_open2() error " + ret + ": Could not open video codec.");
            }
org.bytedeco.javacv.FFmpegFrameRecorder$Exception: avcodec_open2() error -1313558101: Could not open video codec. (For more details, make sure FFmpegLogCallback.set() has been called.)
    at org.bytedeco.javacv.FFmpegFrameRecorder.startUnsafe(FFmpegFrameRecorder.java:777)
    at org.bytedeco.javacv.FFmpegFrameRecorder.start(FFmpegFrameRecorder.java:423)

Is it simply a matter of using a different video codec (instead of AV_CODEC_ID_H264)? Which one should I use instead?

The above code works well enough if I reduce the size of the image by half, i.e. 3000 x 1650, but the resulting video is grainy.

Thanks!

saudet commented 1 year ago

Make sure you're using libx264.

VidVartak commented 1 year ago

Thanks @saudet for your quick reply. I hope you mean I replace AV_CODEC_ID_H264 by what you suggest, but I don't see any value for libx264 in the enum in avcodec presets. Can you specify which one you mean?

saudet commented 1 year ago

We can set it with setVideoCodecName("libx264")

VidVartak commented 1 year ago

Thanks, it generated the video (which is progress), but when I try to open it (via Windows Media Player) it says it cannot open it because it's encoded in an unsupported format. Is there a fix for that?

saudet commented 1 year ago

You're probably going to need to set a different profile, see issue https://github.com/bytedeco/javacv/issues/1899

saudet commented 1 year ago

Duplicate of #1899

saudet commented 1 year ago

Actually, there is no need to add anything to JavaCV to get this working. I've confirmed that, for example, calling setVideoOption("profile", "high") before start() does set the profile to "high", see https://trac.ffmpeg.org/wiki/Encode/H.264#Profile.

VidVartak commented 1 year ago

Hi @saudet, when I tried that, I got a new error (-22). It gives this error regardless of what I set the profile to, so looks like it's not recognising it as a valid parameter.

This is my code:

        recorder = new FFmpegFrameRecorder( videoFile.getAbsolutePath(), 6000, 3300);
        recorder.setVideoCodecName("libx264");
        recorder.setFormat("mp4");
        recorder.setFrameRate(25);
        recorder.setPixelFormat(AV_PIX_FMT_YUV420P);
        recorder.setVideoOption("profile",  "high");
        try {
            recorder.start();
        } catch (Exception e) {
            LOGGER.error("Error when initialising video", e);
        }

and this is the error I get:

org.bytedeco.javacv.FFmpegFrameRecorder$Exception: avcodec_open2() error -22: Could not open video codec. (For more details, make sure FFmpegLogCallback.set() has been called.)
    at org.bytedeco.javacv.FFmpegFrameRecorder.startUnsafe(FFmpegFrameRecorder.java:777)
    at org.bytedeco.javacv.FFmpegFrameRecorder.start(FFmpegFrameRecorder.java:423)

Thanks for your help!

saudet commented 1 year ago

Please make sure that FFmpegLogCallback.set() has been called.

VidVartak commented 1 year ago

OK (should have checked myself before). Setting that gives me this error:

2023-10-01 12:12:28,849 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 - Error: [mpeg4 @ 000002339fc37480] [Eval @ 00000040de6fc0d0] Undefined constant or missing '(' in 'high'
2023-10-01 12:12:28,850 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 - 
2023-10-01 12:12:28,850 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 - Error: [mpeg4 @ 000002339fc37480] Unable to parse option value "high"
2023-10-01 12:12:28,850 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 - 
2023-10-01 12:12:28,850 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 - Error: [mpeg4 @ 000002339fc37480] Error setting option profile to value high.
2023-10-01 12:12:28,850 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 -

Tried different variations (High, HIGH) all have the same problem.

saudet commented 1 year ago

You're not using the libx264 codec. Like I said, make sure to use libx264.

VidVartak commented 1 year ago

This is my code:

        recorder.setVideoCodecName("libx264");
        recorder.setFormat("mp4");
        recorder.setFrameRate(25);
        recorder.setPixelFormat(AV_PIX_FMT_YUV420P);
        recorder.setVideoOption("profile",  "high");
        FFmpegLogCallback.set();
        LOGGER.debug("Created recorder with codec: " + recorder.getVideoCodecName() + ", width: " + recorder.getImageWidth() + ", height: " + recorder.getImageHeight() +", options:" + recorder.getVideoOptions());

I see this in the log: 

2023-10-01 16:54:16,355 [JavaFX Application Thread] DEBUG c.n.u.e.s.ScheduledEventServiceVideo:90 - Created recorder with codec: libx264, width: 6000, height: 3300, options:{profile=high}
2023-10-01 16:54:16,452 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 - Error: [mpeg4 @ 000001a92d2279c0] [Eval @ 0000009913bfc3f0] Undefined constant or missing '(' in 'high'
2023-10-01 16:54:16,452 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 - 
2023-10-01 16:54:16,452 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 - Error: [mpeg4 @ 000001a92d2279c0] Unable to parse option value "high"
2023-10-01 16:54:16,452 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 - 
2023-10-01 16:54:16,452 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 - Error: [mpeg4 @ 000001a92d2279c0] Error setting option profile to value high.
2023-10-01 16:54:16,452 [JavaFX Application Thread] WARN  o.b.j.t.Logger:64 -

So looks like it is libx264 but still the option is not being recognised?

VidVartak commented 1 year ago

Maybe the problem is with:

recorder.setVideoCodecName("libx264");

because after doing that, although recorder.getVideoCodecName() returns libx264, recorder.getVideoCodec() returns 0, which is defined as NONE:

AV_CODEC_ID_NONE = 0,

There is no constant defined in the enum for libx264 that I can see (in org.bytedeco.ffmpeg.global.avcodec).

saudet commented 1 year ago

Please make sure you have ffmpeg-platform-gpl is in your dependencies.

VidVartak commented 1 year ago

I did that and the video generated and was opened in Media Player, but the resolution/quality is even worse for some reason. A short video of 21 frames, each supposedly 6000x3300 pixels is just 117KB!

Here is the log output at the time of recorder creation:

2023-10-03 10:34:59,883 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
2023-10-03 10:34:59,883 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:34:59,901 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] profile High, level 6.0, 4:2:0, 8-bit
2023-10-03 10:34:59,901 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:34:59,901 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] 264 - core 164 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=24 lookahead_threads=4 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=400 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
2023-10-03 10:34:59,901 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:34:59,935 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: Output #0, mp4, to 'C:\Users\vvartak\IdeaProjects\experiment-ui\temp\video.mp4':
2023-10-03 10:34:59,935 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:34:59,935 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info:   Metadata:
2023-10-03 10:34:59,935 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:34:59,935 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info:     encoder         : 
2023-10-03 10:34:59,935 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: Lavf60.3.100
2023-10-03 10:34:59,935 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: 
2023-10-03 10:34:59,935 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:34:59,935 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info:   Stream #0:0
2023-10-03 10:34:59,935 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: : Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 6000x3300, q=2-31, 400 kb/s
2023-10-03 10:34:59,936 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: , 
2023-10-03 10:34:59,936 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: 25 fps, 
2023-10-03 10:34:59,936 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: 12800 tbn
2023-10-03 10:34:59,936 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 - Info: 
2023-10-03 10:34:59,936 [JavaFX Application Thread] WARN  o.b.j.t.Logger:60 -

and these are the messages after the recorder stops:

2023-10-03 10:40:37,630 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] frame I:1     Avg QP:47.65  size: 38743
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] frame P:5     Avg QP:50.99  size:  7335
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] frame B:15    Avg QP:51.00  size:  2789
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] consecutive B-frames:  4.8%  0.0%  0.0% 95.2%
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] mb I  I16..4: 28.1% 71.3%  0.6%
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] mb P  I16..4:  3.9%  0.4%  0.0%  P16..4:  2.9%  0.1%  0.6%  0.0%  0.0%    skip:92.0%
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] mb B  I16..4:  0.1%  0.0%  0.0%  B16..8:  3.4%  0.1%  0.0%  direct: 0.0%  skip:96.4%  L0:41.8% L1:58.2% BI: 0.0%
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] final ratefactor: 62.59
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] 8x8 transform intra:59.7% inter:86.6%
2023-10-03 10:40:37,631 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] coded y,uvDC,uvAC intra: 5.2% 0.0% 0.0% inter: 0.1% 0.0% 0.0%
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] i16 v,h,dc,p: 47% 52%  1%  1%
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 64% 14% 21%  0%  0%  0%  0%  0%  0%
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 50% 28% 11%  2%  3%  2%  2%  2%  1%
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] i8c dc,h,v,p: 100%  0%  0%  0%
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] Weighted P-Frames: Y:0.0% UV:0.0%
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] ref P L0: 86.6%  0.5%  7.7%  5.2%
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] ref B L0: 84.0% 13.5%  2.5%
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] ref B L1: 92.7%  7.3%
2023-10-03 10:40:37,632 [                 Thread-6] WARN  o.b.j.t.Logger:60 - 
2023-10-03 10:40:37,633 [                 Thread-6] WARN  o.b.j.t.Logger:60 - Info: [libx264 @ 00000233d7017700] kb/s:1116.65
2023-10-03 10:40:37,633 [                 Thread-6] WARN  o.b.j.t.Logger:60 -

Hope there is some useful information there? Thanks!