search

intel / libvpl

Intel® Video Processing Library (Intel® VPL) API, dispatcher, and examples
https://intel.github.io/libvpl/
MIT License
262 stars 80 forks source link

ICQ for AV1 not working #82

Closed camthoms closed 1 year ago

camthoms commented 1 year ago

Reference: #79

@oviano helped me find this issue, and can explain it better than I can...

Getting this error while using ICQ:

Error allocating the output packet rate=xxxxkbits/s speed = x.xx Error submitting video frame to the encoder

It looks like neither param.mfx.BufferSizeInKB or param.mfx.BRCParamMultiplier in qsv_retrieve_enc_av1_params get set when the codec is AV1, and are thus still zero

jeffreymcallister commented 1 year ago

Thanks for letting us know about this issue. Can you tell us a bit more about how to replicate this error? For example, is this issue seen from FFmpeg on Windows? What hardware are you running on?

oviano commented 1 year ago

In summary, @camthoms posted the command line:

ffmpeg -init_hw_device qsv=intel,child_device=2 -i "input.mkv" -vcodec av1_qsv -bf 7 -g: 120 -look_ahead_depth 100 -global_quality 12 -preset veryslow -acodec opus -strict -2 "output.mkv" -n

The verbose output was:

ffmpeg version 5.1.git Copyright (c) 2000-2022 the FFmpeg developers
built with msvc
configuration: --disable-ffplay --disable-ffprobe --disable-postproc --enable-libdav1d --enable-cuda-nvcc --enable-libvpl --enable-libsvtav1 --enable-libaom --disable-bzlib --disable-iconv --disable-zlib --disable-lzma --disable-sdl2 --prefix=..♀fmpeg-win32-server
libavutil 57. 43.100 / 57. 43.100
libavcodec 59. 54.100 / 59. 54.100
libavformat 59. 34.102 / 59. 34.102
libavdevice 59. 8.101 / 59. 8.101
libavfilter 8. 51.100 / 8. 51.100
libswscale 6. 8.112 / 6. 8.112
libswresample 4. 9.100 / 4. 9.100
Routing option strict to both codec and muxer layer
[AVHWDeviceContext @ 000001C7849B5EC0] Defaulting child_device_type to AV_HWDEVICE_TYPE_D3D11VA for oneVPL.Please explicitly set child device type via "-init_hw_device" option if needed.
[AVHWDeviceContext @ 000001C7849B3780] Using device 8086:56a5 (Intel(R) Arc(TM) A380 Graphics).
[AVHWDeviceContext @ 000001C7849B5EC0] Use Intel(R) oneVPL to create MFX session, API version is 2.7, the required implementation version is 1.3
[AVHWDeviceContext @ 000001C7849B5EC0] Initialize MFX session: implementation version is 2.7
[h264 @ 000001C794B6E600] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 000001C794B6E600] Increasing reorder buffer to 1
Input #0, matroska,webm, from 'Atlanta S01E05 Nobody Beats the Biebs 1080p WEB-DL DD5.1 H.264-Oosh.mkv':
Metadata:
encoder : libebml v1.3.4 + libmatroska v1.4.5
creation_time : 2016-09-28T09:27:31.000000Z
Duration: 00:21:49.51, start: 0.032000, bitrate: 5583 kb/s
Stream #0:0(eng): Subtitle: subrip
Metadata:
BPS : 122
BPS-eng : 122
DURATION : 00:21:33.891000000
DURATION-eng : 00:21:33.891000000
NUMBER_OF_FRAMES: 518
NUMBER_OF_FRAMES-eng: 518
NUMBER_OF_BYTES : 19828
NUMBER_OF_BYTES-eng: 19828
_STATISTICS_WRITING_APP: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_APP-eng: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_DATE_UTC: 2016-09-28 09:27:31
_STATISTICS_WRITING_DATE_UTC-eng: 2016-09-28 09:27:31
_STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
_STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Stream #0:1: Video: h264 (High), 1 reference frame, yuv420p(tv, bt709, progressive, topleft), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn (default)
Metadata:
BPS : 5198282
BPS-eng : 5198282
DURATION : 00:21:49.392000000
DURATION-eng : 00:21:49.392000000
NUMBER_OF_FRAMES: 31394
NUMBER_OF_FRAMES-eng: 31394
NUMBER_OF_BYTES : 850823717
NUMBER_OF_BYTES-eng: 850823717
_STATISTICS_WRITING_APP: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_APP-eng: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_DATE_UTC: 2016-09-28 09:27:31
_STATISTICS_WRITING_DATE_UTC-eng: 2016-09-28 09:27:31
_STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
_STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Stream #0:2(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 384 kb/s (default)
Metadata:
BPS : 384000
BPS-eng : 384000
DURATION : 00:21:49.344000000
DURATION-eng : 00:21:49.344000000
NUMBER_OF_FRAMES: 40917
NUMBER_OF_FRAMES-eng: 40917
NUMBER_OF_BYTES : 62848512
NUMBER_OF_BYTES-eng: 62848512
_STATISTICS_WRITING_APP: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_APP-eng: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_DATE_UTC: 2016-09-28 09:27:31
_STATISTICS_WRITING_DATE_UTC-eng: 2016-09-28 09:27:31
_STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
_STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Stream mapping:
Stream #0:1 -> #0:0 (h264 (native) -> av1 (av1_qsv))
Stream #0:2 -> #0:1 (ac3 (native) -> opus (native))
Stream #0:0 -> #0:2 (subrip (srt) -> ass (ssa))
Press [q] to stop, [?] for help
[h264 @ 000001C794246FC0] Reinit context to 1920x1088, pix_fmt: yuv420p
[graph_1_in_0_2 @ 000001C79444A140] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:5.1(side)
[format_out_0_1 @ 000001C79444D240] auto-inserting filter 'auto_aresample_0' between the filter 'Parsed_anull_0' and the filter 'format_out_0_1'
[auto_aresample_0 @ 000001C79444D340] ch:6 chl:5.1(side) fmt:fltp r:48000Hz -> ch:2 chl:stereo fmt:fltp r:48000Hz
[graph 0 input from stream 0:1 @ 000001C79444CB40] w:1920 h:1080 pixfmt:yuv420p tb:1/1000 fr:2997/125 sar:1/1
[auto_scale_0 @ 000001C79444C040] w:iw h:ih flags:'' interl:0
[format @ 000001C79444AA40] auto-inserting filter 'auto_scale_0' between the filter 'Parsed_null_0' and the filter 'format'
[auto_scale_0 @ 000001C79444C040] w:1920 h:1080 fmt:yuv420p sar:1/1 -> w:1920 h:1080 fmt:nv12 sar:1/1 flags:0x0
Last message repeated 3 times
[av1_qsv @ 000001C7948E3CC0] Using device intel (type qsv) with av1_qsv encoder.
[av1_qsv @ 000001C7948E3CC0] Encoder: input is system memory surface
[av1_qsv @ 000001C7948E3CC0] Use Intel(R) oneVPL to create MFX session with the specified MFX loader
[av1_qsv @ 000001C7948E3CC0] Using the intelligent constant quality (ICQ) ratecontrol method
[av1_qsv @ 000001C7948E3CC0] profile: av1 main; level: 40
[av1_qsv @ 000001C7948E3CC0] GopPicSize: 120; GopRefDist: 8; GopOptFlag: strict; IdrInterval: 0
[av1_qsv @ 000001C7948E3CC0] TargetUsage: 1; RateControlMethod: ICQ
[av1_qsv @ 000001C7948E3CC0] ICQQuality: 12
[av1_qsv @ 000001C7948E3CC0] NumRefFrame: 4
[av1_qsv @ 000001C7948E3CC0] IntRefType: 0; IntRefCycleSize: 0; IntRefQPDelta: 0; IntRefCycleDist: 0
[av1_qsv @ 000001C7948E3CC0] MaxFrameSize: 0;
[av1_qsv @ 000001C7948E3CC0] BitrateLimit: unknown; MBBRC: OFF; ExtBRC: unknown
[av1_qsv @ 000001C7948E3CC0] VDENC: ON
[av1_qsv @ 000001C7948E3CC0] BRefType: pyramid
[av1_qsv @ 000001C7948E3CC0] PRefType: default
[av1_qsv @ 000001C7948E3CC0] MinQPI: 0; MaxQPI: 0; MinQPP: 0; MaxQPP: 0; MinQPB: 0; MaxQPB: 0
[av1_qsv @ 000001C7948E3CC0] FrameRateExtD: 125; FrameRateExtN: 2997
[av1_qsv @ 000001C7948E3CC0] NumTileRows: 1; NumTileColumns: 1; NumTileGroups: 1
[av1_qsv @ 000001C7948E3CC0] WriteIVFHeaders: OFF
Output #0, matroska, to 'AtlantaFFmpegTest2.mkv':
Metadata:
encoder : Lavf59.34.102
Stream #0:0: Video: av1, 1 reference frame (AV01 / 0x31305641), nv12(tv, bt709, progressive, topleft), 1920x1080 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 1000 kb/s, 23.98 fps, 1k tbn (default)
Metadata:
BPS : 5198282
BPS-eng : 5198282
DURATION : 00:21:49.392000000
DURATION-eng : 00:21:49.392000000
NUMBER_OF_FRAMES: 31394
NUMBER_OF_FRAMES-eng: 31394
NUMBER_OF_BYTES : 850823717
NUMBER_OF_BYTES-eng: 850823717
_STATISTICS_WRITING_APP: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_APP-eng: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_DATE_UTC: 2016-09-28 09:27:31
_STATISTICS_WRITING_DATE_UTC-eng: 2016-09-28 09:27:31
_STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
_STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
encoder : Lavc59.54.100 av1_qsv
Stream #0:1(eng): Audio: opus ([255][255][255][255] / 0xFFFFFFFF), 48000 Hz, stereo, fltp, delay 120, 96 kb/s (default)
Metadata:
BPS : 384000
BPS-eng : 384000
DURATION : 00:21:49.344000000
DURATION-eng : 00:21:49.344000000
NUMBER_OF_FRAMES: 40917
NUMBER_OF_FRAMES-eng: 40917
NUMBER_OF_BYTES : 62848512
NUMBER_OF_BYTES-eng: 62848512
_STATISTICS_WRITING_APP: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_APP-eng: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_DATE_UTC: 2016-09-28 09:27:31
_STATISTICS_WRITING_DATE_UTC-eng: 2016-09-28 09:27:31
_STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
_STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
encoder : Lavc59.54.100 opus
Stream #0:2(eng): Subtitle: ass
Metadata:
BPS : 122
BPS-eng : 122
DURATION : 00:21:33.891000000
DURATION-eng : 00:21:33.891000000
NUMBER_OF_FRAMES: 518
NUMBER_OF_FRAMES-eng: 518
NUMBER_OF_BYTES : 19828
NUMBER_OF_BYTES-eng: 19828
_STATISTICS_WRITING_APP: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_APP-eng: mkvmerge v9.4.2 ('So High') 64bit
_STATISTICS_WRITING_DATE_UTC: 2016-09-28 09:27:31
_STATISTICS_WRITING_DATE_UTC-eng: 2016-09-28 09:27:31
_STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
_STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
encoder : Lavc59.54.100 ssa
[av1_qsv @ 000001C7948E3CC0] Error allocating the output packetrate=4185.4kbits/s speed=1.95x
Error submitting video frame to the encoder
All streams finished for output file #0
Terminating muxer thread 0
[opus @ 000001C794276880] 141 frames left in the queue on closing
[opus @ 000001C794276880] Average Intensity Stereo band: 16.0
[opus @ 000001C794276880] Dual Stereo used: 0.01%
[AVIOContext @ 000001C784A51700] Statistics: 282802994 bytes written, 0 seeks, 1079 writeouts
Terminating demuxer thread 0
[AVIOContext @ 000001C784A52280] Statistics: 382263472 bytes read, 2 seeks
Conversion failed!

From the reference to "built with msvc" you can see that it is on the Windows platform.

The error can be seen in the output above:

[av1_qsv @ 000001C7948E3CC0] Error allocating the output packetrate=4185.4kbits/s speed=1.95x
Error submitting video frame to the encoder

Since I had qsvenc.c open at the time, I searched for the error "Error allocating the output packet" and found it occurs here: 

 ret = av_new_packet(&pkt.pkt, q->packet_size);
    if (ret < 0) {
        av_log(avctx, AV_LOG_ERROR, "Error allocating the output packet\n");
        return ret;
    }

I made the assumption (caveat - without actually debugging it!) that q->packet_size was likely zero following its calculation here:

 q->packet_size = q->param.mfx.BufferSizeInKB * q->param.mfx.BRCParamMultiplier * 1000;

I then looked at this piece of code:

switch (q->param.mfx.RateControlMethod) {
    case MFX_RATECONTROL_CBR:
    case MFX_RATECONTROL_VBR:
        if (q->extbrc) {
            q->extco2.LookAheadDepth = q->look_ahead_depth;
        }
#if QSV_HAVE_VCM
    case MFX_RATECONTROL_VCM:
#endif
    case MFX_RATECONTROL_QVBR:
        q->param.mfx.BufferSizeInKB   = buffer_size_in_kilobytes / brc_param_multiplier;
        q->param.mfx.InitialDelayInKB = initial_delay_in_kilobytes / brc_param_multiplier;
        q->param.mfx.TargetKbps       = target_bitrate_kbps / brc_param_multiplier;
        q->param.mfx.MaxKbps          = max_bitrate_kbps / brc_param_multiplier;
        q->param.mfx.BRCParamMultiplier = brc_param_multiplier;
        if (q->param.mfx.RateControlMethod == MFX_RATECONTROL_QVBR)
            q->extco3.QVBRQuality = av_clip(avctx->global_quality, 0, 51);
        break;
    case MFX_RATECONTROL_CQP:
        quant = avctx->global_quality / FF_QP2LAMBDA;
        if (avctx->codec_id == AV_CODEC_ID_AV1) {
            q->param.mfx.QPI = av_clip_uintp2(quant * fabs(avctx->i_quant_factor) + avctx->i_quant_offset, 8);
            q->param.mfx.QPP = av_clip_uintp2(quant, 8);
            q->param.mfx.QPB = av_clip_uintp2(quant * fabs(avctx->b_quant_factor) + avctx->b_quant_offset, 8);
        } else {
            q->param.mfx.QPI = av_clip(quant * fabs(avctx->i_quant_factor) + avctx->i_quant_offset, 0, 51);
            q->param.mfx.QPP = av_clip(quant, 0, 51);
            q->param.mfx.QPB = av_clip(quant * fabs(avctx->b_quant_factor) + avctx->b_quant_offset, 0, 51);
        }
        q->old_global_quality = avctx->global_quality;
        q->old_i_quant_factor = avctx->i_quant_factor;
        q->old_i_quant_offset = avctx->i_quant_offset;
        q->old_b_quant_factor = avctx->b_quant_factor;
        q->old_b_quant_offset = avctx->b_quant_offset;

        break;
#if QSV_HAVE_AVBR
    case MFX_RATECONTROL_AVBR:
        q->param.mfx.TargetKbps  = target_bitrate_kbps / brc_param_multiplier;
        q->param.mfx.Convergence = q->avbr_convergence;
        q->param.mfx.Accuracy    = q->avbr_accuracy;
        q->param.mfx.BRCParamMultiplier = brc_param_multiplier;
        break;
#endif
    case MFX_RATECONTROL_LA:
        q->param.mfx.TargetKbps  = target_bitrate_kbps / brc_param_multiplier;
        q->extco2.LookAheadDepth = q->look_ahead_depth;
        q->param.mfx.BRCParamMultiplier = brc_param_multiplier;
        break;
    case MFX_RATECONTROL_LA_ICQ:
        q->extco2.LookAheadDepth = q->look_ahead_depth;
    case MFX_RATECONTROL_ICQ:
        q->param.mfx.ICQQuality  = av_clip(avctx->global_quality, 1, 51);
        break;
    }

...and noticed that param.mfx.BufferSizeInKB and param.mfx.BRCParamMultiplier were not being set for the ICQ rate control options.

Assuming they were zero, this would lead to the packet size calculation being zero and the error being produced.

A possible patch would be to insert some lines to set these fields as follows:

    case MFX_RATECONTROL_LA_ICQ:
        q->param.mfx.BufferSizeInKB  = buffer_size_in_kilobytes / brc_param_multiplier;
        q->param.mfx.BRCParamMultiplier = brc_param_multiplier;
        q->extco2.LookAheadDepth = q->look_ahead_depth;
    case MFX_RATECONTROL_ICQ:
        q->param.mfx.ICQQuality  = av_clip(avctx->global_quality, 1, 51);
        break;

Note that I've made a lot of assumptions, so you would need to debug this properly and I do not even know if ICQ is meant to be supported for AV1 so it might all be irrelevant.

channeladam commented 1 year ago

I can't get ICQ working with hevc_qsv either - but it does work with h264_qsv.

I'm running the latest of everything... on an ARC A770 - onevpl 2023.1.1, onevpl-intel-gpu 22.6.5, linux-drm-tip-git, intel-media-driver 22.6.4, ffmpeg with cartwheel patches applied, etc

$ ffmpeg -y -loglevel trace -init_hw_device qsv -i "input.mkv" -c:v hevc_qsv -global_quality:v 24 "output.mkv"

[hevc_qsv @ 0x55f3edf29ec0] Initialized an internal MFX session using hardware accelerated implementation
[hevc_qsv @ 0x55f3edf29ec0] Using the intelligent constant quality (ICQ) ratecontrol method
[hevc_qsv @ 0x55f3edf29ec0] Selected ratecontrol mode is unsupported
[hevc_qsv @ 0x55f3edf29ec0] Low power mode is unsupported
[hevc_qsv @ 0x55f3edf29ec0] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

If I change the command above to h264_qsv it works.

jeffreymcallister commented 1 year ago

Investigation complete from oneVPL perspective. Please follow links above to check progress in media-driver and FFmpeg cartwheel projects.

eero-t commented 1 year ago

Investigation complete from oneVPL perspective. Please follow links above to check progress in media-driver and FFmpeg cartwheel projects.

@jeffreymcallister Latter bug was closed after this and the driver bug were filed, and driver bug does not have any comments from the Intel side. No info on what's the status of the issue, is it going to be fixed, or when.