search

intel / cartwheel-ffmpeg

Intel developer staging area for unmerged upstream patch contributions to FFmpeg
GNU Lesser General Public License v2.1
99 stars 33 forks source link

The encoder `vp9_qsv` fails with `Invalid FrameType:0` on some inputs. #360

Open Brainiarc7 opened 1 week ago

Brainiarc7 commented 1 week ago

I'm running FFmpeg Cartwheel git tip with a simple invocation for 1:N ladder calling up vp9_qsv as thus:

ffmpeg -v verbose \
-init_hw_device vaapi=intel:/dev/dri/renderD128 \
-hwaccel vaapi -hwaccel_output_format vaapi \
-hwaccel_device intel -filter_hw_device intel \
-loglevel info \
-fflags +genpts \
-i "meridian_hdr.mp4" \
-filter_complex \
"[0:v]hwmap=derive_device=qsv,format=qsv,vpp_qsv=deinterlace=2:async_depth=1,split=5[s1][s2][s3][s4][s5]; \
[s1]vpp_qsv=w=1920:h=1080:async_depth=1:scale_mode=1[v1]; \
[s2]vpp_qsv=w=1280:h=720:async_depth=1:scale_mode=1[v2]; \
[s3]vpp_qsv=w=960:h=540:async_depth=1:scale_mode=1[v3]; \
[s4]vpp_qsv=w=768:h=432:async_depth=1:scale_mode=1[v4]; \
[s5]vpp_qsv=w=640:h=360:async_depth=1:scale_mode=1[v5]" \
-map "[v1]" -c:v vp9_qsv -b:v 5000k -minrate:v 5000k -maxrate:v 5000k -bufsize:v 1250k -rc_init_occupancy:v 625k \
-g:v 120 -async_depth:v 1 -refs:v 5 -bf:v 0 -preset:v medium -strict:v 1 -low_power:v 1 \
-flags +global_header+cgop -f mp4 -y "1080p-vp9_qsv.mp4" \
-map "[v2]" -c:v vp9_qsv -b:v 4000k -minrate:v 4000k -maxrate:v 4000k -bufsize:v 1000k -rc_init_occupancy:v 500k \
-g:v 120 -async_depth:v 1 -refs:v 5 -bf:v 0 -preset:v medium -strict:v 1 -low_power:v 1 \
-flags +global_header+cgop -f mp4 -y "720p-vp9_qsv.mp4" \
-map "[v3]" -c:v vp9_qsv -b:v 2500k -minrate:v 2500k -maxrate:v 2500k -bufsize:v 625k -rc_init_occupancy:v 312k \
-g:v 120 -async_depth:v 1 -refs:v 5 -bf:v 0 -preset:v medium -strict:v 1 -low_power:v 1 \
-flags +global_header+cgop -f mp4 -y "540p-vp9_qsv.mp4" \
-map "[v4]" -c:v vp9_qsv -b:v 1500k -minrate:v 1500k -maxrate:v 1500k -bufsize:v 375k -rc_init_occupancy:v 187k \
-g:v 120 -async_depth:v 1 -refs:v 5 -bf:v 0 -preset:v medium -strict:v 1 -low_power:v 1 \
-flags +global_header+cgop -f mp4 -y "432p-vp9_qsv.mp4" \
-map "[v5]" -c:v vp9_qsv -b:v 1000k -minrate:v 1000k -maxrate:v 1000k -bufsize:v 250k -rc_init_occupancy:v 125k \
-g:v 120 -async_depth:v 1 -refs:v 5 -bf:v 0 -preset:v medium -strict:v 1 -low_power:v 1 \
-flags +global_header+cgop -f mp4 -y "360p-vp9_qsv.mp4"

I'm using Netflix's Meridian short film for this test.

The command dies with:

[vost#4:0/vp9_qsv @ 0x59d69f27f2c0] [enc:vp9_qsv @ 0x59d69f27ff40] Error submitting video frame to the encoder
[vost#4:0/vp9_qsv @ 0x59d69f27f2c0] [enc:vp9_qsv @ 0x59d69f27ff40] Error encoding a frame: Invalid data found when processing input
[vost#4:0/vp9_qsv @ 0x59d69f27f2c0] Task finished with error code: -1094995529 (Invalid data found when processing input)
[vost#4:0/vp9_qsv @ 0x59d69f27f2c0] Terminating thread with return code -1094995529 (Invalid data found when processing input)
[out#0/mp4 @ 0x59d69f12d1c0] video:34187KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.041609%
[out#1/mp4 @ 0x59d69f147180] video:27266KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.051597%
[out#2/mp4 @ 0x59d69f12d9c0] video:16909KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.082624%
[out#3/mp4 @ 0x59d69f23a8c0] video:10164KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.136379%
[out#4/mp4 @ 0x59d69f27e3c0] video:6691KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.204200%
frame= 3278 fps= 46 q=-0.0 Lq=-0.0 q=-0.0 q=-0.0 q=-0.0 size=   34201KiB time=00:00:54.03 bitrate=5184.8kbits/s speed=0.756x    
Conversion failed!

The same file encodes correctly with hevc_qsv and h264_qsv on this platform with (near) identical settings across multiple runs, both with and without low-power encoding.

Platform information:

vainfo 
Trying display: wayland
Trying display: x11
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 24.3.4 ()
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointFEI
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointFEI
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointFEI
      VAProfileHEVCMain               : VAEntrypointEncSliceLP
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointEncSliceLP
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointEncSliceLP
      VAProfileVP9Profile1            : VAEntrypointVLD
      VAProfileVP9Profile1            : VAEntrypointEncSliceLP
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointEncSliceLP
      VAProfileVP9Profile3            : VAEntrypointVLD
      VAProfileVP9Profile3            : VAEntrypointEncSliceLP
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileHEVCMain12             : VAEntrypointEncSlice
      VAProfileHEVCMain422_10         : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointEncSlice
      VAProfileHEVCMain422_12         : VAEntrypointVLD
      VAProfileHEVCMain422_12         : VAEntrypointEncSlice
      VAProfileHEVCMain444            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         : VAEntrypointVLD
      VAProfileHEVCMain444_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_12         : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain10          : VAEntrypointVLD
      VAProfileHEVCSccMain10          : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444         : VAEntrypointVLD
      VAProfileHEVCSccMain444         : VAEntrypointEncSliceLP
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointEncSliceLP

Kernel info:

uname -r
6.8.0-48-generic

And from mfx-inspect: 

vpl-inspect -b    
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/local/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/local/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0

Implementation #0: mfx-gen
  Library path: /usr/local/lib/libmfx-gen.so.1.2.13
  AccelerationMode: MFX_ACCEL_MODE_VIA_VAAPI
  ApiVersion: 2.13
  Impl: MFX_IMPL_TYPE_HARDWARE
  VendorImplID: 0x0000
  ImplName: mfx-gen
  License: MIT License
  Version: 1.2
  Keywords: 
  VendorID: 0x8086
  mfxAccelerationModeDescription:
    Version: 1.0
    Mode: MFX_ACCEL_MODE_VIA_VAAPI
  mfxPoolPolicyDescription:
    Version: 1.0
    Policy: MFX_ALLOCATION_OPTIMAL
    Policy: MFX_ALLOCATION_UNLIMITED
    Policy: MFX_ALLOCATION_LIMITED
  mfxDeviceDescription:
    MediaAdapterType: MFX_MEDIA_INTEGRATED
    DeviceID: a7a0/0
    Version: 1.1
  mfxSurfaceTypesSupported:
    Version: 1.0
    SurfaceType: MFX_SURFACE_TYPE_VAAPI
      SurfaceComponent: MFX_SURFACE_COMPONENT_ENCODE
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY
      SurfaceComponent: MFX_SURFACE_COMPONENT_DECODE
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY
      SurfaceComponent: MFX_SURFACE_COMPONENT_VPP_INPUT
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY
      SurfaceComponent: MFX_SURFACE_COMPONENT_VPP_OUTPUT
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY

Total number of implementations found = 1
root@187a59386a36:/tmp/workdir# vpl-inspect -b
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/local/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/local/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0

Implementation #0: mfx-gen
  Library path: /usr/local/lib/libmfx-gen.so.1.2.13
  AccelerationMode: MFX_ACCEL_MODE_VIA_VAAPI
  ApiVersion: 2.13
  Impl: MFX_IMPL_TYPE_HARDWARE
  VendorImplID: 0x0000
  ImplName: mfx-gen
  License: MIT License
  Version: 1.2
  Keywords: 
  VendorID: 0x8086
  mfxAccelerationModeDescription:
    Version: 1.0
    Mode: MFX_ACCEL_MODE_VIA_VAAPI
  mfxPoolPolicyDescription:
    Version: 1.0
    Policy: MFX_ALLOCATION_OPTIMAL
    Policy: MFX_ALLOCATION_UNLIMITED
    Policy: MFX_ALLOCATION_LIMITED
  mfxDeviceDescription:
    MediaAdapterType: MFX_MEDIA_INTEGRATED
    DeviceID: a7a0/0
    Version: 1.1
  mfxSurfaceTypesSupported:
    Version: 1.0
    SurfaceType: MFX_SURFACE_TYPE_VAAPI
      SurfaceComponent: MFX_SURFACE_COMPONENT_ENCODE
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY
      SurfaceComponent: MFX_SURFACE_COMPONENT_DECODE
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY
      SurfaceComponent: MFX_SURFACE_COMPONENT_VPP_INPUT
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY
      SurfaceComponent: MFX_SURFACE_COMPONENT_VPP_OUTPUT
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY

Total number of implementations found = 1
root@187a59386a36:/tmp/workdir# 
root@187a59386a36:/tmp/workdir# vpl-inspect -b
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/local/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/local/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0

Implementation #0: mfx-gen
  Library path: /usr/local/lib/libmfx-gen.so.1.2.13
  AccelerationMode: MFX_ACCEL_MODE_VIA_VAAPI
  ApiVersion: 2.13
  Impl: MFX_IMPL_TYPE_HARDWARE
  VendorImplID: 0x0000
  ImplName: mfx-gen
  License: MIT License
  Version: 1.2
  Keywords: 
  VendorID: 0x8086
  mfxAccelerationModeDescription:
    Version: 1.0
    Mode: MFX_ACCEL_MODE_VIA_VAAPI
  mfxPoolPolicyDescription:
    Version: 1.0
    Policy: MFX_ALLOCATION_OPTIMAL
    Policy: MFX_ALLOCATION_UNLIMITED
    Policy: MFX_ALLOCATION_LIMITED
  mfxDeviceDescription:
    MediaAdapterType: MFX_MEDIA_INTEGRATED
    DeviceID: a7a0/0
    Version: 1.1
  mfxSurfaceTypesSupported:
    Version: 1.0
    SurfaceType: MFX_SURFACE_TYPE_VAAPI
      SurfaceComponent: MFX_SURFACE_COMPONENT_ENCODE
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY
      SurfaceComponent: MFX_SURFACE_COMPONENT_DECODE
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY
      SurfaceComponent: MFX_SURFACE_COMPONENT_VPP_INPUT
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY
      SurfaceComponent: MFX_SURFACE_COMPONENT_VPP_OUTPUT
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_IMPORT_COPY
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_SHARED
        SurfaceFlags:     MFX_SURFACE_FLAG_EXPORT_COPY

Total number of implementations found = 1
feiwan1 commented 1 week ago

The issue happens because of MFXVideoCORE_SyncOperation returns MFX_ERR_NOT_ENOUGH_BUFFER error which seems invalid according vpl doc: https://intel.github.io/libvpl/latest/API_ref/VPL_func_vidcore.html#_CPPv426MFXVideoCORE_SyncOperation10mfxSession12mfxSyncPoint6mfxU32

And bug report to vpl-gpu-rt: https://github.com/intel/vpl-gpu-rt/issues/353