search

opencv / opencv

Open Source Computer Vision Library
https://opencv.org
Apache License 2.0
79.04k stars 55.82k forks source link

Capture MJPEG from V4l2 using VAAPI hardware acceleration for decoding #22768

Open kallaballa opened 2 years ago

kallaballa commented 2 years ago

System Information

OpenCV version: OpenCV 4.x branch (https://github.com/opencv/opencv/commit/2aad039b4f64d8e4a6136388c48d1368c120feec) Operating System / Platform: openSUSE Tumbleweed (Version: 20221102) Compiler & compiler version: g++ (SUSE Linux) 12.2.1 20221020 [revision 0aaef83351473e8f4eb774f8f999bbe87a4866d7]

Detailed description

I'd like to capture an MJPEG stream from a V4L2 device and accelerate decoding using VAAPI. My attempts failed because VAAPI acceleration is not implemented in the VIDEOIO/V4L2-backend and ffmpeg implements V4L2 as de-/muxer which made it impossible for me to activate VAAPI at the same time. Gstreamer is not an option for me because it doesn't support OpenCL/VAAPI interop. Using an MJPEG file as input works.

I use following environment variables:

export OPENCV_LOG_LEVEL=VERBOSE
export OPENCV_FFMPEG_LOGLEVEL=56
export OPENCV_VIDEOIO_DEBUG=1
export OPENCV_VIDEOWRITER_DEBUG=1
export OPENCV_VIDEOCAPTURE_DEBUG=1
export OPENCV_FFMPEG_DEBUG=1
export OPENCV_OPENCL_RAISE_ERROR=1
export OPENCV_OPENCL_ABORT_ON_BUILD_ERROR=1
export OPENCV_DUMP_ERRORS=1
export OPENCV_DUMP_CONFIG=1
export OPENCV_TRACE=1
export OPENCV_TRACE_DEPTH_OPENCV=1
export OPENCV_TRACE_SYNC_OPENCL=1

And run the following command:

src/v4l2-vaapi/v4l2-vaapi /dev/video1

With resulting debug log:

OpenCV build configuration is:

General configuration for OpenCV 4.6.0-dev =====================================
  Version control:               4.6.0-374-gd9c5686d01-dirty

  Extra modules:
    Location (extra):            /home/elchaschab/devel/opencv_contrib-4.x/modules
    Version control (extra):     unknown

  Platform:
    Timestamp:                   2022-10-26T21:10:43Z
    Host:                        Linux 6.0.3-1-default x86_64
    CMake:                       3.24.2
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/gmake
    Configuration:               Debug

  CPU/HW features:
    Baseline:                    SSE SSE2 SSE3
      requested:                 SSE3
    Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
      requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
      SSE4_1 (16 files):         + SSSE3 SSE4_1
      SSE4_2 (1 files):          + SSSE3 SSE4_1 POPCNT SSE4_2
      FP16 (0 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
      AVX (4 files):             + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
      AVX2 (32 files):           + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
      AVX512_SKX (5 files):      + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                /usr/bin/c++  (ver 12.2.1)
    C++ flags (Release):         -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -Wl,--exclude-libs,libippicv.a -Wl,--exclude-libs,libippiw.a   -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined  
    Linker flags (Debug):        -Wl,--exclude-libs,libippicv.a -Wl,--exclude-libs,libippiw.a   -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined  
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          dl m pthread rt
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 aruco barcode bgsegm bioinspired calib3d ccalib core cvv datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 alphamat cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev hdf java julia matlab ovis python2 sfm ts viz
    Applications:                apps
    Documentation:               NO
    Non-free algorithms:         NO

  GUI:                           QT5
    QT:                          YES (ver 5.15.6 )
      QT OpenGL support:         YES (Qt5::OpenGL 5.15.6)
    GTK+:                        YES (ver 3.24.34)
      GThread :                  YES (ver 2.74.0)
      GtkGlExt:                  YES (ver 1.2.0)
    OpenGL support:              YES (/lib64/libGL.so /lib64/libGLU.so)
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /lib64/libz.so (ver 1.2.12)
    JPEG:                        /lib64/libjpeg.so (ver 62)
    WEBP:                        /lib64/libwebp.so (ver encoder: 0x020f)
    PNG:                         /lib64/libpng.so (ver 1.6.38)
    TIFF:                        /lib64/libtiff.so (ver 42 / 4.4.0)
    JPEG 2000:                   OpenJPEG (ver 2.5.0)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      YES (2.2.6)
    FFMPEG:                      YES
      avcodec:                   YES (59.37.100)
      avformat:                  YES (59.27.100)
      avutil:                    YES (57.28.100)
      swscale:                   YES (6.7.100)
      avresample:                NO
    GStreamer:                   YES (1.20.4)
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            pthreads

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Intel IPP:                   2020.0.0 Gold [2020.0.0]
           at:                   /home/elchaschab/devel/opencv/build/3rdparty/ippicv/ippicv_lnx/icv
    Intel IPP IW:                sources (2020.0.0)
              at:                /home/elchaschab/devel/opencv/build/3rdparty/ippicv/ippicv_lnx/iw
    VA:                          YES
    Lapack:                      YES (/lib64/libopenblas.so)
    Eigen:                       NO
    Custom HAL:                  NO
    Protobuf:                    build (3.19.1)

  OpenCL:                        YES (SVM INTELVA)
    Include path:                /home/elchaschab/devel/opencv/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.10.7)
    Libraries:                   /lib64/libpython3.10.so (ver 3.10.7)
    numpy:                       /usr/lib64/python3.10/site-packages/numpy/core/include (ver 1.21.6)
    install path:                lib/python3.10/site-packages/cv2/python-3.10

  Python (for build):            /usr/bin/python2.7

  Java:                          
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /usr/local
-----------------------------------------------------------------

static cv::ocl::OpenCLExecutionContext& cv::ocl::OpenCLExecutionContext::getCurrent()
    bool cv::ocl::haveOpenCL()
[ INFO:0@0.011] global /home/elchaschab/devel/opencv/modules/core/src/ocl.cpp (1186) haveOpenCL Initialize OpenCL runtime...
[ INFO:0@0.031] global /home/elchaschab/devel/opencv/modules/core/src/ocl.cpp (1192) haveOpenCL OpenCL: found 1 platforms
[ INFO:0@0.031] global /home/elchaschab/devel/opencv/modules/core/src/ocl.cpp (984) getInitializedExecutionContext OpenCL: initializing thread execution context
[ INFO:0@0.031] global /home/elchaschab/devel/opencv/modules/core/src/ocl.cpp (994) getInitializedExecutionContext OpenCL: creating new execution context...
[ INFO:0@0.031] global /home/elchaschab/devel/opencv/modules/core/src/ocl.cpp (1012) getInitializedExecutionContext OpenCL: device=Intel(R) Iris(R) Xe Graphics [0x9a40]
static cv::ocl::Context::Impl* cv::ocl::Context::Impl::findOrCreateContext(cl_context)
    static cv::ocl::Context::Impl* cv::ocl::Context::Impl::findContext(const std::string&)
cv::VideoCapture::VideoCapture(const cv::String&, int, const std::vector<int>&)
    virtual bool cv::VideoCapture::open(const cv::String&, int, const std::vector<int>&)
[DEBUG:0@0.032] global /home/elchaschab/devel/opencv/modules/videoio/src/videoio_registry.cpp (206) VideoBackendRegistry VIDEOIO: Builtin backends(9): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); V4L2(970); CV_IMAGES(960); CV_MJPEG(950); FIREWIRE(940); UEYE(930); OBSENSOR(920)
[DEBUG:0@0.032] global /home/elchaschab/devel/opencv/modules/videoio/src/videoio_registry.cpp (230) VideoBackendRegistry VIDEOIO: Available backends(9): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); V4L2(970); CV_IMAGES(960); CV_MJPEG(950); FIREWIRE(940); UEYE(930); OBSENSOR(920)
[ INFO:0@0.032] global /home/elchaschab/devel/opencv/modules/videoio/src/videoio_registry.cpp (232) VideoBackendRegistry VIDEOIO: Enabled backends(9, sorted by priority): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); V4L2(970); CV_IMAGES(960); CV_MJPEG(950); FIREWIRE(940); UEYE(930); OBSENSOR(920)
[ WARN:0@0.032] global /home/elchaschab/devel/opencv/modules/videoio/src/cap.cpp (130) open VIDEOIO(FFMPEG): trying capture filename='/dev/video1' ...
[OPENCV:FFMPEG:48] Opening '/dev/video1' for reading
[OPENCV:FFMPEG:48] Setting default whitelist 'file,crypto,data'
[OPENCV:FFMPEG:40] Statistics: 0 bytes read, 0 seeks
warning: Error opening file (/home/elchaschab/devel/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:1139)
warning: /dev/video1 (/home/elchaschab/devel/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:1140)
[ WARN:0@0.148] global /home/elchaschab/devel/opencv/modules/videoio/src/cap.cpp (153) open VIDEOIO(FFMPEG): can't create capture
ERROR! Unable to open camera
[ INFO:0@0.150] global /home/elchaschab/devel/opencv/modules/core/src/trace.cpp (882) ~TraceManager Trace: Total events: 9
[ WARN:0@0.150] global /home/elchaschab/devel/opencv/modules/core/src/trace.cpp (886) ~TraceManager Trace: Total skipped events: 10

Steps to reproduce

Build OpenCV 4.x with:

ccmake -DCMAKE_BUILD_TYPE=Debug -DWITH_OPENGL=ON -DWITH_FFMPEG=ON -DWITH_QT=ON -DOPENCV_FFMPEG_SKIP_BUILD_CHECK=ON -DWITH_VA=ON -DWITH_VA_INTEL=ON -DBUILD_PERF_TESTS=OFF -DBUILD_TESTS=OFF -DOPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-4.x/modules/ ..

Clone and build my example:

git clone https://github.com/kallaballa/OpenCV-Issues.git
cd OpenCV-Issues
make

run it on a video device and it will fail. e.g:

src/v4l2-vaapi/v4l2-vaapi /dev/video1

ffplay works:

ffplay /dev/video1

Run it on an example (attached) MJPEG file (stream copied from V4l2).

src/v4l2-vaapi/v4l2-vaapi example.mkv

The example file was created with following command:

ffmpeg -y -t 10 -video_size 1280x720 -input_format mjpeg -framerate 30 -i /dev/video1 -f matroska -c:v copy example.mkv

Please rename to example.mkv

vainfo:

Trying display: wayland
Trying display: x11
libva info: VA-API version 1.16.0
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /usr/local/lib64/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_16
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.16 (libva 2.16.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 22.1.1 (066a864)
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointFEI
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointFEI
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointFEI
      VAProfileHEVCMain               : VAEntrypointEncSliceLP
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointEncSliceLP
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointEncSliceLP
      VAProfileVP9Profile1            : VAEntrypointVLD
      VAProfileVP9Profile1            : VAEntrypointEncSliceLP
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointEncSliceLP
      VAProfileVP9Profile3            : VAEntrypointVLD
      VAProfileVP9Profile3            : VAEntrypointEncSliceLP
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileHEVCMain12             : VAEntrypointEncSlice
      VAProfileHEVCMain422_10         : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointEncSlice
      VAProfileHEVCMain422_12         : VAEntrypointVLD
      VAProfileHEVCMain422_12         : VAEntrypointEncSlice
      VAProfileHEVCMain444            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         : VAEntrypointVLD
      VAProfileHEVCMain444_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_12         : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain10          : VAEntrypointVLD
      VAProfileHEVCSccMain10          : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444         : VAEntrypointVLD
      VAProfileHEVCSccMain444         : VAEntrypointEncSliceLP
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointEncSliceLP

v4l2-ctl --list-formats-ext --device /dev/video1:

ioctl: VIDIOC_ENUM_FMT
    Type: Video Capture

    [0]: 'MJPG' (Motion-JPEG, compressed)
        Size: Discrete 1280x720
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 320x180
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 320x240
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 352x288
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 424x240
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 640x360
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 640x480
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 848x480
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 960x540
            Interval: Discrete 0.033s (30.000 fps)
    [1]: 'YUYV' (YUYV 4:2:2)
        Size: Discrete 640x480
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 320x180
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 320x240
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 352x288
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 424x240
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 640x360
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 848x480
            Interval: Discrete 0.050s (20.000 fps)
        Size: Discrete 960x540
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 1280x720
            Interval: Discrete 0.100s (10.000 fps)

Issue submission checklist

kallaballa commented 2 years ago

The only workaround i found is this reasonably low latency pipe:

ffmpeg -v quiet -fflags nobuffer -fflags discardcorrupt -flags low_delay -avioflags direct -video_size 1280x720 -input_format mjpeg -framerate 30 -i /dev/video1 -f matroska -c:v copy pipe:/dev/stdout | src/v4l2-vaapi/v4l2-vaapi /dev/stdin