GStreamer memory:DMABuf is not supported in OpenCV

nedo99 commented 1 year ago

System Information

OpenCV version: 4.7.0 Operating System / Platform: Ubuntu 22.04 Compiler & compiler version: GCC 11.3.0

Detailed description

I want to run video capture using Gstreamer backend. As I see OpenCV is accepting only appsink (which makes things complicated), but found the workaround for that one, but facing the issue where gstremeamer pipeline works when running from the command line, but once used with VideoCapture then it fails.

Video config for OpenCV cmake:

--   Video I/O:
--     DC1394:                      YES (2.2.6)
--     FFMPEG:                      YES
--       avcodec:                   YES (58.134.100)
--       avformat:                  YES (58.76.100)
--       avutil:                    YES (56.70.100)
--       swscale:                   YES (5.9.100)
--       avresample:                NO
--     GStreamer:                   YES (1.20.3)
--     v4l/v4l2:                    YES (linux/videodev2.h)
--     Intel Media SDK:             YES (VPL::dispatcher VPL::api VPL::cppapi /usr/lib/x86_64-linux-gnu/libva.so /usr/lib/x86_64-linux-gnu/libva-drm.so)

Steps to reproduce

If I run in command line:

gst-launch-1.0 filesrc location=/path/to/video ! qtdemux ! vah264dec ! 'video/x-raw(memory:DMABuf)' ! appsink sync=false

it works.

If I run the same pipeline with VideoCapture, for instance:

ostringstream line = "filesrc location=/path/to/video ! qtdemux ! vah264dec ! 'video/x-raw(memory:DMABuf)' ! appsink sync=false";
makePtr<VideoCapture>(line.str(), CAP_GSTREAMER);

I get:

Created GStreamer capture  ( filesrc location="/path/to/video" ! qtdemux ! vah264dec ! video/x-raw(memory:DMABuf) ! appsink sync=false )
[ WARN:0@0.087] global cap_gstreamer.cpp:2784 handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module qtdemux0 reported: Internal data stream error.
[ WARN:0@0.088] global cap_gstreamer.cpp:1679 open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0@0.088] global cap_gstreamer.cpp:1164 isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

Issue submission checklist

[X] I report the issue, it's not a question
[X] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
[ ] I updated to the latest OpenCV version and the issue is still there
[x] There is reproducer code and related data files (videos, images, onnx, etc)

mshabunin commented 1 year ago

I believe OpenCV does not support memory:DMABuf. Try to remove explicit filter before appsink, OpenCV should add its own.

Also videoconvert node might be needed as OpenCV only accepts BGR frames by default.

nedo99 commented 1 year ago

I see. Is there a timeline for when this will be supported?

mshabunin commented 1 year ago

Currently there are no such plans AFAIK.

asmorkalov commented 4 months ago

I'm looking on the hardware acceleration topic right now and found several items to try:

Use vaapipostproc: https://gstreamer.freedesktop.org/documentation/vaapi/vaapipostproc.html?gi-language=c. It has sink with video/x-raw and many color space options.
Relay on automatically assembled pipeline with vaapidecodebin, e.g. filesrc location=<filename> ! avidemux ! vaapidecodebin ! appsink. There was some benchmark and it worked before for OpenCV: https://github.com/opencv/opencv/wiki/Video-capture-and-write-benchmark

opencv / opencv