Video composition using USB cameras

[jbolduc01]

Hello, I was wondering if there was a way using VAAPI in order to compose / mix USB video cameras and do some PiP layouts, side by side video and put the result over vaapisink using gstreamer? I see that gstreamer have "videomixer" and "compositor" plugins but these are not hardware accelerated. Do you know any mean how we can get this to work? We have minimal performance when using a gstreamer pipeline like this one:

LIBVA_DRIVER_NAME=i965 gst-launch-1.0 compositor name=comp background=1 \
sink_0::xpos=0 sink_0::ypos=0 \
sink_1::xpos=1920 sink_1::ypos=0 \
sink_2::xpos=0 sink_2::ypos=1080  \
sink_3::xpos=1920 sink_3::ypos=1080 \
! queue ! fpsdisplaysink video-sink="vaapisink force-modesetting=true" text-overlay=false sync=true \
v4l2src device=/dev/video0 io-mode=dmabuf ! 'video/x-raw,width=1920,height=1080,framerate=30/1,format=YUY2'  ! comp. \
v4l2src device=/dev/video2 io-mode=dmabuf ! 'video/x-raw,width=1920,height=1080,framerate=30/1,format=YUY2'  ! comp.  \
v4l2src device=/dev/video4 io-mode=dmabuf ! 'video/x-raw,width=1920,height=1080,framerate=30/1,format=YUY2'  ! comp.  \
v4l2src device=/dev/video8 io-mode=dmabuf ! 'image/jpeg,width=1920,height=1080,framerate=30/1'  ! vaapijpegdec ! comp.  -v

Thanks for your help. Jérôme

[uartie]

VAAPI subpicture API's, perhaps?

Also, a vaapioverlay plugin was recently added to gstreamer-vaapi, but only iHD driver intel/media-driver can support it.

Also, it is probably better to ask this question in the gstreamer-vaapi project. IIRC, there is some subpicture code in gstreamer-vaapi, but not sure how/if it works or how to activate those paths with a gst pipeline command. Maybe check the gstreamer-vaapi tests for clues?

[jbolduc01]

Thanks for your feeback! Can you tell me how to use this vaapioverlay plugin? I am testing over Ubuntu 19.10 where i can install iHD driver through apt. I have installed gstreamer1.0-vaapi package but as soon as I do: export LIBVA_DRIVER_NAME=iHD Here is what I get over vainfo:

libva info: VA-API version 1.5.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_5
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.5 (libva 2.5.0)
vainfo: Driver version: Intel iHD driver - 1.0.0
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD

When listing VAAPI plugins here is what I get:

gst-inspect-1.0 | grep vaapi
vaapi:  vaapijpegdec: VA-API JPEG decoder
vaapi:  vaapimpeg2dec: VA-API MPEG2 decoder
vaapi:  vaapih264dec: VA-API H264 decoder
vaapi:  vaapivp8dec: VA-API VP8 decoder
vaapi:  vaapivp9dec: VA-API VP9 decoder
vaapi:  vaapih265dec: VA-API H265 decoder
vaapi:  vaapipostproc: VA-API video postprocessing
vaapi:  vaapidecodebin: VA-API Decode Bin
vaapi:  vaapisink: VA-API sink
vaapi:  vaapijpegenc: VA-API JPEG encoder
vaapi:  vaapih264enc: VA-API H264 encoder

There is no vaapioverlay plugin and also when I do: gst-launch-1.0 v4l2src device=/dev/video4 ! image/jpeg,width=3840,height=2160 ! vaapijpegdec ! fpsdisplaysink video-sink="vaapisink fullscreen=true" text-overlay=false sync=false -v here is what I get:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Got context from element 'vaapisink0': gst.vaapi.Display=context, gst.vaapi.Display=(GstVaapiDisplay)"\(GstVaapiDisplayGLX\)\ vaapidisplayglx0";
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstVaapiSink:vaapisink0: sync = false
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = image/jpeg, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = image/jpeg, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstVaapiDecode_jpeg:vaapidecode_jpeg0.GstPad:sink: caps = image/jpeg, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = image/jpeg, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
Caught SIGSEGV
Spinning.  Please run 'gdb gst-launch-1.0 13009' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

I know it is maybe related to gstreamer but I can find issue section on their git rep.

Thanks, Jerome

[uartie]

The 'vaapioverlay' plugin is relatively new and has not been included in any release, yet. So, you will need to compile the latest gstreamer-vaapi from source. The 'vaapioverlay' plugin usage is basically the same as the 'compositor' plugin. You also need the iHD driver with VPP supported, 'intel-media-va-driver-non-free'. I'm not sure if this distro package provides new enough iHD, though... so you might need to compile that from source, too.