search

Xilinx / video-sdk

https://xilinx.github.io/video-sdk
Other
31 stars 14 forks source link

Complex filter with smartblur and alphamerge #48

Open adeelabbas opened 1 year ago

adeelabbas commented 1 year ago

Hi, I am running a complex filter command which involves smartblur, alphamerge and overlay filters. The command is:

/opt/ffmpeg/bin/ffmpeg -y -hide_banner \
 -t 00:11:01.897 -i Video1.MP4 \
 -t 00:11:01.897 -i Video2 \
 -t 00:11:01.897 -i Audio.M4A \
-filter_complex " \
[0:v]curves=all='0.0/0.1 0.09/0.15 1/1',split[vinpA0][vinpB0]; [vinpA0][1:v]alphamerge[merge0]; [vinpB0]smartblur=luma_radius=5:chroma_radius=0[bokeh0]; [bokeh0][merge0]overlay,setpts=PTS-STARTPTS[vout0]" -map "[vout0]" \
  -temporal-aq 1 -tag:v avc1 -c:v mpsoc_vcu_h264 -g 144 -b:v 8709120 -bf 1 -threads 8 -spatial-aq 1 -color_primaries bt709 -color_trc bt709 -colorspace smpte170m -map 2:a:0? -c:a copy -ss 00:00:00.041 output.mp4

I am observing a very slow performance (20fps). If I remove smartblur, alphamerge and overlay filters, the performance jumps to 80fps. Can someone explain how I can get best performance for this scenario.

I tried to put xvbm_convert at the beginning of curves command but that did not make any difference.

NastoohX commented 1 year ago

Hi, Thank you for bring this to our attention. I have confirmed the issue and asked our dev team for a remedy if possible. Specifically, running both filter and encode on cpu seems to achieve the desired fps, with my test inputs. It seems that this pipeline is cpu limited due to communication between cpu and U30. (Note the output of the top command in the case where libx264 is replaced with hw encoder, with some modification to encoder options.) Meanwhile, can you provide the output of the following commands: 1- ffmpeg -version 2- ffmpeg -filters | grep -i smartblur Cheers,

adeelabbas commented 1 year ago 
ffmpeg -version

outputs

ffmpeg version n4.4.xlnx.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/opt/ffmpeg/ --datadir=/opt/ffmpeg/etc --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --enable-gpl --enable-nonfree --extra-libs=-ldl --disable-static --enable-shared
  WARNING: library configuration mismatch
  avutil      configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  avcodec     configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  avformat    configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  avdevice    configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  avfilter    configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  swscale     configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  swresample  configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  postproc    configuration: --prefix=/tmp/ffmpeg --datadir=/tmp/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared --enable-gpl --enable-nonfree --extra-libs=-ldl
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  3.100
adeelabbas commented 1 year ago 
ffmpeg -filters | grep -i smartblur

outputs

ffmpeg version n4.4.xlnx.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/opt/ffmpeg/ --datadir=/opt/ffmpeg/etc --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --enable-gpl --enable-nonfree --extra-libs=-ldl --disable-static --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100

<<<<<<<==  FFmpeg xrm ===>>>>>>>>
No device set hence falling to default device 0
------------------i=0------------------------------------------

   xclbin_name :  /opt/xilinx/xcdr/xclbins/transcode.xclbin
   device_id   :  0
------------------------------------------------------------

 T.. smartblur         V->V       Blur the input video without impacting the outlines.

Please note that default ffmpeg that ships with VT1 AMI image does not have smartblur - so I had to rebuild ffmpeg.

duthv-amd commented 1 year ago

@NastoohX is this issue still open, can we close it if it has been resolved?

NastoohX commented 1 year ago

Hi, This issue has been filed internally, under https://jira.xilinx.com/browse/CR-1146467. Updates will be provided here as they become available. Cheers,