search

Xilinx / video-sdk

https://xilinx.github.io/video-sdk
Other
31 stars 14 forks source link

-copyts option does not work #9

Closed ri0007 closed 1 year ago

ri0007 commented 2 years ago

(Since I am not very good at English, I use the DeepL translation. Sorry.)

Hi. I want to apply a watermark to a fragment of an HLS video that is being streamed live, and deliver the watermarked video to my viewers. Therefore, we used the following command on an AWS vt1.3xlarge instance to transcode the .ts file, which is a fragment of HLS, while applying filters.

$ ffmpeg -y  -copyts -c:v mpsoc_vcu_h264 -i input.ts -i filter.png -filter_complex '[0]xvbm_convert[vid], [1]scale=500:100[logo], [vid][logo]overlay=16:16' -c:v mpsoc_vcu_h264 -c:a copy output.ts
ffmpeg version 4.1.9 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100

<<<<<<<==  FFmpeg xrm ===>>>>>>>>
Input #0, mpegts, from 'input.ts':
  Duration: 00:00:05.13, start: 4997.665667, bitrate: 5499 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
    Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s
Input #1, png_pipe, from 'filter.png':
  Duration: N/A, bitrate: N/A
    Stream #1:0: Video: png, rgba(pc), 624x624, 25 tbr, 25 tbn, 25 tbc
------------------------------------------------------------

   xclbin_name :  /opt/xilinx/xcdr/xclbins/transcode.xclbin
   device_id   :  0 
------------------------------------------------------------

Stream mapping:
  Stream #0:0 (mpsoc_vcu_h264) -> xvbm_convert
  Stream #1:0 (png) -> scale
  overlay -> Stream #0:0 (mpsoc_vcu_h264)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[Parsed_xvbm_convert_0 @ 0x5646f37fcd40] xvbm_conv:: wait for conversion to finish...
[mpsoc_vcu_h264 @ 0x5646f3a15840] Custom Rate Control Mode is Disabled
Output #0, mpegts, to 'output.ts':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (mpsoc_vcu_h264), nv12, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 5000 kb/s, 30 fps, 90k tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc58.35.100 mpsoc_vcu_h264
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s
[Parsed_xvbm_convert_0 @ 0x5646f37fcd40] xvbm_conv:: wait for conversion to finish...
frame=  150 fps=0.0 q=-0.0 Lsize=    3451kB time=01:23:22.67 bitrate=   5.7kbits/s speed=5.66e+03x    
video:3098kB audio:82kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.506529%

If you use ffprobe to display the information in the .ts file output by this command, you will see that the values for duration and start have changed from the original .ts file, even though -copyts option was used. (I used the copy-ts option and expected the timestamp information to be passed on to the output ts file, but in fact it is not.)

$ ffprobe -i input.ts -hide_banner
Input #0, mpegts, from 'input.ts':
  Duration: 00:00:05.13, start: 4997.665667, bitrate: 5499 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
    Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s
$ ffprobe -i output.ts -hide_banner
Input #0, mpegts, from 'output.ts':
  Duration: 01:23:22.68, start: 1.433333, bitrate: 5 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
    Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s

Also, the copyts option is not working even when the filter is not applied as shown below.

ffmpeg -y -copyts -c:v mpsoc_vcu_h264 -i input.ts -c:v mpsoc_vcu_h264 -c:a copy output.ts
ffmpeg version 4.1.9 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100

<<<<<<<==  FFmpeg xrm ===>>>>>>>>
Input #0, mpegts, from 'input.ts':
  Duration: 00:00:05.13, start: 4997.665667, bitrate: 5499 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
    Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s
------------------------------------------------------------

   xclbin_name :  /opt/xilinx/xcdr/xclbins/transcode.xclbin
   device_id   :  0 
------------------------------------------------------------

Stream mapping:
  Stream #0:0 -> #0:0 (h264 (mpsoc_vcu_h264) -> h264 (mpsoc_vcu_h264))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mpsoc_vcu_h264 @ 0x5607579e1640] Custom Rate Control Mode is Disabled
Output #0, mpegts, to 'output.ts':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (mpsoc_vcu_h264), xlnx_xvbm, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 5000 kb/s, 30 fps, 90k tbn, 30 tbc
    Metadata:
      encoder         : Lavc58.35.100 mpsoc_vcu_h264
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s
frame=  150 fps=0.0 q=-0.0 Lsize=    3456kB time=01:23:22.67 bitrate=   5.7kbits/s speed=5.78e+03x    
video:3103kB audio:82kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.523341%

$ ffprobe -i output.ts -hide_banner
Input #0, mpegts, from 'output.ts':
  Duration: 01:23:22.68, start: 1.433333, bitrate: 5 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
    Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s

Is there anything I can do to solve this problem? Or does mpsoc_vcu_h264 not support -copyts option?

jmouroux commented 2 years ago

Hi @ri0007

Thank you for highlighting this issue and apologies for the late reply.

In order to better understand the problem, could you clarify the following : • Can you try to run this against libx265 or libx264 within ffmpeg for both decode & encode, to see if the behavior there differs? • If the clip is publicly available (input.ts), can you point us to the clip? • Upon running ffprobe on input.ts can you confirm if you see any of the following messages or not :

[mpegts @ 0x5566a79ef080] Invalid timestamps stream=0, pts=903003, dts=906006, size=1288

ri0007 commented 2 years ago

Hi @jmouroux

Thanks for the reply too.

Can you try to run this against libx265 or libx264 within ffmpeg for both decode & encode, to see if the behavior there differs?

The following are the execution results when mpsoc_vcu_h264 is not used for both encoder and decoder, but h264 is used for decoder and libx264 is used for encoder.

The command with -filter-complex causes an error (No such filter: 'xvbm_convert') and the encoding cannot be performed. (I know that this behavior is probably natural)

ffmpeg -y  -copyts -c:v h264 -i input.ts -i filter.png -filter_complex '[0]xvbm_convert[vid], [1]scale=500:100[logo], [vid][logo]overlay=16:16' -c:v libx264 -c:a copy output.ts
ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, mpegts, from 'input.ts':
  Duration: 00:00:05.13, start: 4997.665667, bitrate: 5499 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
    Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s
Input #1, png_pipe, from 'filter.png':
  Duration: N/A, bitrate: N/A
    Stream #1:0: Video: png, rgba(pc), 624x624, 25 tbr, 25 tbn, 25 tbc
[AVFilterGraph @ 0x563eda13b840] No such filter: 'xvbm_convert'
Error initializing complex filters.
Invalid argument

The following is the result of running the command without -filter-complex

$ ffmpeg -y -copyts -c:v h264 -i input.ts -c:v libx264 -c:a copy output.ts
ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, mpegts, from 'input.ts':
  Duration: 00:00:05.13, start: 4997.665667, bitrate: 5499 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
    Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[libx264 @ 0x55705a80dd60] using SAR=1/1
[libx264 @ 0x55705a80dd60] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[libx264 @ 0x55705a80dd60] profile High, level 3.1
Output #0, mpegts, to 'output.ts':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (libx264), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 30 fps, 90k tbn, 30 tbc
    Metadata:
      encoder         : Lavc57.107.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s
frame=  150 fps=105 q=-1.0 Lsize=    1856kB time=01:23:22.70 bitrate=   3.0kbits/s speed=3.5e+03x    
video:1619kB audio:82kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.124640%
[libx264 @ 0x55705a80dd60] frame I:1     Avg QP:24.19  size: 44731
[libx264 @ 0x55705a80dd60] frame P:38    Avg QP:23.30  size: 18035
[libx264 @ 0x55705a80dd60] frame B:111   Avg QP:25.81  size:  8358
[libx264 @ 0x55705a80dd60] consecutive B-frames:  1.3%  0.0%  0.0% 98.7%
[libx264 @ 0x55705a80dd60] mb I  I16..4:  9.0% 66.3% 24.7%
[libx264 @ 0x55705a80dd60] mb P  I16..4:  4.9% 26.4%  3.9%  P16..4: 30.4%  9.7%  4.2%  0.0%  0.0%    skip:20.6%
[libx264 @ 0x55705a80dd60] mb B  I16..4:  1.5%  7.0%  0.9%  B16..8: 31.6%  6.9%  1.0%  direct: 2.2%  skip:48.9%  L0:41.6% L1:48.2% BI:10.1%
[libx264 @ 0x55705a80dd60] 8x8 transform intra:74.3% inter:82.8%
[libx264 @ 0x55705a80dd60] coded y,uvDC,uvAC intra: 59.4% 12.5% 1.0% inter: 13.7% 5.5% 0.4%
[libx264 @ 0x55705a80dd60] i16 v,h,dc,p:  7% 50%  9% 34%
[libx264 @ 0x55705a80dd60] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  8% 41% 15%  5%  4%  3%  7%  3% 13%
[libx264 @ 0x55705a80dd60] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu:  9% 50% 12%  4%  4%  3%  5%  2% 11%
[libx264 @ 0x55705a80dd60] i8c dc,h,v,p: 78% 13%  8%  0%
[libx264 @ 0x55705a80dd60] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x55705a80dd60] ref P L0: 62.8% 12.8% 18.1%  6.2%
[libx264 @ 0x55705a80dd60] ref B L0: 90.3%  7.0%  2.7%
[libx264 @ 0x55705a80dd60] ref B L1: 96.6%  3.4%
[libx264 @ 0x55705a80dd60] kb/s:2652.48

The result of checking this output.ts in ffprobe is as follows.

$ ffprobe -i output.ts -hide_banner
Input #0, mpegts, from 'output.ts':
  Duration: 00:00:05.13, start: 4999.065667, bitrate: 2962 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
    Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s

If the clip is publicly available (input.ts), can you point us to the clip?

It's not available on the Web, so I'll share it in a zip file. issue_9.zip

Upon running ffprobe on input.ts can you confirm if you see any of the following messages or not

there is no such message displayed.

NastoohX commented 1 year ago

Hi, The issue has been noted and resolved in the upcoming version.