Xilinx / video-sdk

https://xilinx.github.io/video-sdk
Other
31 stars 14 forks source link

live stream (rtmp) input w/ output scaler #51

Closed scottf-tvw closed 1 year ago

scottf-tvw commented 1 year ago

I am using a aws VT1 and wanted to try out the scaler for HLS transcoding. I swapped out the bars and tone filter in the live hls example with a rtmp live stream input and it crashes on the filter-graphs for the scaler. my first assumption is that its a syntax mistake by me. but I wanted to pose the question here in case the documentation needed updating or there is an actual issue happening. Other means of HLS live transcode with filters not using multiscale_xma work just fine. I just wanted to take advantage of the scaler.

ffmpeg -c:v mpsoc_vcu_h264 -i rtmp://ingress.w1.invintus.com/srcEncoders/247airtestfeed \
-filter_complex "multiscale_xma=outputs=5: \
 out_1_width=1920: out_1_height=1080: out_1_rate=full: \
 out_2_width=1280: out_2_height=720:  out_2_rate=full: \
 out_3_width=848:  out_3_height=480:  out_3_rate=full: \
 out_4_width=640:  out_4_height=360:  out_4_rate=full: \
 out_5_width=288:  out_5_height=160:  out_5_rate=full  \
 [vid1][vid2][vid3][vid4][vid5]; [1]volume=1[aud1]; [2]volume=1[aud2]; [3]volume=1[aud3]; [4]volume=1[aud4]; [5]volume=1[aud5]" \
-map "[vid1]" -b:v:0 2M   -minrate:v:0 2M   -maxrate:v:0 2M   -bufsize:v:0 4M   -c:v:0 mpsoc_vcu_h264 \
-map "[vid2]" -b:v:1 1M   -minrate:v:1 1M   -maxrate:v:1 1M   -bufsize:v:1 1M   -c:v:1 mpsoc_vcu_h264 \
-map "[vid3]" -b:v:2 750K -minrate:v:2 750K -maxrate:v:2 750K -bufsize:v:2 750K -c:v:2 mpsoc_vcu_h264 \
-map "[vid4]" -b:v:3 375K -minrate:v:2 375K -maxrate:v:2 375K -bufsize:v:3 375K -c:v:3 mpsoc_vcu_h264 \
-map "[vid5]" -b:v:4 250k -minrate:v:4 250k -maxrate:v:4 250k -bufsize:v:4 250k -c:v:4 mpsoc_vcu_h264 \
-map "[aud1]" -c:a:0 aac \
-map "[aud2]" -c:a:1 aac \
-map "[aud3]" -c:a:2 aac \
-map "[aud4]" -c:a:3 aac \
-map "[aud5]" -c:a:4 aac \
-var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2 v:3,a:3 v:4,a:4" \
-f hls \
-hls_wrap 5 \
-hls_time 6 \
-master_pl_name "test.m3u8" -hls_segment_filename  "${HLS_DIR}/test_%v-%d.ts" "${HLS_DIR}/test_%v.m3u8"

here is the full debug output from ffmpeg:


ffmpeg version n4.4.xlnx.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100

<<<<<<<==  FFmpeg xrm ===>>>>>>>>
Splitting the commandline.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'mpsoc_vcu_h264'.
Reading option '-i' ... matched as input url with argument 'rtmp://ingress.w1.invintus.com/srcEncoders/247airtestfeed'.
Reading option '-filter_complex' ... matched as option 'filter_complex' (create a complex filtergraph) with argument 'multiscale_xma=outputs=5:  out_1_width=1920: out_1_height=1080: out_1_rate=full:  out_2_width=1280: out_2_height=720:  out_2_rate=full:  out_3_width=848:  out_3_height=480:  out_3_rate=full:  out_4_width=640:  out_4_height=360:  out_4_rate=full:  out_5_width=288:  out_5_height=160:  out_5_rate=full   [vid1][vid2][vid3][vid4][vid5]; [1]volume=1[aud1]; [2]volume=1[aud2]; [3]volume=1[aud3]; [4]volume=1[aud4]; [5]volume=1[aud5]'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[vid1]'.
Reading option '-b:v:0' ... matched as option 'b' (video bitrate (please use -b:v)) with argument '2M'.
Reading option '-minrate:v:0' ... matched as AVOption 'minrate:v:0' with argument '2M'.
Reading option '-maxrate:v:0' ... matched as AVOption 'maxrate:v:0' with argument '2M'.
Reading option '-bufsize:v:0' ... matched as AVOption 'bufsize:v:0' with argument '4M'.
Reading option '-c:v:0' ... matched as option 'c' (codec name) with argument 'mpsoc_vcu_h264'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[vid2]'.
Reading option '-b:v:1' ... matched as option 'b' (video bitrate (please use -b:v)) with argument '1M'.
Reading option '-minrate:v:1' ... matched as AVOption 'minrate:v:1' with argument '1M'.
Reading option '-maxrate:v:1' ... matched as AVOption 'maxrate:v:1' with argument '1M'.
Reading option '-bufsize:v:1' ... matched as AVOption 'bufsize:v:1' with argument '1M'.
Reading option '-c:v:1' ... matched as option 'c' (codec name) with argument 'mpsoc_vcu_h264'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[vid3]'.
Reading option '-b:v:2' ... matched as option 'b' (video bitrate (please use -b:v)) with argument '750K'.
Reading option '-minrate:v:2' ... matched as AVOption 'minrate:v:2' with argument '750K'.
Reading option '-maxrate:v:2' ... matched as AVOption 'maxrate:v:2' with argument '750K'.
Reading option '-bufsize:v:2' ... matched as AVOption 'bufsize:v:2' with argument '750K'.
Reading option '-c:v:2' ... matched as option 'c' (codec name) with argument 'mpsoc_vcu_h264'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[vid4]'.
Reading option '-b:v:3' ... matched as option 'b' (video bitrate (please use -b:v)) with argument '375K'.
Reading option '-minrate:v:2' ... matched as AVOption 'minrate:v:2' with argument '375K'.
Reading option '-maxrate:v:2' ... matched as AVOption 'maxrate:v:2' with argument '375K'.
Reading option '-bufsize:v:3' ... matched as AVOption 'bufsize:v:3' with argument '375K'.
Reading option '-c:v:3' ... matched as option 'c' (codec name) with argument 'mpsoc_vcu_h264'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[vid5]'.
Reading option '-b:v:4' ... matched as option 'b' (video bitrate (please use -b:v)) with argument '250k'.
Reading option '-minrate:v:4' ... matched as AVOption 'minrate:v:4' with argument '250k'.
Reading option '-maxrate:v:4' ... matched as AVOption 'maxrate:v:4' with argument '250k'.
Reading option '-bufsize:v:4' ... matched as AVOption 'bufsize:v:4' with argument '250k'.
Reading option '-c:v:4' ... matched as option 'c' (codec name) with argument 'mpsoc_vcu_h264'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[aud1]'.
Reading option '-c:a:0' ... matched as option 'c' (codec name) with argument 'aac'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[aud2]'.
Reading option '-c:a:1' ... matched as option 'c' (codec name) with argument 'aac'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[aud3]'.
Reading option '-c:a:2' ... matched as option 'c' (codec name) with argument 'aac'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[aud4]'.
Reading option '-c:a:3' ... matched as option 'c' (codec name) with argument 'aac'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '[aud5]'.
Reading option '-c:a:4' ... matched as option 'c' (codec name) with argument 'aac'.
Reading option '-var_stream_map' ... matched as AVOption 'var_stream_map' with argument 'v:0,a:0 v:1,a:1 v:2,a:2 v:3,a:3 v:4,a:4'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'hls'.
Reading option '-hls_wrap' ... matched as AVOption 'hls_wrap' with argument '5'.
Reading option '-hls_time' ... matched as AVOption 'hls_time' with argument '6'.
Reading option '-master_pl_name' ... matched as AVOption 'master_pl_name' with argument 'test.m3u8'.
Reading option '-hls_segment_filename' ... matched as AVOption 'hls_segment_filename' with argument '/var/www/hls/test_%v-%d.ts'.
Reading option '/var/www/hls/test_%v.m3u8' ... matched as output url.
No device set hence falling to default device 0
------------------i=0------------------------------------------

   xclbin_name :  /opt/xilinx/xcdr/xclbins/transcode.xclbin
   device_id   :  0
------------------------------------------------------------

Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument debug.
Applying option filter_complex (create a complex filtergraph) with argument multiscale_xma=outputs=5:  out_1_width=1920: out_1_height=1080: out_1_rate=full:
out_2_width=1280: out_2_height=720:  out_2_rate=full:  out_3_width=848:  out_3_height=480:  out_3_rate=full:  out_4_width=640:  out_4_height=360:  out_4_rate=full:  out_5_width=288:  out_5_height=160:  out_5_rate=full   [vid1][vid2][vid3][vid4][vid5]; [1]volume=1[aud1]; [2]volume=1[aud2]; [3]volume=1[aud3]; [4]volume=1[aud4]; [5]volume=1[aud5].
Successfully parsed a group of options.
Parsing a group of options: input url rtmp://ingress.w1.invintus.com/srcEncoders/247airtestfeed.
Applying option c:v (codec name) with argument mpsoc_vcu_h264.
Successfully parsed a group of options.
Opening an input file: rtmp://ingress.w1.invintus.com/srcEncoders/247airtestfeed.
[NULL @ 0x562431bb9000] Opening 'rtmp://ingress.w1.invintus.com/srcEncoders/247airtestfeed' for reading
[rtmp @ 0x562431b87000] No default whitelist set
[tcp @ 0x562431af6000] No default whitelist set
[tcp @ 0x562431af6000] Original list of addresses:
[tcp @ 0x562431af6000] Address 34.222.84.145 port 1935
[tcp @ 0x562431af6000] Interleaved list of addresses:
[tcp @ 0x562431af6000] Address 34.222.84.145 port 1935
[tcp @ 0x562431af6000] Starting connection attempt to 34.222.84.145 port 1935
[tcp @ 0x562431af6000] Successfully connected to 34.222.84.145 port 1935
[rtmp @ 0x562431b87000] Handshaking...
[rtmp @ 0x562431b87000] Type answer 3
[rtmp @ 0x562431b87000] Server version 3.0.1.1
[rtmp @ 0x562431b87000] Proto = rtmp, path = /srcEncoders/247airtestfeed, app = srcEncoders, fname = 247airtestfeed
[rtmp @ 0x562431b87000] Window acknowledgement size = 2500000
[rtmp @ 0x562431b87000] Max sent, unacked = 2500000
[rtmp @ 0x562431b87000] New incoming chunk size = 4096
[rtmp @ 0x562431b87000] Creating stream...
[rtmp @ 0x562431b87000] Sending play command for '247airtestfeed'
[flv @ 0x562431bb9000] Format flv probed with size=2048 and score=100
[flv @ 0x562431bb9000] Before avformat_find_stream_info() pos: 13 bytes read:65536 seeks:0 nb_streams:0
[flv @ 0x562431bb9000] Unknown type onStatus
[NULL @ 0x562431bb3000] nal_unit_type: 7(SPS), nal_ref_idc: 3
[NULL @ 0x562431bb3000] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x562431bb3000] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x562431bb3000] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x562431bb3000] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x562431bb3000] Format yuv420p chosen by get_format().
[h264 @ 0x562431bb3000] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 0x562431bb3000] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x562431bb3000] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 2
    Last message repeated 4 times
[flv @ 0x562431bb9000] All info found
[flv @ 0x562431bb9000] rfps: 29.666667 0.013716
[flv @ 0x562431bb9000] rfps: 29.750000 0.007248
[flv @ 0x562431bb9000] rfps: 29.833333 0.002841
[flv @ 0x562431bb9000] rfps: 29.916667 0.000495
[flv @ 0x562431bb9000] rfps: 30.000000 0.000209
[flv @ 0x562431bb9000] rfps: 60.000000 0.000836
[flv @ 0x562431bb9000] rfps: 120.000000 0.003344
[flv @ 0x562431bb9000] rfps: 240.000000 0.013376
[flv @ 0x562431bb9000] rfps: 29.970030 0.000075
[flv @ 0x562431bb9000] rfps: 59.940060 0.000298
[flv @ 0x562431bb9000] After avformat_find_stream_info() pos: 561825 bytes read:561825 seeks:0 frames:105
Input #0, flv, from 'rtmp://ingress.w1.invintus.com/srcEncoders/247airtestfeed':
  Metadata:
    |RtmpSampleAccess: false
    length          : 1683456
    timescale       : 48000
    sampletype      : mp4a
    audiochannels   : 2
    videoframerate  : 30
    aacaot          : 2
    avclevel        : 42
    avcprofile      : 100
    frameWidth      : 1920
    frameHeight     : 1080
    displayWidth    : 1920
    displayHeight   : 1080
    moovposition    : 32
    encoder         : Lavf58.76.100
  Duration: 00:00:00.00, start: 0.000000, bitrate: N/A
  Stream #0:0, 1, 1/1000: Data: none, 0/1
  Stream #0:1, 42, 1/1000: Video: h264 (Constrained Baseline), 1 reference frame, yuv420p(tv, bt709, progressive, left), 1920x1080 (1920x1088), 0/1, 2000 kb/s, 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc
  Stream #0:2, 62, 1/1000: Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s
Successfully opened the file.
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'outputs' to value '5'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_1_width' to value '1920'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_1_height' to value '1080'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_1_rate' to value 'full'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_2_width' to value '1280'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_2_height' to value '720'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_2_rate' to value 'full'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_3_width' to value '848'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_3_height' to value '480'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_3_rate' to value 'full'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_4_width' to value '640'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_4_height' to value '360'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_4_rate' to value 'full'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_5_width' to value '288'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_5_height' to value '160'
[Parsed_multiscale_xma_0 @ 0x562431cc2000] Setting 'out_5_rate' to value 'full'
[Parsed_volume_1 @ 0x562431c06000] Setting 'volume' to value '1'
[Parsed_volume_2 @ 0x562431c0f000] Setting 'volume' to value '1'
[Parsed_volume_3 @ 0x562431d49000] Setting 'volume' to value '1'
[Parsed_volume_4 @ 0x562431ab5000] Setting 'volume' to value '1'
[Parsed_volume_5 @ 0x562431cd3000] Setting 'volume' to value '1'
Invalid file index 1 in filtergraph description multiscale_xma=outputs=5:  out_1_width=1920: out_1_height=1080: out_1_rate=full:  out_2_width=1280: out_2_height=720:  out_2_rate=full:  out_3_width=848:  out_3_height=480:  out_3_rate=full:  out_4_width=640:  out_4_height=360:  out_4_rate=full:  out_5_width=288:  out_5_height=160:  out_5_rate=full   [vid1][vid2][vid3][vid4][vid5]; [1]volume=1[aud1]; [2]volume=1[aud2]; [3]volume=1[aud3]; [4]volume=1[aud4]; [5]volume=1[aud5].
[AVIOContext @ 0x562431ba5000] Statistics: 561825 bytes read, 0 seeks
[rtmp @ 0x562431b87000] Deleting stream...```
NastoohX commented 1 year ago

Hi, Thank you for bringing this issue to our attention. The problem that you are observing is due to improper FFmpeg cli, namely, in filter definition of audio streams. For the specified stream, the correct command line is:

ffmpeg -c:v mpsoc_vcu_h264 -i rtmp://ingress.w1.invintus.com/srcEncoders/247airtestfeed \
-filter_complex "multiscale_xma=outputs=5: \
 out_1_width=1920: out_1_height=1080: out_1_rate=full: \
 out_2_width=1280: out_2_height=720:  out_2_rate=full: \
 out_3_width=848:  out_3_height=480:  out_3_rate=full: \
 out_4_width=640:  out_4_height=360:  out_4_rate=full: \
 out_5_width=288:  out_5_height=160:  out_5_rate=full  \
 [vid1][vid2][vid3][vid4][vid5]; [0:2]asplit=outputs=5[aud1][aud2][aud3][aud4][aud5]" \
-map "[vid1]" -b:v:0 2M   -minrate:v:0 2M   -maxrate:v:0 2M   -bufsize:v:0 4M   -c:v:0 mpsoc_vcu_h264 \
-map "[vid2]" -b:v:1 1M   -minrate:v:1 1M   -maxrate:v:1 1M   -bufsize:v:1 1M   -c:v:1 mpsoc_vcu_h264 \
-map "[vid3]" -b:v:2 750K -minrate:v:2 750K -maxrate:v:2 750K -bufsize:v:2 750K -c:v:2 mpsoc_vcu_h264 \
-map "[vid4]" -b:v:3 375K -minrate:v:3 375K -maxrate:v:3 375K -bufsize:v:3 375K -c:v:3 mpsoc_vcu_h264 \
-map "[vid5]" -b:v:4 250k -minrate:v:4 250k -maxrate:v:4 250k -bufsize:v:4 250k -c:v:4 mpsoc_vcu_h264 \
-map "[aud1]" -c:a:0 aac \
-map "[aud2]" -c:a:1 aac \
-map "[aud3]" -c:a:2 aac \
-map "[aud4]" -c:a:3 aac \
-map "[aud5]" -c:a:4 aac \
-var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2 v:3,a:3 v:4,a:4" \
-f hls \
-hls_wrap 5 \
-hls_time 6 \
-master_pl_name "test.m3u8" -hls_segment_filename  "${HLS_DIR}/test_%v-%d.ts" "${HLS_DIR}/test_%v.m3u8"

With the above, you should get an output similar to:

ffmpeg version n4.4.xlnx.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
...
Input #0, flv, from 'rtmp://ingress.w1.invintus.com/srcEncoders/247airtestfeed':
  Metadata:
    |RtmpSampleAccess: false
    length          : 1683456
    timescale       : 48000
    sampletype      : mp4a
    audiochannels   : 2
...
Stream mapping:
  Stream #0:1 (mpsoc_vcu_h264) -> multiscale_xma
  Stream #0:2 (aac) -> asplit
  multiscale_xma:output0 -> Stream #0:0 (mpsoc_vcu_h264)
  multiscale_xma:output1 -> Stream #0:1 (mpsoc_vcu_h264)
  multiscale_xma:output2 -> Stream #0:2 (mpsoc_vcu_h264)
  multiscale_xma:output3 -> Stream #0:3 (mpsoc_vcu_h264)
  multiscale_xma:output4 -> Stream #0:4 (mpsoc_vcu_h264)
  asplit:output0 -> Stream #0:5 (aac)
  asplit:output1 -> Stream #0:6 (aac)
  asplit:output2 -> Stream #0:7 (aac)
  asplit:output3 -> Stream #0:8 (aac)
  asplit:output4 -> Stream #0:9 (aac)
...
[hls @ 0x55e6ef938000] Opening '/tmp/hls/test_0-0.ts' for writing
[hls @ 0x55e6ef938000] Opening '/tmp/hls/test_0.m3u8.tmp' for writing
[hls @ 0x55e6ef938000] Opening '/tmp/hls/test_1-0.ts' for writing
[hls @ 0x55e6ef938000] Opening '/tmp/hls/test_1.m3u8.tmp' for writing
[hls @ 0x55e6ef938000] Opening '/tmp/hls/test_2-0.ts' for writing
...

Let's know if this solves your problem. Cheers,

scottf-tvw commented 1 year ago

Ok thank you! I knew it was probably just a cli syntax issue. I don't often have to break the audio down in this manor. I do appreciate the guidance.and I do now see the ffmpeg asplit directive spelled out in the sdk documentation: https://xilinx.github.io/video-sdk/v2.0/using_ffmpeg.html#mapping-audio-streams