FFMPEG keeps crashing randomly on VT1 instances

[moerradi]

Command

ffmpeg -i  rtmp://127.0.0.1:1935/transcode/$1 \
-filter_complex "multiscale_xma=outputs=4: \
out_1_width=1280: out_1_height=720:  out_1_rate=full: \
out_2_width=848:  out_2_height=480:  out_2_rate=half: \
out_3_width=640:  out_3_height=360:  out_3_rate=half: \
out_4_width=288:  out_4_height=160:  out_4_rate=half  \
[a][b][c][d]; [a]split[aa][ab]; [ab]fps=30[abb]" \
-map "[aa]"  -b:v 4M    -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 7 -hls_flags delete_segments -y /hls/$1_720p60.m3u8 \
-map "[abb]" -b:v 3M    -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 7 -hls_flags delete_segments -y /hls/$1_720p30.m3u8 \
-map "[b]"   -b:v 2500K -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 7 -hls_flags delete_segments -y /hls/$1_480p30.m3u8 \
-map "[c]"   -b:v 1250K -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 7 -hls_flags delete_segments -y /hls/$1_360p30.m3u8 \
-map "[d]"   -b:v 625K  -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 7 -hls_flags delete_segments -y /hls/$1_288p30.m3u8

Output

PATH              : /opt/xilinx/xrt/bin:/home/ec2-user/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/bin/remote-cli:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin
LD_LIBRARY_PATH   : /opt/xilinx/xrt/lib:
PYTHONPATH        : /opt/xilinx/xrt/python:
XILINX_XRM      : /opt/xilinx/xrm
PATH            : /opt/xilinx/xrm/bin:/opt/xilinx/xrt/bin:/home/ec2-user/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/bin/remote-cli:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin
LD_LIBRARY_PATH : /opt/xilinx/xrm/lib:/opt/xilinx/xrt/lib:
ffmpeg version 4.1.9 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7 (GCC)
  configuration: --prefix=/opt/xilinx/ffmpeg --datadir=/opt/xilinx/ffmpeg/etc --enable-x86asm --enable-libxma2api --disable-doc --enable-libxvbm --enable-libxrm --enable-libfreetype --enable-libfontconfig --extra-cflags=-I/opt/xilinx/xrt/include/xma2 --extra-ldflags=-L/opt/xilinx/xrt/lib --extra-libs=-lxma2api --extra-libs=-lxrt_core --extra-libs=-lxrt_coreutil --extra-libs=-lpthread --extra-libs=-ldl --disable-static --enable-shared
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100

<<<<<<<==  FFmpeg xrm ===>>>>>>>>
Input #0, flv, from 'rtmp://127.0.0.1:1935/transcode/hello':
  Metadata:
    Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
    displayWidth    : 1920
    displayHeight   : 1080
    fps             : 60
    profile         : 
    level           : 
  Duration: 00:00:00.00, start: 348.160000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp, 163 kb/s
    Stream #0:1: Video: h264 (Constrained Baseline), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 2560 kb/s, 60 fps, 60 tbr, 1k tbn, 120 tbc
------------------------------------------------------------

   xclbin_name :  /opt/xilinx/xcdr/xclbins/transcode.xclbin
   device_id   :  0 
------------------------------------------------------------

Stream mapping:
  Stream #0:1 (h264) -> multiscale_xma
  split:output0 -> Stream #0:0 (mpsoc_vcu_h264)
  fps -> Stream #1:0 (mpsoc_vcu_h264)
  multiscale_xma:output1 -> Stream #2:0 (mpsoc_vcu_h264)
  multiscale_xma:output2 -> Stream #3:0 (mpsoc_vcu_h264)
  multiscale_xma:output3 -> Stream #4:0 (mpsoc_vcu_h264)
Press [q] to stop, [?] for help
[mpsoc_vcu_h264 @ 0x2470a40] Custom Rate Control Mode is Disabled
XRT build version: 2.10.50
Build hash: 244440c52fdc373a80a17f5b54d751648f69b9ec
Build date: 2021-08-27 23:28:22
Git branch: u30
PID: 3069
UID: 1000
[Tue May 24 16:39:51 2022 GMT]
HOST: ip-172-31-7-108.eu-west-1.compute.internal
EXE: /opt/xilinx/ffmpeg/bin/ffmpeg
[XMA] ERROR: ffmpeg xmautils Session id: 3, type: encoder, Unexpected ERT_CMD_STATE. state=SKCRASHED
terminate called after throwing an instance of 'std::runtime_error'
  what():  XMA FATAL: Session id: 3, type: encoder; Unexpected ERT_CMD_STATE. state=SKCRASHED```

[NastoohX]

Hi moerradi,

Thank you for bringing this issue to our attention. Could you, kindly, provide more details regarding: 1- OS type, e.g., Ubuntu, Amazon Linux, etc. 2- VT1 Instance type 3- Confirm if this crash is specific to a particular input 4- Confirm that the same behaviour is observed if the input rtmp file is ingested directly, i.e., -f flv -i $1. If so, would you be able to provide a small sample of this file?

Cheers,

[moerradi]

I am using a VT1.3xlarge instance, using the Xilinx Video SDK AMI with ECS support for VT1 Instances (Amazon Linux 2). I was using the scrips in a docker Environment, then I switched to using it directly in the instance and it gave the same output, I created a new instance tried to run the same script and the problem stopped occurring and the script worked normally, I think it was some kind of driver/temporary hardware related problem. However here is the script I was using.

#!/bin/bash
source /opt/xilinx/xrt/setup.sh
source /opt/xilinx/xrm/setup.sh
export LD_LIBRARY_PATH=/opt/xilinx/ffmpeg/lib:$LD_LIBRARY_PATH
export PATH=/opt/xilinx/ffmpeg/bin:/opt/xilinx/xcdr/bin:/opt/xilinx/launcher/bin:/opt/xilinx/jobSlotReservation/bin:$PATH

#check this for refernce
#https://docs.peer5.com/guides/production-ready-hls-vod/#how-to-choose-the-right-bitrate
on_die ()
{
    # kill all children
    pkill -KILL -P $$
}

trap 'on_die' TERM

mkdir -p /hls/$1/1080
mkdir -p /hls/$1/720p60
mkdir -p /hls/$1/720
mkdir -p /hls/$1/480
mkdir -p /hls/$1/360
mkdir -p /hls/$1/160

# cat /scripts/exmple.m3u8 | sed "s/XXXX/$1/g" > /hls/$1/index.m3u8``

ffmpeg -i  rtmp://localhost:1935/transcode/$1 \
-filter_complex "multiscale_xma=outputs=6: \
out_1_width=1920: out_1_height=1080: out_1_rate=full: \
out_2_width=1280: out_2_height=720:  out_2_rate=full: \
out_3_width=848:  out_3_height=480:  out_3_rate=half: \
out_4_width=640:  out_4_height=360:  out_4_rate=half: \
out_5_width=288:  out_5_height=160:  out_5_rate=half  \
[t][a][b][c][d]; [a]split[ab][ba]; [ab]fps=30[bb]; asplit=outputs=6[audt][aud2][aud1][aud3][aud4][aud5]" \
-map "[t]"  -b:v 7M    -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 10 -hls_flags delete_segments -map "[audt]" -c:a aac -y /hls/$1/1080/index.m3u8  \
-map "[ba]"  -b:v 4M    -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 10 -hls_flags delete_segments -map "[aud1]" -c:a aac -y /hls/$1/720p60/index.m3u8  \
-map "[bb]" -b:v 3M    -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 10 -hls_flags delete_segments -map "[aud2]" -c:a aac -y /hls/$1/720/index.m3u8  \
-map "[b]"   -b:v 2500K -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 10 -hls_flags delete_segments -map "[aud3]" -c:a aac -y /hls/$1/480/index.m3u8  \
-map "[c]"   -b:v 1250K -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 10 -hls_flags delete_segments -map "[aud4]" -c:a aac -y /hls/$1/360/index.m3u8  \
-map "[d]"   -b:v 625K  -c:v mpsoc_vcu_h264 -f hls -hls_time 2 -hls_list_size 10 -hls_flags delete_segments -map "[aud5]" -c:a aac -y /hls/$1/160/index.m3u8   2>&1 | tee /hls/test.log

wait

[NastoohX]

Great news. Closing this issue.