"Extention failed" occurs when using multiple multiscale_xma

[Keukhan]

Hello.

I am developing a program using the FFmpeg library on an AWS vt1.3xlarge instance. I have created a structure where frames decoded by the Decoder (mpsoc_vcu_h264) are passed to multiple Scalers. to be specific, Cloning the AVFrame using the av_frame_clone function and passing it to each of the scaler. However, I intermittently encounter the "Extension failed" error. The error occurs when the frame is retrieved from the filter using the av_buffersink_get_frame function. What could be the problem?

Although the Xilinx guide suggests using multiscale_xma to have multiple outputs from a single input, it is not feasible in my program structure. Therefore, I am creating multiple multiscale_xma filters for each resolution.

Help me. I need support.

Transcoding structure

Decoder(mpsoc_vcu_h264) → Scaler(multiscale_xma) → Encoder(mpsoc_vpu_h264)
                        → Scaler(multiscale_xma) → Encoder(mpsoc_vpu_h264)
                        → Scaler(multiscale_xma) → Encoder(mpsoc_vpu_h264)

Scaler options

1) multiscale_xma=outputs=1:out_1_width=1920:out_1_height=1080:out_1_rate=full
2) multiscale_xma=outputs=1:out_1_width=1280:out_1_height=720:out_1_rate=full
3) multiscale_xma=outputs=1:out_1_width=960:out_1_height=540:out_1_rate=full

Error message

[prep_and_write_input_buffer:1108] ERROR : Extention failed
XRT build version: 2.11.722
Build hash: d1ecc641da7b8805e778926e3354e8dd3fdaa6a8
Build date: 2023-01-25 23:37:46
Git branch: 2021.1
PID: 16824
UID: 1000
[Wed Jun 28 06:06:57 2023 GMT]
HOST: ip-172-31-24-53.ap-northeast-1.compute.internal
EXE: /home/ec2-user/OvenMediaEngine/src/bin/DEBUG/OvenMediaEngine
[XMA] ERROR: OvenMediaEngine xma-multiscaler [prep_and_write_input_buffer:1108] ERROR : Extention failed
[06-28 06:06:57.185] E [Rescaler:16880] FFmpeg | third_parties.cpp:111  | scaler error: multiscaler filter_frame failed: ffmpeg pid 16824 on device index =  0 cu index = 0

$ xbutil examine
System Configuration
  OS Name              : Linux
  Release              : 5.10.179-168.710.amzn2.x86_64
  Version              : #1 SMP Mon May 22 23:10:22 UTC 2023
  Machine              : x86_64
  CPU Cores            : 12
  Memory               : 22514 MB
  Distribution         : Amazon Linux 2
  GLIBC                : 2.26
  Model                : vt1.3xlarge

XRT
  Version              : 2.11.722
  Branch               : 2021.1
  Hash                 : d1ecc641da7b8805e778926e3354e8dd3fdaa6a8
  Hash Date            : 2023-01-25 23:37:46
  XOCL                 : 2.11.722, d1ecc641da7b8805e778926e3354e8dd3fdaa6a8
  XCLMGMT              : unknown, unknown

Devices present
  [0000:00:1f.0] : xilinx_u30_gen3x4_base_2 
  [0000:00:1e.0] : xilinx_u30_gen3x4_base_2 

[NastoohX]

Hi, Thank you for bringing this issue to our attention. It would helpful if you could shed some light on the following: 1- Multiple multiscale_xma, in a single pipeline, in not a typical use case, and as such, understating the reasoning behind it can help us in providing useful feedback. 2- Can you provide a ffmpeg cli that exhibits the same issue?

Meanwhile, our engineering team will investigate the usage of multiple multiscale_xma, in a single pipeline. Cheers,

[Keukhan]

Thank you for responding.

I haven't used the FFmpeg cli, but I'll try to create a reproducible cli. If not, I'll make a simple code.

👍

[Keukhan]

@NastoohX

The transcoding pipeline structure that uses the multiscale_xma framework can be created with the following command in FFmpeg. However, I didn't get an "Extention Failed" error when using this command on multiple scalers.

 ffmpeg  -c:v mpsoc_vcu_h264 -i input.mp4 \
  -map 0:v:0 -map 0:v:0 -map 0:v:0  -map 0:v:0 \
  -ss 0 -t 30 -c:v mpsoc_vcu_h264 \
  -filter:v:0 multiscale_xma=outputs=1:out_1_width=640:out_1_height=360:out_1_rate=full -maxrate:v:0 600k  \
  -filter:v:1 multiscale_xma=outputs=1:out_1_width=1280:out_1_height=720:out_1_rate=full -maxrate:v:1 1500k \
  -filter:v:2 multiscale_xma=outputs=1:out_1_width=1920:out_1_height=1080:out_1_rate=full -maxrate:v:2 3000k \
  -filter:v:3 multiscale_xma=outputs=1:out_1_width=1920:out_1_height=1080:out_1_rate=full -maxrate:v:3 5000k \
  -y "output.mp4" -loglevel debug

After thinking about it, the difference between this command and my program is that the scaler is running in each thread. The structure of my program is like the attached image.
When av_buffersink_get_frame function is called in this structure, multiscale_xma_filter_frame function is called, and Extention Failed error occurs intermittently. so, when av_buffersink_get_frame was called, I synchronized it with a mutex, but no error occurred. Perhaps, there seems to be a non-thread safe part in the multiscale_xma_filter_frame function. ㅡ

I hope this helps in resolving this issue.

Thanks.

[NastoohX]

Hi, Great to hear that you found a solution to your problem. The thread issue that you have noted, is a possibility and will be investigated, on our side. (Admittedly, I am not sure if multiscale_xma filter is meant to be thread safe; however.) BTW, Have you experimented with adding split filter, prior to multiscale_xma filter? Cheers,

[Keukhan]

Thank you for your reply. As you guided, I will experiment by adding a split filter before multiscale_xma.

[Keukhan]

@NastoohX I experimented by adding a split filter before multiscale_xma, but the same "Extention failed" error occurs.

When I get extension failed error, I am using method to recreate filter.

I'm waiting for good news. :)

[NastoohX]

Hi, It was an educated guess... However, given that you currently have a workaround, it doesn't seem to be detrimental to your use case. At this point, there are no plans to make any further changes to our code base. As such, I'll be closing this ticket. However, if we can provide further assistance, on a different issue, feel free to open a new ticket. Cheers,