Hardware accelerated 360 livestream

[avilabss]

I am trying to display a live video feed coming from Insta 360 X3 on my Orange Pi 5 Plus. I want to hardware accelerate the whole thing, but I am failing to make any progress. So far, only hardware-accelerated encoding works, but that doesn't seem to benefit much. So, I am trying to make hardware-accelerated decoding and filtering work. I would really appreciate it if you could check what I have done so far and suggest fixes that can make this work.

Code:

# Emulate Stream

import subprocess
import numpy as np
import threading
import select
import platform
import cv2

h264_path = './output/sample.raw'
frame_width, frame_height = 1440, 720
frame_size = frame_width * frame_height * 3  
timeout = 0.5

# Hardware Acceleration Configuration
enable_decoding_hwaccel = False
enable_filtering_hwaccel_device = False
enable_filtering_hwaccel = False
enable_encoding_hwaccel = True

# Order in which the FFmpeg command is constructed matters a lot
ffmpeg_cmd = [
    'ffmpeg',
    '-loglevel', 'error',                       # Suppress FFmpeg logs
    '-threads', '1',                            # Limit the number of threads used by FFmpeg       
    '-filter_complex_threads', '2',             # Limit the number of threads used by the filter_complex
]

# Hardware Acceleration Decoding according to the platform (Not working)
if enable_decoding_hwaccel:
    if platform.system() == 'Darwin':
        print("Enabling Hardware Accelerated Decoding for macOS")
        ffmpeg_cmd += [
            '-c:v', 'h264_videotoolbox',        
        ]

    elif platform.system() == 'Linux':
        print("Enabling Hardware Accelerated Decoding for Linux")
        ffmpeg_cmd += [
            '-hwaccel', 'rkmpp',
            # '-hwaccel_device', '/dev/dri/renderD128',
            # '-c:v', 'h264_rkmpp',
            '-hwaccel_output_format', 'drm_prime', # Output format for hardware acceleration
            '-afbc', 'rga'                         # ??? For Rockchip devices
        ]

    else:
        raise Exception("Unsupported platform")

# Experimental: Hardware Accelerated Filtering (Init works, but filter fails)
if enable_filtering_hwaccel_device:
    if platform.system() == 'Darwin':
        raise Exception("Hardware Accelerated Filtering is not configured for macOS")

    elif platform.system() == 'Linux':
        print("Enabling Hardware Accelerated Filtering for Linux")
        ffmpeg_cmd += [
            '-init_hw_device', 'rkmpp=hw',
            '-filter_hw_device', 'hw',
        ]

    else:
        raise Exception("Unsupported platform")

ffmpeg_cmd += [                                                
    '-i', 'pipe:0',

    '-filter_complex', 
]

if enable_filtering_hwaccel:
    if platform.system() == 'Darwin':
        raise Exception("Hardware Accelerated Filtering is not configured for macOS")

    elif platform.system() == 'Linux':
        print("Using Hardware Accelerated Filters for Linux")
        ffmpeg_cmd += [f'[0:v]scale_rkrga=w={frame_width}:h={frame_height},v360=dfisheye:e:ih_fov=193:iv_fov=193,hwmap,format=bgr24[out]']
else:
    print("Using Software Filters")
    ffmpeg_cmd += [f'[0:v]scale={frame_width}:{frame_height},v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]'] 

ffmpeg_cmd += [
   '-map', '[out]',                                                                        # Map the output of the filter_complex to the output file
    '-f', 'rawvideo',                                                                       # Raw video format
    '-an',                                                                                  # No audio output

    # H.264 encoding parameters (Working)
    '-flags', '+low_delay',
    '-b:v', '4M',
    '-minrate', '2M',
    '-maxrate', '5M', 
    # '-profile:v', 'high',    # Enabling this crashes FFmpeg
    '-level', '4.1',

    'pipe:1',        
]

# Hardware Accelerated Encoding according to the platform (Working)
if enable_encoding_hwaccel:
    if platform.system() == 'Darwin':
        print("Enabling Hardware Accelerated Encoding for macOS")
        ffmpeg_cmd += [
            '-c:v', 'h264_videotoolbox'
        ]

    elif platform.system() == 'Linux':
        print("Enabling Hardware Accelerated Encoding for Linux")
        ffmpeg_cmd += [
            '-hwaccel', 'rkmpp',
            '-hwaccel_device', '/dev/dri/renderD128',
            '-c:v', 'h264_rkmpp',
        ]

    else:
        raise Exception("Unsupported platform")

# Launch FFmpeg process
ffmpeg_proc = subprocess.Popen(ffmpeg_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# Write H.264 packets to FFmpeg process
def write_packets():
    with open(h264_path, 'rb') as fh:
        while True:
            chunk = fh.read(1 << 16)
            if not chunk:
                break

            ffmpeg_proc.stdin.write(chunk)
            ffmpeg_proc.stdin.flush()

w_thread = threading.Thread(target=write_packets)
w_thread.start()

# Display the video stream frame-by-frame
while True:
    rlist, _, _ = select.select([ffmpeg_proc.stdout], [], [], timeout)
    if not rlist:
        print("Timeout reached while waiting for frame data")
        break

    # Read raw video frame from stdout
    raw_frame = ffmpeg_proc.stdout.read(frame_size)

    if len(raw_frame) != frame_size:
        break

    # Convert the raw frame to a numpy array
    frame = np.frombuffer(raw_frame, np.uint8).reshape((frame_height, frame_width, 3))

    # Display the frame using OpenCV
    cv2.imshow('Preview', frame)
    cv2.waitKey(1)

# Cleanup
ffmpeg_proc.stdin.close()
ffmpeg_proc.stdout.close()
ffmpeg_proc.stderr.close()
ffmpeg_proc.wait()
cv2.destroyAllWindows()

[nyanmisaka]

The v360 filter is a CPU-based software filter, which cannot be accelerated by MPP and RGA. To accelerate the v360 filter please write an OpenCL/GPU-based filter v360_opencl.

Below is the corrected filter pipeline, only the scaling is accelerated by RGA.

scale_rkrga=w={frame_width}:h={frame_height}:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193

[avilabss]

understood, can you also help me understand why hardware accelerated decoding is not working? (currently when I enabled it, ffmpeg just crashes)

[nyanmisaka]

What is the error message? Please provide mediainfo of input video.

[avilabss]

When running manually:

Command

ffmpeg -loglevel error -threads 1 -filter_complex_threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -i './output/sample.raw' -filter_complex '[0:v]scale_rkrga=w=1440:h=720:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]' -map '[out]' -f rawvideo -an -flags +low_delay -b:v 4M -minrate 2M -maxrate 5M -level 4.1 './output/test.raw' -hwaccel rkmpp -hwaccel_device /dev/dri/renderD128 -c:v h264_rkmpp

Error

[Parsed_scale_rkrga_0 @ 0x5599aa8620] Input format 'nv12' with AFBC modifier is not supported by RGA2
[Parsed_scale_rkrga_0 @ 0x5599aa8620] Failed to submit frame on input: 0
[fc#0 @ 0x5599aa7580] Error while filtering: Cannot allocate memory
Failed to inject frame into filter network: Cannot allocate memory
Error while filtering: Cannot allocate memory
[out#0/rawvideo @ 0x5599b362f0] Nothing was written into output file, because at least one of its streams received no packets.

Running with python error:

Enabling Hardware Accelerated Decoding for Linux
Using Hardware Accelerated Filters for Linux
Enabling Hardware Accelerated Encoding for Linux
FFmpeg Command:  ['ffmpeg', '-loglevel', 'error', '-threads', '1', '-filter_complex_threads', '2', '-hwaccel', 'rkmpp', '-hwaccel_output_format', 'drm_prime', '-afbc', 'rga', '-i', 'pipe:0', '-filter_complex', '[0:v]scale_rkrga=w=1440:h=720:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]', '-map', '[out]', '-f', 'rawvideo', '-an', '-flags', '+low_delay', '-b:v', '4M', '-minrate', '2M', '-maxrate', '5M', '-level', '4.1', 'pipe:1', '-hwaccel', 'rkmpp', '-hwaccel_device', '/dev/dri/renderD128', '-c:v', 'h264_rkmpp']
Exception in thread Thread-1 (write_packets):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/root/insta360/emulate_stream.py", line 132, in write_packets
    ffmpeg_proc.stdin.write(chunk)
BrokenPipeError: [Errno 32] Broken pipe

[nyanmisaka]

-afbc rga is incompatible with yuv420p. Drop it.

[avilabss]

Done, now some processing happens for a short time and things error out again in Python

(insta360-py3.10) root@orangepi5plus:~/insta360# python3 emulate_stream.py 
Enabling Hardware Accelerated Decoding for Linux
Using Hardware Accelerated Filters for Linux
Enabling Hardware Accelerated Encoding for Linux
FFmpeg Command:  ['ffmpeg', '-loglevel', 'error', '-threads', '1', '-filter_complex_threads', '2', '-hwaccel', 'rkmpp', '-hwaccel_output_format', 'drm_prime', '-i', 'pipe:0', '-filter_complex', '[0:v]scale_rkrga=w=1440:h=720:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]', '-map', '[out]', '-f', 'rawvideo', '-an', '-flags', '+low_delay', '-b:v', '4M', '-minrate', '2M', '-maxrate', '5M', '-level', '4.1', 'pipe:1', '-hwaccel', 'rkmpp', '-hwaccel_device', '/dev/dri/renderD128', '-c:v', 'h264_rkmpp']
Timeout reached while waiting for frame data
Exception in thread Thread-1 (write_packets):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/root/insta360/emulate_stream.py", line 132, in write_packets
    ffmpeg_proc.stdin.write(chunk)
ValueError: write to closed file

When executed directly, it runs but fails to write anything

(insta360-py3.10) root@orangepi5plus:~/insta360# ffmpeg -threads 1 -filter_complex_threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -i './output/sample.raw' -filter_complex '[0:v]scale_rkrga=w=1440:h=720:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]' -map '[out]' -f rawvideo -an -flags +low_delay -b:v 4M -minrate 2M -maxrate 5M -level 4.1 -y './output/test.raw' -hwaccel rkmpp -hwaccel_device /dev/dri/renderD128 -c:v h264_rkmpp
ffmpeg version 9efe5bc Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Trailing option(s) found in the command: may be ignored.
Input #0, h264, from './output/sample.raw':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1440x720 [SAR 1:1 DAR 2:1], 25 fps, 59.94 tbr, 1200k tbn
Stream mapping:
  Stream #0:0 (h264_rkmpp) -> scale_rkrga:default
  format:default -> Stream #0:0 (rawvideo)
Press [q] to stop, [?] for help
rga_api version 1.9.3_[2]
[swscaler @ 0x558e19aad0] [swscaler @ 0x558e25def0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x558e19aad0] [swscaler @ 0x558e26ae90] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x558e19aad0] [swscaler @ 0x558e25def0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x558e19aad0] [swscaler @ 0x558e26ae90] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x558e19aad0] [swscaler @ 0x558e25def0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x558e19aad0] [swscaler @ 0x558e26ae90] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x558e19aad0] [swscaler @ 0x558e25def0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x558e19aad0] [swscaler @ 0x558e26ae90] No accelerated colorspace conversion found from yuv420p to bgr24.
[vost#0:0/rawvideo @ 0x558e114fd0] No filtered frames for output stream, trying to initialize anyway.
Output #0, rawvideo, to './output/test.raw':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24(progressive), 1440x720 [SAR 1:1 DAR 2:1], q=2-31, 745750 kb/s, 29.97 fps, 29.97 tbn
    Metadata:
      encoder         : Lavc60.31.102 rawvideo
[out#0/rawvideo @ 0x558e18d110] video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[out#0/rawvideo @ 0x558e18d110] Output file is empty, nothing was encoded(check -ss / -t / -frames parameters if used)
frame=    0 fps=0.0 q=0.0 Lsize=       0kB time=N/A bitrate=N/A dup=0 drop=659 speed=N/A    

[nyanmisaka]

ffmpeg -threads 1 -filter_complex_threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -i './output/sample.raw' -filter_complex '[0:v]scale_rkrga=w=1440:h=720:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]' -map '[out]' -f rawvideo -an -flags +low_delay -b:v 4M -minrate 2M -maxrate 5M -level 4.1 -y './output/test.raw' -hwaccel rkmpp -hwaccel_device /dev/dri/renderD128 -c:v h264_rkmpp

Do you want to encode to h264 or get rawvideo in bgr24 format?

[avilabss]

I think I want a raw video stream in bgr24 format so i can display it live using cv2.

insta360 x3 --live_video_packets--> orange_ip -> python -> ffmpeg (for applying filters) -> display_frames_using_cv2

[nyanmisaka]

ffmpeg -threads 1 -filter_complex_threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -i './output/sample.raw' -filter_complex '[0:v]scale_rkrga=w=1440:h=720:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]' -map '[out]' -f rawvideo -y './output/test.raw'

[avilabss]

Still no luck

Python

(insta360-py3.10) root@orangepi5plus:~/insta360# python3 ./emulate_stream.py 
Enabling Hardware Accelerated Decoding for Linux
Using Hardware Accelerated Filters for Linux
FFmpeg Command:  ['ffmpeg', '-loglevel', 'error', '-threads', '1', '-filter_complex_threads', '2', '-hwaccel', 'rkmpp', '-hwaccel_output_format', 'drm_prime', '-i', 'pipe:0', '-filter_complex', '[0:v]scale_rkrga=w=1440:h=720:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]', '-map', '[out]', '-f', 'rawvideo', 'pipe:1']
Timeout reached while waiting for frame data
Exception in thread Thread-1 (write_packets):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/root/insta360/./emulate_stream.py", line 84, in write_packets
    ffmpeg_proc.stdin.write(chunk)
ValueError: write to closed file

Direct

(insta360-py3.10) root@orangepi5plus:~/insta360# ffmpeg -threads 1 -filter_complex_threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -i './output/sample.raw' -filter_complex '[0:v]scale_rkrga=w=1440:h=720:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]' -map '[out]' -f rawvideo -y './output/test.raw'
ffmpeg version 9efe5bc Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Input #0, h264, from './output/sample.raw':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1440x720 [SAR 1:1 DAR 2:1], 25 fps, 59.94 tbr, 1200k tbn
Stream mapping:
  Stream #0:0 (h264_rkmpp) -> scale_rkrga:default
  format:default -> Stream #0:0 (rawvideo)
Press [q] to stop, [?] for help
rga_api version 1.9.3_[2]
[swscaler @ 0x55930ef8e0] [swscaler @ 0x55931b2d00] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55930ef8e0] [swscaler @ 0x55931bfca0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55930ef8e0] [swscaler @ 0x55931b2d00] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55930ef8e0] [swscaler @ 0x55931bfca0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55930ef8e0] [swscaler @ 0x55931b2d00] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55930ef8e0] [swscaler @ 0x55931bfca0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55930ef8e0] [swscaler @ 0x55931b2d00] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55930ef8e0] [swscaler @ 0x55931bfca0] No accelerated colorspace conversion found from yuv420p to bgr24.
[vost#0:0/rawvideo @ 0x55930e24d0] No filtered frames for output stream, trying to initialize anyway.
Output #0, rawvideo, to './output/test.raw':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24(progressive), 1440x720 [SAR 1:1 DAR 2:1], q=2-31, 745750 kb/s, 29.97 fps, 29.97 tbn
    Metadata:
      encoder         : Lavc60.31.102 rawvideo
[out#0/rawvideo @ 0x55930e1c00] video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[out#0/rawvideo @ 0x55930e1c00] Output file is empty, nothing was encoded(check -ss / -t / -frames parameters if used)
frame=    0 fps=0.0 q=0.0 Lsize=       0kB time=N/A bitrate=N/A dup=0 drop=659 speed=N/A 

[nyanmisaka]

Try this ffmpeg bin instead. https://github.com/MarcA711/Rockchip-FFmpeg-Builds/releases/tag/6.1-4

[avilabss]

It's pretty much the same as before. Ps. I really appreciate you taking the time to help me cause I'm very clueless about this right now.

Python

(insta360-py3.10) root@orangepi5plus:~/insta360# python3 emulate_stream.py 
Enabling Hardware Accelerated Decoding for Linux
Using Hardware Accelerated Filters for Linux
FFmpeg Command:  ['./ffmpeg', '-loglevel', 'error', '-threads', '1', '-filter_complex_threads', '2', '-hwaccel', 'rkmpp', '-hwaccel_output_format', 'drm_prime', '-i', 'pipe:0', '-filter_complex', '[0:v]scale_rkrga=w=1440:h=720:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]', '-map', '[out]', '-f', 'rawvideo', 'pipe:1']
Timeout reached while waiting for frame data
Exception in thread Thread-1 (write_packets):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/root/insta360/emulate_stream.py", line 84, in write_packets
    ffmpeg_proc.stdin.write(chunk)
ValueError: write to closed file

Direct

(insta360-py3.10) root@orangepi5plus:~/insta360# ./ffmpeg -threads 1 -filter_complex_threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -i './output/sample.raw' -filter_complex '[0:v]scale_rkrga=w=1440:h=720:format=yuv420p,hwmap,format=yuv420p,v360=dfisheye:e:ih_fov=193:iv_fov=193,format=bgr24[out]' -map '[out]' -f rawvideo -y './output/test.raw'
ffmpeg version 9efe5bcff0-20240602 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.2.0 (crosstool-NG 1.26.0.65_ecc5e41)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=aarch64-ffbuild-linux-gnu- --arch=aarch64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-openssl --enable-fontconfig --enable-libharfbuzz --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --disable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libaribcaption --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --disable-libvpx --enable-libwebp --enable-lv2 --disable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-rkmpp --enable-rkrga --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --disable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-libs='-ldl -lstdc++ -lstdc++ -lgomp' --extra-ldflags=-pthread --extra-ldexeflags=-pie --cc=aarch64-ffbuild-linux-gnu-gcc --cxx=aarch64-ffbuild-linux-gnu-g++ --ar=aarch64-ffbuild-linux-gnu-gcc-ar --ranlib=aarch64-ffbuild-linux-gnu-gcc-ranlib --nm=aarch64-ffbuild-linux-gnu-gcc-nm --extra-version=20240602
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Input #0, h264, from './output/sample.raw':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1440x720 [SAR 1:1 DAR 2:1], 25 fps, 59.94 tbr, 1200k tbn
Stream mapping:
  Stream #0:0 (h264_rkmpp) -> scale_rkrga:default
  format:default -> Stream #0:0 (rawvideo)
Press [q] to stop, [?] for help
rga_api version 1.10.0_[8]
[swscaler @ 0x55b903f800] [swscaler @ 0x55b904c7a0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55b903f800] [swscaler @ 0x55b9059740] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55b903f800] [swscaler @ 0x55b904c7a0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55b903f800] [swscaler @ 0x55b9059740] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55b903f800] [swscaler @ 0x55b904c7a0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55b903f800] [swscaler @ 0x55b9059740] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55b903f800] [swscaler @ 0x55b904c7a0] No accelerated colorspace conversion found from yuv420p to bgr24.
[swscaler @ 0x55b903f800] [swscaler @ 0x55b9059740] No accelerated colorspace conversion found from yuv420p to bgr24.
[vost#0:0/rawvideo @ 0x55b8f2f010] No filtered frames for output stream, trying to initialize anyway.
Output #0, rawvideo, to './output/test.raw':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24(progressive), 1440x720 [SAR 1:1 DAR 2:1], q=2-31, 745750 kb/s, 29.97 fps, 29.97 tbn
    Metadata:
      encoder         : Lavc60.31.102 rawvideo
[out#0/rawvideo @ 0x55b8f2e7b0] video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[out#0/rawvideo @ 0x55b8f2e7b0] Output file is empty, nothing was encoded(check -ss / -t / -frames parameters if used)
frame=    0 fps=0.0 q=0.0 Lsize=       0kB time=N/A bitrate=N/A dup=0 drop=659 speed=N/A 

[nyanmisaka]

Set framerate for the input file. ... -r 25 -i './output/sample.raw' ...

[avilabss]

Woohoo! I'm finally getting output with hardware acceleration enabled! The only thing I'm noticing is that the video is a bit too slow. The camera was originally set to give output at 30FPS, but I see it's slower now, almost as if a small slow-motion effect has been applied, but the output is very clean. Maybe I need to increase thread counts for FFmpeg or complex filters?

[nyanmisaka]

Slowed down video is caused by inaccurate timebase/framerate etc. 25 fps, 59.94 tbr, 1200k tbn Search for ffmpeg tbc tbn tbr to find out how to set it up correctly.

[avilabss]

I see. I'm looking into it next! Thank you for all the guidance @nyanmisaka 🙌