[BUG] H264Encoder: enable_sps_framerate unsupported on Raspberry Pi 5

[agrear]

I am using a Raspberry Pi 5 running Bookworm 64bit (Picamera2 v0.3.22-2) to stream a Raspberry Pi High Quality Camera encoded to H.264 over RTSP using MediaMTX.

Since the RPi 5 lacks hardware encoding, passing the enable_sps_framerate parameter to H264Encoder results in an error due to (what I assume) H264Encoder being an alias for LibavH264Encoder which does not support the parameter.

Trying to open the H.264 encoded stream on an RPi 3B+ with ffplay results in the following output:

ffplay version 5.1.6-0+deb12u1+rpt1 Copyright (c) 2003-2024 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr --extra-version=0+deb12u1+rpt1 --toolchain=hardened --incdir=/usr/include/aarch64-linux-gnu --enable-gpl --disable-stripping --disable-mmal --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sand --enable-sdl2 --disable-sndio --enable-libjxl --enable-neon --enable-v4l2-request --enable-libudev --enable-epoxy --libdir=/usr/lib/aarch64-linux-gnu --arch=arm64 --enable-pocketsphinx --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-vout-drm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[rtsp @ 0x7f78000c20] max delay reached. need to consume packet/0
[rtsp @ 0x7f78000c20] RTP: missed 36 packets
[h264 @ 0x7f78004eb0] error while decoding MB 46 19, bytestream -9
[h264 @ 0x7f78004eb0] concealing 2083 DC, 2083 AC, 2083 MV errors in I frame
Input #0, rtsp, from 'rtsp://1.1.1.1:8000/main':
  Metadata:
    title           : No Name
  Duration: N/A, start: 0.133144, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1280x720, 30 fps, 29.97 tbr, 90k tbn
[rtsp @ 0x7f78000c20] max delay reached. need to consume packet/0
[rtsp @ 0x7f78000c20] RTP: missed 35 packets
[h264 @ 0x7f78033ad0] error while decoding MB 46 19, bytestream -9
[h264 @ 0x7f78033ad0] concealing 2083 DC, 2083 AC, 2083 MV errors in I frame

Opening the stream with VLC on a Windows 10 PC does work, however I noticed some inconsistencies regarding frame rate / duration of recorded video.

As a workaround I am also encoding the stream to MJPEG which does play on the RPi 3B+ but only if it is 1280x720 pixels. Needless to say this doesn't utilize the Pi's hardware decoder and is quite taxing on the CPU.

This is a downsized version of the script I'm using:

#!/usr/bin/python3

from picamera2 import Picamera2
from picamera2.encoders import H264Encoder, JpegEncoder, Quality
from picamera2.outputs import FfmpegOutput

picam2 = Picamera2()

config = picam2.create_video_configuration(
    main={'size': (1920, 1080), 'format': 'YUV420'},
    lores={'size': (1280, 720), 'format': 'YUV420'},
    sensor={'output_size': (2028, 1080), 'bit_depth': 12},
    controls={'FrameDurationLimits': (33333, 33333)})  # 30 FPS

picam2.align_configuration(config)
picam2.configure(config)

output_main = FfmpegOutput("-f rtsp -rtsp_transport udp -bsf:v dump_extra rtsp://localhost:8000/main", audio=False)

output_lores = FfmpegOutput("-f rtsp -rtsp_transport udp -bsf:v dump_extra rtsp://localhost:8000/lores", audio=False)

# Encoder settings
encoder_main = H264Encoder(
    qp=30,
    repeat=True,
    iperiod=30,
    #enable_sps_framerate=True,  # <- Unsupported on RPi 5
    profile='main',
    framerate=30)

encoder_lores = JpegEncoder()
encoder_lores = frame_skip_count = 2

try:
    picam2.start_recording(encoder_main, output_main)
    picam2.start_recording(encoder_lores, output_lores, quality=Quality.MEDIUM)

    while True:
        time.sleep(5)
except:
    picam2.stop_recording()

Would it be possible to get SPS working on RPi 5? Or is there any other way to encode the required information, using FFmpeg for instance?

[davidplowman]

Hi, thanks for the question. I'm afraid I don't really know the answer, but maybe there are some things to look into.

Firstly, how do you know that the lack of enable_sps_framerate is the problem? I did a brief search for this error message, but was left rather uncertain as to the root cause. I don't think I saw anything specifically relating this to the presence of a framerate in the SPS parameters - have you maybe found any links that suggest this?

I had a look in the libav AVCodecContext structure to see if it has any support for something named enable_sps_framerate. There wasn't, but there is something called just framerate, so if you were up for some experimentation, we could perhaps give this a try.

Check out the latest Picamera2. You can set your PYTHONPATH environment variable to point to the folder where you've put it, to make sure that you're using this version. Go to this line and add

        self._stream.codec_context.framerate = Fraction(30, 1)

(substitute your own framerate if it's not 30fps.)

I tried this and it did seem to change the SPS timing headers, so let's see if it helps with your problem.