fps Performance with camera-streamer over libcamera

[goamberg]

Hi That´s a really cool project, playing around with it for a couple of evenings now...

But I do have a question on running the Arducam AR0234 with your camera-streamer through libcamera on 3B+ and Zero 2W

I am using this for normal udp tests: libcamera-vid -t 0 --width 1920 --height 1080 --framerate 50 --level 4.2 --bitrate 8000000 --denoise off -n -o udp://10.0.0.20:9998

It is running out of specs for a raspberry, but for the 2-3 minutes I need it and overclocked, it runs better than expected. When I record the stream and put it in a Adobe Premiere timeline with 50fps I can visually confirm it that it makes 50fps over UDP.

Now when I try to mimic the same with camera-streamer with libcamera and webrtc I never get the same fps out of it. It basically runs at 25fps (sometimes it goes even a little below that). Same with rtsp and and mp4.

Here is an example of my libcamera_camera.sh I am using for testing at the moment:

#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
cd "$SCRIPT_DIR/.."

set -xeo pipefail
make -j$(nproc)
$GDB ./camera-streamer \
  -camera-type=libcamera \
  -camera-option=noise_reduction_mode=4 \
  -camera-width=1920 \
  -camera-height=1080 \
  -camera-fps=50 \
  -camera-video.height=1080 \
  -camera-video.options=h264_level=13 \
  -camera-video.options=videobitrate=8000000 \
  "$@"

I also tested through my available noise reduction modes [0..4] in the options to switch it off. But it doesn't seem to do much (except 0 is cdn_hq I think)

Should it be possible to get around the same performance out of camera-streamer over webrtc like it would with a libcamera UDP stream? If so, my best guess is that I cannot switch off denoise completely (spatial and colour noise) like I can with --denoise off ? Any hint would be greatly appreciated

That is the camera I am testing: https://docs.arducam.com/Raspberry-Pi-Camera/Pivariety-Camera/AR0234/ And here is the camera dump from tools: Dump_camera.txt

Goam

ps. do you have any "buymeacoffee" link? ;-)

[ayufan]

With default settings the most I could achieve was 37fps on 1080p. The 50 should be doable, but did not test it, will try with my arducam tomorrow.

[ayufan]

@goamberg

I tested the libcamera-vid on my RPI4B and I'm unable to achieve 50FPS encoding H264.

The max I could:

libcamera-vid -t 0 --width 1920 --height 1080 --framerate 50 --level 4.2 --bitrate 8000000 --denoise off -n -o /run/video.h264 --save-pts /run/video.pts
wget https://raw.githubusercontent.com/Hermann-SW/userland/master/tools/ptsanalyze
bash ptsanalyze  /run/video.pts

This seems to correlate with what camera-streamer can achieve. There's some randomness at frame rate since it is at the limit of HW encoder.

camera-streamer --camera-type=libcamera --camera-path=i2c0mux --camera-width=1920 --camera-height=1080 --camera-fps=60 --log-stats --camera-nbufs=4 -camera-video.options=h264_level=13 -camera-video.options=videobitrate=8000000

Statistics: [  CAMERA 46 FPS/ 0 D/ 31ms/ 46ms/S/Q0:O0:C2] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 48 FPS/ 0 D/ 74ms/ 33ms/S/Q0:O2:C1]
Statistics: [  CAMERA 42 FPS/ 0 D/ 24ms/ 47ms/S/Q0:O0:C2] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 42 FPS/ 0 D/ 49ms/ 22ms/S/Q0:O2:C1]
Statistics: [  CAMERA 43 FPS/ 0 D/ 33ms/ 53ms/S/Q0:O0:C2] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 44 FPS/ 0 D/ 64ms/ 30ms/S/Q0:O1:C0]
Statistics: [  CAMERA 42 FPS/ 0 D/ 27ms/ 36ms/S/Q0:O0:C1] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 39 FPS/ 0 D/ 73ms/ 38ms/S/Q1:O2:C2]
Statistics: [  CAMERA 39 FPS/ 0 D/ 27ms/ 37ms/S/Q0:O0:C1] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 39 FPS/ 0 D/ 76ms/ 42ms/S/Q1:O2:C2]

[goamberg]

Thanks for looking into this. It was a lot of testing to get it to 50fps but it can work for some minutes at my end. With proper cooling indefinetly. Very important is to switch off the denoising and some overclocking.

...
> after skip frame indices (middle column)
0 frame skips (0%)
average framerate 50fps

Admittedly frame times are a little wide spread for my flavor, but 0 skipped frames I consider it working. Also with 52fps testing I already get high skipped frames, with 60fps it is as high as 30-40%.

What I find interesting, that here the camera-streamer directly goes to 25fps. In your statistics I can see that there are camera readouts of ~43fps and so on. Mine go directly to 25.

camera-streamer --camera-type=libcamera --camera-width=1920 --camera-height=1080 --camera-fps=50 --log-stats --camera-nbufs=4 -camera-option=noise_reduction_mode=[0..4] -camera-video.options=h264_level=13 -camera-video.options=videobitrate=8000000

Statistics: [  CAMERA 25 FPS/ 0 D/ 28ms/ 79ms/S/Q0:O0:C2] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 25 FPS/ 0 D/ 56ms/ 26ms/S/Q0:O1:C0]
Statistics: [  CAMERA 25 FPS/ 0 D/ 32ms/ 84ms/S/Q0:O0:C2] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 25 FPS/ 0 D/ 60ms/ 26ms/S/Q0:O1:C0]
Statistics: [  CAMERA 25 FPS/ 0 D/ 28ms/ 80ms/S/Q0:O0:C2] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 25 FPS/ 0 D/ 55ms/ 26ms/S/Q0:O1:C0]
Statistics: [  CAMERA 25 FPS/ 0 D/ 28ms/ 80ms/S/Q0:O0:C2] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 25 FPS/ 0 D/ 54ms/ 26ms/S/Q0:O1:C0]
Statistics: [  CAMERA 25 FPS/ 0 D/ 29ms/ 50ms/S/Q0:O0:C2] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 25 FPS/ 0 D/ 55ms/ 25ms/S/Q0:O1:C0]

In theory, is there any reason why I would not be able to translate the 50fps over to camera-streamer from my direct libcamera-vid results?

[ayufan]

Sure. I do not overclock, so this is why it is reaching 43FPS, not 50FPS.

[ayufan]

@goamberg What version of camera-streamer you run? It seems that h264 encoder is under-run.

[goamberg]

I just install/compile from the master branch

[ayufan]

But what version, what is git sha?

[goamberg]

I had to google how to look this up ;-) Is this what you are asking for?

commit bbb1b32e08c825708418ecb7bd13e96fbef9a78a (HEAD -> master, origin/master, origin/HEAD)
Author: Kamil Trzcinski <ayufan@ayufan.eu>
Date:   Mon Mar 6 10:08:04 2023 +0100

    Revert "debug: add more logs for VIDIOC_QBUF"

    This reverts commit 5a3085a2f63c7ede601841fd42c602e697ad4662.

diff --git a/device/v4l2/buffer.c b/device/v4l2/buffer.c
index 5de5efa..614d997 100644
--- a/device/v4l2/buffer.c
+++ b/device/v4l2/buffer.c
@@ -119,13 +119,7 @@ int v4l2_buffer_enqueue(buffer_t *buf, const char *who)
   v4l2_buf.timestamp.tv_sec = buf->captured_time_us / (1000LL * 1000LL);
   v4l2_buf.timestamp.tv_usec = buf->captured_time_us % (1000LL * 1000LL);

-  ERR_IOCTL(
-    buf, buf->buf_list->v4l2->dev_fd, VIDIOC_QBUF, &v4l2_buf,
-    "Can't queue buffer (type=%d, index=%d, keyframe=%d, mmap=%d, mplanes=%d, bytesused=%d, dmafd=%d).",
-    buf->buf_list->v4l2->type, buf->index, buf->flags.is_keyframe,
-    buf->buf_list->do_mmap, buf->buf_list->v4l2->do_mplanes, v4l2_buf.bytesused,
-    buf->dma_source ? buf->dma_source->dma_fd : -1
-  );
+  ERR_IOCTL(buf, buf->buf_list->v4l2->dev_fd, VIDIOC_QBUF, &v4l2_buf, "Can't queue buffer.");

   return 0;

[ayufan]

Yes, it seems OK. So, I don't fully understand why it is under-run. How you fetch data from? Ah. This is also 3B+. I need to test it, as I was testing this on 4B :)

[goamberg]

You mean install? I use the compile steps in your readme.md

Yes it is compiled on a 3B+, but mostly used on a zero 2 W. Compiling camera-streamer does not work on a zero 2 W. It crashes at some point (probably RAM?).

[ayufan]

It will compile, but with make -j1 - due to RAM.

[goamberg]

thanks for the input, much appreciated !

[ayufan]

Ensure that you do make -j1 install just in case you were running some older version.

[goamberg]

I did some additional testing out of curiosity... I compiled it now directly on a fresh installed and latest updated Zero 2 W and tested with a stock camera module v3 for comparison and to eliminate eventual problems from arducam pivariety libcamera-dev and libcamera-apps.

With this I found similar behavior. Normal libcamera-vid commands can push the videos outside of specs. Inside of the camera-streamer I found that trying a 50fps stream falls back to 30fps directly. But 25fps and 30fps get accepted in 1080p. I tried also 1280x720 with 50fps, which in theory should run well within spec of level 4.0 but only creates sub 40fps stream.

camera-streamer --camera-type=libcamera --camera-width=1280 --camera-height=720 --camera-fps=50 --log-stats -camera-video.options=h264_level=13 -camera-video.options=videobitrate=8000000

Statistics: [  CAMERA 37 FPS/ 0 D/ 16ms/ 31ms/S/Q0:O0:C1] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 36 FPS/ 0 D/ 37ms/ 13ms/S/Q1:O1:C1]
Statistics: [  CAMERA 33 FPS/ 0 D/ 17ms/ 34ms/S/Q0:O0:C1] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 35 FPS/ 0 D/ 33ms/ 15ms/S/Q0:O1:C0]
Statistics: [  CAMERA 38 FPS/ 0 D/ 15ms/ 55ms/S/Q0:O0:C1] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 37 FPS/ 0 D/ 42ms/ 22ms/S/Q0:O2:C1]
Statistics: [  CAMERA 38 FPS/ 0 D/ 16ms/ 34ms/S/Q0:O0:C1] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 38 FPS/ 0 D/ 40ms/ 14ms/S/Q0:O2:C1]
Statistics: [  CAMERA 37 FPS/ 0 D/ 19ms/ 39ms/S/Q0:O0:C1] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 37 FPS/ 0 D/ 40ms/ 14ms/S/Q0:O2:C1]
Statistics: [  CAMERA 38 FPS/ 0 D/ 19ms/ 48ms/S/Q0:O0:C2] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 39 FPS/ 0 D/ 34ms/ 15ms/S/Q0:O1:C0]
Statistics: [  CAMERA 38 FPS/ 0 D/ 55ms/ 71ms/S/Q0:O0:C2] [SNAPSHOT  0 FPS/ 0 D/ -1ms/ -1ms/P/Q0:O0:C0] [   VIDEO 38 FPS/ 0 D/ 88ms/ 14ms/S/Q0:O1:C0]

[ayufan]

Interesting. I just tested this on 2B and see basically the same. It caps at somewhere around 33FPS. Will debug.

[ayufan]

But only on /webrtc, the /video seems to have much higher frame rate.

[goamberg]

Same here. /video is around the 50fps for the 720p stream

[goamberg]

I wanted to ask, if there is any progress on this one, or if there is anything I can help with?

[ayufan]

No progress. I think you need to take a look at it yourself for time being, play with different buffer formats, and buffer counts to see which one gives highest performance.

[ayufan]

@goamberg I think this is related to some mix of CPU governor with fixing the GPU clocks and process priority.