Dual cameras recording with SSD

[GuyPerets106]

Hi, I'm using an 8GB CM4 with IO board which has 2 CSI connectors. I connected 2 Waveshares OV9281 to it and I'm trying to record 1280x720 8bit video at 120fps from both cameras in parallel. An SSD is connected to the IO board which gives me a write speed of around 380MB/sec. When I'm using libcamera-vid --level 4.2 -o testR.h264 --mode 1280:720:8:P --width 1280 --height 720 --framerate 120 --frames 1200 -n --denoise off --camera 1 libcamera-vid --level 4.2 -o testL.h264 --mode 1280:720:8:P --width 1280 --height 720 --framerate 120 --frames 1200 -n --denoise off --camera 0 at the same time (from 2 terminals as for now), I get a 10 seconds video from each, which actually took 20sec to record, and the video is at high speed (like a timelapse video), and this is not what I intended. Am I doing something wrong? Is there a quick fix for this problem? Thanks!

[6by9]

https://datasheets.raspberrypi.com/rpi4/raspberry-pi-4-product-brief.pdf

Multimedia: H.265 (4Kp60 decode); H.264 (1080p60 decode, 1080p30 encode);

1080p30 being H264 level 4.0. https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels will tell you that limit is 245,760 macroblocks (16x16) per second.

Under many circumstances you can push it up to 1080p60 for level 4.2. Level 4.2 is 522,240 macroblocks per second.

1280x720@120 would be 432000 macroblocks/sec, so above level 4.0, but not needing the full capabilities of level 4.2.

With two simultaneous encodes you're looking at 864000 macroblocks/sec. So you're trying to push the encoder at over 3.5x the rated spec. The hardware can't keep up with that and will result in dropped frames.

[GuyPerets106]

https://datasheets.raspberrypi.com/rpi4/raspberry-pi-4-product-brief.pdf

Multimedia: H.265 (4Kp60 decode); H.264 (1080p60 decode, 1080p30 encode);

1080p30 being H264 level 4.0. https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels will tell you that limit is 245,760 macroblocks (16x16) per second.

Under many circumstances you can push it up to 1080p60 for level 4.2. Level 4.2 is 522,240 macroblocks per second.

1280x720@120 would be 432000 macroblocks/sec, so above level 4.0, but not needing the full capabilities of level 4.2.

With two simultaneous encodes you're looking at 864000 macroblocks/sec. So you're trying to push the encoder at over 3.5x the rated spec. The hardware can't keep up with that and will result in dropped frames.

I managed to record 1280x800 253fps 8bit video with Innomaker OV9281-V2 , using their driver and an SSD for fast write speeds. Of course it's a different situation because that was RAW video , but the thing you're describing sounds like it's related to CM4 hardware. So if that problem is related to hardware encoding using libcamera-vid , does libcamera-raw can reach the same results as Innomaker's driver , without skipping frames? Their git: https://github.com/INNO-MAKER/CAM-OV9281RAW-V2

[6by9]

I managed to record 1280x800 253fps 8bit video with Innomaker OV9281-V2 , using their driver and an SSD for fast write speeds

Really? That exceeds the spec that Omnivision publish for the sensor. They state 1280x800 @ 120fps, although that is 10bit. I don't see dropping to 8 bit over doubling the frame rate (it normally adds 25%. 640x400 @ ~250fps is plausible. Commented last week that 640x400 in 8bit mode can in theory do 320fps. 10bit should do 260fps.

So if that problem is related to hardware encoding using libcamera-vid , does libcamera-raw can reach the same results as Innomaker's driver , without skipping frames?

It's a limitation of the H264 codec that you're seeing. Yes libcamera-raw should be able to go faster. There's still a limit to the speed that the ISP can process the frames for statistics purposes, but IIRC that should be up around 400MPix/s for Pi4/CM4.

Their git: https://github.com/INNO-MAKER/CAM-OV9281RAW-V2

Another GPL licence violation by them - nice. They just don't care as it was raised over 2 years ago for one of their other sensors - https://gitee.com/inno-maker/cam-mipi327raw/issues/I22HTM

[GuyPerets106]

I managed to record 1280x800 253fps 8bit video with Innomaker OV9281-V2 , using their driver and an SSD for fast write speeds

Really? That exceeds the spec that Omnivision publish for the sensor. They state 1280x800 @ 120fps, although that is 10bit. I don't see dropping to 8 bit over doubling the frame rate (it normally adds 25%. 640x400 @ ~250fps is plausible. Commented last week that 640x400 in 8bit mode can in theory do 320fps. 10bit should do 260fps.

Sorry , I meant mode 1 (which gets 144fps at 1280x800 resolution 8bit).

So if that problem is related to hardware encoding using libcamera-vid , does libcamera-raw can reach the same results as Innomaker's driver , without skipping frames?

It's a limitation of the H264 codec that you're seeing. Yes libcamera-raw should be able to go faster. There's still a limit to the speed that the ISP can process the frames for statistics purposes, but IIRC that should be up around 400MPix/s for Pi4/CM4.

So H264 is the best codec to use in my situation? MJPEG , yuv420 or libav could perform better?

And if libcamera-raw could potentially record what I'm trying to see , how can I know the pixel format of IMX708? (Module 3) I assume it's SBGGR10 from what I'm seeing in Selected sensor format: 1536x864-SBGGR10_1X10 - Selected unicam format: 1536x864-BG10 but FFmpeg doesn't have this pixel format for conversion , so maybe I'm wrong?

[GuyPerets106]

By the way , you once told me about the Waveshare board oscillator which has a 12MHz written on it , like it shows on the picture on their website - https://www.waveshare.com/OV9281-120-Camera.htm But I checked on the back of the board I actually got , and it shows 24MHz. So I'm not sure what is right about it , and that was the only reason I could think about.

[6by9]

IMX708 is a raw Bayer sensor, hence producing SBGGR10. FFmpeg doesn't generally handle that as it needs significant sensor specific processing to make it useful, and that is what libcamera is all about.

yuv420 for uncompressed frames is largely your only option for very high framerates, but be aware that you're then handling lots of data.

[naushir]

Anything further to look at here? If not, I'll close this down shortly.

[GuyPerets106]

Anything further to look at here? If not, I'll close this down shortly.

I'm still trying to understand if there's a limitation to memory writing with libcamera-raw and SSD attached. Theoretically, I should have no problem writing raw data at high resolution and high framerate , am I right? You know why I'm still getting frame drops if the data don't go through the H264 encoder?

[Schaggo]

This maybe anecdotal and might actually have been solved in the current version. I’m running a custom libcamera app designed to capture a defined number of precisely timed frames at max res/fps.

One bottle neck I encountered was memory management. I don’t know the exact number anymore, but it was something like after 10 frames RAW the 11th would be captured with a significant delay. The solution was to implement allocate memory ahead of time, so that the program allocates buffers for the specified amount of frames before filling them.

This will be only useful for a couple of frames limited by the system’s RAM, but maybe a worker that writes the buffer array to disk could improve usability of this concept.

[naushir]

Anything further to look at here? If not, I'll close this down shortly.

I'm still trying to understand if there's a limitation to memory writing with libcamera-raw and SSD attached. Theoretically, I should have no problem writing raw data at high resolution and high framerate , am I right? You know why I'm still getting frame drops if the data don't go through the H264 encoder?

libcamera-raw still processes frames through the ISP (since it needs statistics for AE/AWB correction). As such, you will still need --denoise off with libcamera-raw. Can you provide the exact command line you are using with libcamera-raw?

As @Schaggo mentioned, there may be an issue with lack of buffering and slower IO operations cause some frame drops. You can try increasing the number of buffers used by the --buffer-count argument. The default is 6 buffers, so perhaps try 10 and see if that helps?