[BUG] Video recording dropped images (freeze)

[matmicro]

I was using picamera (v1) for a long time. Reconding video as h264 file format, and i never faced any video freezing, it was working perfectly.

I moved to picamera2 (in order to benefit from las camera v3 and autofocus), but now i am facing some freeze on videos. Like few images dropped.

Is this a know issue of picamera2 which is specific of libcamera usage ? I this probably due to recording directling in MP4 file format ? Thus is it better to record as h264 ?

Regards

[davidplowman]

Can you include the exact script (a self-contained version that is preferably as short as possible) that demonstrates the problem? I take it that you're using a camera module 3, can you say what kind of a Pi you have? Can you give any indication roughly how many frames get dropped? Thanks.

[matmicro]

I am using Pi4 I don't know how much frames are dropped, but i am recording MP4 at 40fps.

Is it better to switch to recording in H264 file output or it does not change anything at all ?

[davidplowman]

It's true that recording to a flat h.264 file will be more efficient than writing through FFmpeg to an mp4 file. But whether that's necessary is difficult to judge - I don't know what kind of camera configuration you're using, what the output resolution is, whether you're using the most efficient options. If you could create a very small self-contained example that demonstrates the problems you're having then I could offer you better advice.

For example, I ran the following on Pi 4, using Bookworm and the latest Picamera2:

import time
from picamera2 import Picamera2
from picamera2.encoders import H264Encoder
from picamera2.outputs import FfmpegOutput

picam2 = Picamera2()
main = {'size': (1920, 1080)}
controls = {'FrameRate': 40}
config = picam2.create_video_configuration(main=main, controls=controls, buffer_count=12)
encoder = H264Encoder(bitrate=10000000)
output = FfmpegOutput("test.mp4")
output.ptsoutput = "timestamps.txt"
picam2.configure(config)

picam2.start_recording(encoder, output)
time.sleep(30)
picam2.stop_recording()

Analysing the timestamp file suggests it recorded 1194 frames and didn't drop any. However, there could be many other factors involved. There isn't an easy way to pass correct timestamps into FFmpeg, so it has to resample them - and this could be "lumpy". Maybe the system starts to struggle at times, especially if other stuff is running. Maybe the file writing is a bottleneck. Displaying a preview while recording would slow it down. In some respects, 40fps is slightly beyond the official spec which is "up to 30fps", though it normally works, but perhaps not so well with FFmpeg. It would really help to see a (simple) example that is not working for you.

[matmicro]

Thanks for details, i will try as flat h264 to check if it's better. I will change also from 40 fps to 25 fps.

I am writing through network, but with picamera 1.13 i never got such problem, so i am attending the same here.

What is the parameter "buffer_count" ?

FYI here is some part of the code :

vdo_format = 'h264'
vdo_quality = Quality.VERY_HIGH
vdo_qualitySD = Quality.VERY_LOW
vdo_resolution = (1920, 1080)
vdo_resolutionSD = (1024, 768)

    camera = Picamera2()
    video_config = camera.create_video_configuration(main={"size": vdo_resolution},
                                                     lores={"size": vdo_resolutionSD})
    camera.configure(video_config)
    camera.set_controls({"FrameRate": 25})
    camera.set_controls({"Sharpness": 16})
    camera.set_controls({"Saturation": 0.99}) # 1.0 = normal colour saturation (the default), 0.0 makes it greyscale and > 1.0 increases the colour saturation
    camera.set_controls({"Brightness": 0.0})  # 1.0 makes (just about) everything bright white, and -1.0 makes (nearly) everything black, with 0.0 being the default setting
    camera.set_controls({"Contrast": 0.9})  # range of 

    encoder = H264Encoder(bitrate=8000000)
    encoderSD = H264Encoder(bitrate=5000000)

        if vdo_format == 'mp4':
            output = FfmpegOutput(path)
        else:
            output = path
        camera.start_recording(encoder, output, quality=vdo_quality)
        if path_sd:
            if vdo_format == 'mp4':
                outputSD = FfmpegOutput(path_sd)
            else:
                outputSD = path_sd
            camera.start_encoder(encoderSD, outputSD, quality=vdo_qualitySD)

[davidplowman]

buffer_count is how many camera images are in flight. Making this larger makes it more resistant to the vagaries of encode/output times.

A couple of minor things: I'm not sure the quality parameter has any effect if you have given a bitrate. Not totally sure about that, though. The sharpness value of 16 seems extreme - you'll get tons of horrible sharpening artefacts, and then spend a large part of your bit budget encoding them. Also I'm not sure it's pointing the encoderSD at the "lores" output stream when that option is selected. But these are possibly not very important.

It's also not clear to me how you're writing to the network. It might be worth taking things a step at a time. Does it keep up if you write without FFmpeg? Does it keep up if FFmpeg writes to /dev/null? To a real file? What is the effect of the network here? I would also check what bitrate you're actually getting. Sorry to have only vague suggestions.

[matmicro]

Thanks for all details, i will try them one by one to see what improve the situation.

Another thing is that on a video indoor with lot of spot lights in front of camera and contrast it's hard to find the best settings, and i am facing kind of sparkle / blink / twinkle white effect (sorry i don't know how to translate precisely).

Is there a control settings that i could use to reduce this ?

[davidplowman]

It's hard to say without knowing more. Are you able to post a photo or short video of the effect that I could download?

Some random suggestions would be to use the matrix metering mode, which is just a whole image average and therefore less sensitive to any one part of the image. You could perhaps slow up the AGC a bit (that would be a setting in the tuning file). But some kind of picture would clearly be helpful here as this is otherwise total guesswork!

[matmicro]

Here is a sample: [....] Do you have sample for matrix metering ? What is AGC ?

[davidplowman]

I guess that you're referring to the "winking" effect, most visible in the bright lights towards the top of the image?

I think it may be at least partly due to the AEC/AGC (automatic exposure and automatic gain control) algorithm (which controls the scene brightness) reacting to the fast changes in the scene. But I also wonder if there might also be some lighting flicker go on (for instance, many mains powered lights flicker at 100Hz), there certainly seems to be some element of frames going alternately light/dark.

I must confess I'd be very tempted to fix the AEC/AGC while the people are flying around like that. So when the camera starts, maybe let it run normally for 1/2 a second or so, then disable it with

camera.set_controls({'AeEnable': 0})

(use 1 instead of 0 if you want to enable it again later). Or if your lighting is completely fixed and unchanging, you could set explicit exposure and gain values. Disabling the AEC/AGC should certainly indicate to us whether there's anything funny going on with the lights. Please do post another sample video if you're able to do this - a video with no people in it, just a still scene, would be very informative too.

In the clip that you sent I couldn't help noticing that the bitrate seems rather low. All those macroblock compression artefacts possibly contribute to the nasty lighting effects. Maybe that was just for sharing the clip? Have you tried turning up the bitrate? In the worst case, a lower resolution at a higher bitrate would look better.

I'm also not totally sure about the camera module 3 in this application. The danger is that you'll get focus hunting while people fly in and out, and that could be visually annoying too. You could disable the autofocus, but then there's not a lot of point in using the v3 camera. Note that even when you "fix" the focus it's still on the end of a spring, and quite prone to wobble and vibration. (Was this clip filmed with a v3 camera - is that why the image "twitches" in and out?)

[matmicro]

Thanks for advices, i will test it and get back to you.

The quality of the video sent is the lowres version as the goal is to have a small file. (bitrate 1500Kbps). But the winking effect is the same with HD (1920x1080 and 11Mbps bitrate).

Yes it is recorded with camera v3. What do you mean by "why the image "twitches" in and out?" ?

[matmicro]

Regarding the focus and vibration, i am doing also this:

    camera.set_controls({"AfMode": libcamera.controls.AfModeEnum.Manual})  # Manual
    camera.set_controls({"LensPosition" : 1.0 / 4.5})  # (1 / distance in metres), so that zero can be used to denote"infinity"

But i am also facing the same winking effect.

You mean that with v3 camera the vibration will affect the focus hardware which will provoque this ? Rather than camera v2 which has a fixed focus, and then this problem cannot appear ?

Regards.

[davidplowman]

The trouble with the autofocus is that it involves a lens on a wobbly spring (that's how VCMs, voice coil motors, work). Therefore vibrations will cause the lens to wobble slightly, with the result that you will get images that appear to "zoom in and out" ever so slightly. This is what I was noticing in the video.

If it's just wind causing it, then a camera module housing might help, but if it's physical vibration that's more difficult - a fixed focus lens (but maybe where you can adjust the focus to your "several metre" range) may work better.

I don't believe that the lens twitching has anything to do with the brightness flickering, though it's hard to be absolutely sure without testing it. But it would be interesting to test using disabled AGC first to rule that out as an effect. If we discover that the brightness really is changing for some reason then that needs investigating - is it the lights or something else? It is sometimes possible to combat lighting flicker by using particular exposure times, or even ND (neutral density) filters as a last resort.

[matmicro]

Changing the AeEnable to false juste after recording, does not seems to change stuff.

But recording directly into H264 looks solving the winking effect. What could be the reason ?

But while recording with H264, ffpmeg does not encode correctly the FPS into MP4 so playing the video file is faster than real clock. I will try doing ff,peg -r fps

[davidplowman]

I don't understand how recording directly to a flat h.264 file can make any difference. The images will be just the same, the encoder runs just the same. The only difference would be how it's packaged into the file, which shouldn't affect anything that you can see. So that's pretty puzzling.

The h.264 file is just a flat binary file, it's not a true container format. So it won't have proper timestamps in it. Recent versions of vlc can't play timestamp-less files (lots of jumping and glitching), whereas older versions are OK. ffplay will play OK but always at 30 fps, which is what the underlying ffmpeg assumes. -r will probably work well enough!

Alternatively, you could also output a "timestamp" file (as I did in my example). There is software that will allow you to mux from h.264 into a container using a file of timestamps such as makemkv, but it won't make mp4 files. Perhaps you could use ffmpeg to convert mkv to mp4 afterwards!?? Sounds like a bit of a hassle though if -r works for you!