[BUG] Number of frames is higher than set with FrameDurationLimits

[marcoabrate]

Describe the bug I am setting the FPS of the camera to quite a low value by changing the FrameDurationLimits camera control. However, the output recording has (much) more frames than expected.

With FPS set to 15 (i.e. FrameDurationLimits set to 66,666) and length of recording set to 600 seconds, number of frames is 9,712 when calculated with two different methods (ffprobe and opencv, see later), while it should be 9,000. Almost a 10% error.

To Reproduce To record the h264 video

length = 600 # seconds
FPS = 15
frame_duration = int((1./FPS)*1e6)
picam2 = Picamera2()
config = picam2.create_video_configuration(
    sensor = {
        'bit_depth': 10,
        'output_size': (2592, 1944)
    },
    main = {
        'size': (1296, 972)
    },
    controls = {
        'FrameDurationLimits': (frame_duration, frame_duration) # (min, max) microseconds
    }
)
picam2.configure(config)
pprint(config)

encoder = H264Encoder(framerate=FPS, enable_sps_framerate=True)
output = "test_hq.h264"

picam2.start_recording(encoder, output, quality=Quality.VERY_HIGH)

start_time = time.time()
while True:
    seconds = time.time() - start_time
    if seconds >= length : break

picam2.stop_recording()
picam2.close()

To count the number of frames

ffprobe -v error -select_streams v:0 -count_frames -show_entries stream=nb_read_frames -of csv=p=0 test_hq.h264

or

video = cv2.VideoCapture("test_hq.h264")

n_frames = 0
while True:
    ret, frame = video.read()
    if not ret : break

    n_frames += 1

print(f"There are {n_frames} frames")

Hardware : Raspberry Pi 4 model B (8GB) Camera Arducam OV5647

[davidplowman]

Not quite sure what's happening there. I wonder if the camera driver is not getting things quite right. If you save a timestamp file (use output = FileOutput("test.h264", pts="timestamps.txt"), what are the frame to frame diffs that you see in there?

[marcoabrate]

Hi @davidplowman thank you for your reply. I get the same result in timestamps.txt. In particular, the length of the file is 9712 as previously and the average delta is 61.725 instead of 66.666, with the highest delta being 61.75.

I thought it might be related to some hardware settings, so the camera just falls back to the nearest FPS hw setting? I am getting delivered an official Raspberry Camera module 3 tomorrow and I will try with that one and let you know how it performs.

[davidplowman]

Indeed, it sounds like the camera driver isn't setting things up quite right. The discrepancy in those times does seem to account for the extra frames. The camera not going fast enough is a well enough known phenomenon, but not slow enough is a bit weird!

I guess the old ov5647 doesn't get too much love these days, but I'd be mildly curious if there are framerates that it gets right. Maybe you can confirm that you have a recent kernel (output of uname -a). It might be worth filing a camera driver bug under the raspberrypi/linux repo, though I expect a v3 camera will behave properly.

[marcoabrate]

thank you @davidplowman I can confirm that with a Raspberry Camera module 3 the number of frames corresponds with the framerate set in the code. I am closing this issue.