[BUG] Time synchronization of dual-camera Pi5 system

[475651582]

Describe the bug Basically I need to make sure the two HQ cameras on my Pi5 board to capture a picture simultaneuosly within 1ms. I noticed there is 'SensorTimestamp' in the metadata of each captured request. So by controlling the frame duration, I aligned the requests of two HQ cameras by 'SensorTimestamp' with some PID-like algorithm.

However, I found that when two HQ cameras capture images in a very close SensorTimestamp, the captured images were not strictly time synchronized. For instance, they are shooting at a high-refresh-rate stopwatch, and the images they captured showed the images were not captured in the same time, always with a 8~10ms uncertain delay. I pasted a example above, left camera (b) captured a '13:23:702' at while right camera (a) captured '13:23:712' with about 10ms delay. But the timestamp from the metadata showed b is 1460269180us while a is 1460269315us, only less than 200us, which is much smaller than real delay.

I do know that camera need some readout time to transfer image buffer to Pi board, but will that takes about 10ms delay? Could anyone help to explain why that 10ms delay exists?

Below is my python code for reproduce (Pi5 4GB board, 2 HQ cameras connected) and note that you need a very high refresh speed monitor for the stopwatch (my is a 240hz screen). To Reproduce

from picamera2 import Picamera2
from picamera2 import MappedArray
import time
from PIL import Image

def P_controller(Kp: float = 0.05, setpoint: float = 0, measurement: float = 0, output_limits=(-10000, 10000)):
    e = setpoint - measurement
    P = Kp * e

    output_value = P

    # output and limit if output_limits set
    lower, upper = output_limits
    if (upper is not None) and (output_value > upper):
        return upper
    elif (lower is not None) and (output_value < lower):
        return lower
    return output_value

if len(Picamera2.global_camera_info()) <= 1:
    print("SKIPPED (one camera)")
    quit()

# Primary (leads)
picam2a = Picamera2(0)
# need buffer_count > 1 because if a frame is skipped, there will be a jump in SensorTimestamp due to dropped frame which messes with the control
config2a = picam2a.create_video_configuration(controls={"FrameRate": 10, "ExposureTime": 100, "AnalogueGain": 22.0,"AeEnable": False,"AwbEnable": False,"AeFlickerMode": 0}, buffer_count=3)
picam2a.configure(config2a)

# Secondary (follows)
picam2b = Picamera2(1)
# need buffer_count > 1 because if a frame is skipped, there will be a jump in SensorTimestamp due to dropped frame which messes with the control
config2b = picam2b.create_video_configuration(controls={"FrameRate": 10, "ExposureTime": 100, "AnalogueGain": 22.0,"AeEnable": False,"AwbEnable": False,"AeFlickerMode": 0}, buffer_count=3)
picam2b.configure(config2b)

# picam2a.request_callback = capture_callback_2a
# picam2b.request_callback = capture_callback_2b
picam2a.start()
picam2b.start()

print("Press Ctrl+C to exit")
try:
    while True:
        job_a = picam2a.capture_request(wait=False)
        job_b = picam2b.capture_request(wait=False)
        request_a = picam2a.wait(job_a)
        request_b = picam2b.wait(job_b)
        metadata_picam2a = request_a.get_metadata()
        metadata_picam2b = request_b.get_metadata()

        timestamp_picam2a = metadata_picam2a["SensorTimestamp"] / 1000  #  convert ns to µs because all other values are in µs
        timestamp_picam2b = metadata_picam2b["SensorTimestamp"] / 1000  #  convert ns to µs because all other values are in µs
        timestamp_delta = timestamp_picam2b - timestamp_picam2a

        controller_output_frameduration_delta = int(P_controller(0.05, 0, timestamp_delta, (-10000, 10000)))
        control_out_frameduration = int(metadata_picam2a["FrameDuration"] + controller_output_frameduration_delta) 
        print("FrameDurationDelta: ", controller_output_frameduration_delta, "new FrameDurationLimit: ", control_out_frameduration,metadata_picam2a['FrameDuration'],metadata_picam2b['FrameDuration'])

        with picam2b.controls as ctrl:
            # set new FrameDurationLimits based on P_controller output.
            ctrl.FrameDurationLimits = (control_out_frameduration, control_out_frameduration)

        last_frame_duration = control_out_frameduration

        if abs(controller_output_frameduration_delta) <= 1000:

            # Save images with timestamp as filename
            filename_a = f"a/image_cam_a_{int(timestamp_picam2a)}.jpg"
            filename_b = f"b/image_cam_b_{int(timestamp_picam2b)}.jpg"
            request_a.save("main",filename_a)
            request_b.save("main",filename_b)
            print(f"Images saved: {filename_a}, {filename_b}")
            request_a.release()
            request_b.release()

except KeyboardInterrupt:
    print("got Ctrl+C, exiting")

picam2a.stop()
picam2b.stop()

[davidplowman]

Hi, it's difficult to know what's happening. If the sensor timestamps are close, then I'm struggling to see why there should be such a difference. Just to be sure I'm understanding, you're saying that timestamp_delta is less than 200, but that you get images like the ones you attached, showing a ~12ms difference on the clock.

Can you just confirm, is the stopwatch located in the same place in both camera images? With the HQ cam using a rolling shutter sensor, this would make a difference.

[475651582]

Hi, it's difficult to know what's happening. If the sensor timestamps are close, then I'm struggling to see why there should be such a difference. Just to be sure I'm understanding, you're saying that timestamp_delta is less than 200, but that you get images like the ones you attached, showing a ~12ms difference on the clock.

Can you just confirm, is the stopwatch located in the same place in both camera images? With the HQ cam using a rolling shutter sensor, this would make a difference.

Hi Davidplowman, thanks very much for your help! I reviewed the images. If the clocks are on the same position of the images, the syncs are good, otherwise the syncs are bad. The issue solved! Thanks a lot!