Focal Length Calibration using Python Code

[Hasnain1997-ai]

I have a realsense D415 camera, connected to my Xavier and I am working on it using python

I found the Focal Length Calibration code from this repository, and used the same for focal length calibariton.

`def run_focal_length_calibration(device, target_size, adjust_side): """ Run the focal length calibration process. """ number_of_images = 25 timeout_s = 30

# Configure streams
cfg = rs.config()
cfg.enable_stream(rs.stream.infrared, 1, 1280, 720, rs.format.y8, 30)
cfg.enable_stream(rs.stream.infrared, 2, 1280, 720, rs.format.y8, 30)

# Create frame queues
lq = rs.frame_queue(capacity=number_of_images, keep_frames=True)
rq = rs.frame_queue(capacity=number_of_images, keep_frames=True)

counter = 0
flags = [False, False]

def cb(frame):
    """
    Callback function to process incoming frames.
    """
    nonlocal counter, flags
    if counter > number_of_images:
        return
    for f in frame.as_frameset():
        p = f.get_profile()
        if p.stream_index() == 1:
            lq.enqueue(f)
            flags[0] = True
        if p.stream_index() == 2:
            rq.enqueue(f)
            flags[1] = True
        if all(flags):
            counter += 1
    flags = [False, False]

# Start pipeline
pipe = rs.pipeline()
pp = pipe.start(cfg, cb)

try:
    print('Starting Focal-Length calibration...')
    print(f'\tTarget Size:\t{target_size}')
    print(f'\tSide Adjustment:\t{adjust_side}')
    stime = time.time()
    while counter < number_of_images:
        time.sleep(0.5)
        if timeout_s < (time.time() - stime):
            raise RuntimeError(f"Failed to capture {number_of_images} frames in {timeout_s} seconds, got only {counter} frames")

    # Run calibration
    adev = device.as_auto_calibrated_device()
    fl_adjust_map = {'right_only': 0, 'both_sides': 1}
    table, ratio, angle = adev.run_focal_length_calibration(lq, rq, target_size[0], target_size[1],
                                                            fl_adjust_map[adjust_side], progress_callback)
    print('Focal-Length calibration finished')
    print(f'\tRatio:\t{ratio}')
    print(f'\tAngle:\t{angle}')
    adev.set_calibration_table(table)
    adev.write_calibration()
    return True
except Exception as e:
    print(f"Calibration failed: {str(e)}")
    return False
finally:
    pipe.stop()`

My questions are:

1. Does this code update the focal length of the camera? In the RealSense Viewer app, you have the option to apply changes (new focal length) or not, so does this code update it automatically?
2. Which part of this code is responsible for updating the camera calibration?
3. I want to save the important camera parameters to a JSON file. How can I save the intrinsic and extrinsic camera parameters using Python?

[MartyG-RealSense]

1. The section of the self-calibration white paper document linked to below explains that ideally the focal length of the camera's left and right sensors will be identical, but when they are not identical and negative effects are caused as a result, left and right focal length can be adjusted with the focal length calibration function to correct this imbalance.

https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras#addendum-b-march-2022-focal-length-calibration

2. The focal length calibration operation is activated with the run_focal_length_calibration instruction.

https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras#appendix-c-on-chip-calibration-python-api

In the Python scripting provided in this guide, the focal length calibration command is preceded by cal. instead of adev.

cal.run_focal_length_calibration

3. A RealSense user of Python at https://github.com/IntelRealSense/librealsense/issues/4061 took the approach of saving the intrinsics and extrinsics into an rs2.intrinsics object instead of saving to a json.

I could not find a Python reference for saving intrinsics and extrinsics to a json file. Someone who looked at the problem commented though, "If anyone is interested, I have successfully serialized and saved the intrinsics to disk. They are simple data-type structs, so a memcpy to a char* array and using ofstream to write the file (and ifstream to read it back) does indeed work, and reversing is also very simple as well."

[Hasnain1997-ai]

I adjusted the code as

`def run_focal_length_calibration(device, target_size, adjust_side): """ Run the focal length calibration process. """ number_of_images = 25 timeout_s = 30

# Configure streams
cfg = rs.config()
cfg.enable_stream(rs.stream.infrared, 1, 1280, 720, rs.format.y8, 30)
cfg.enable_stream(rs.stream.infrared, 2, 1280, 720, rs.format.y8, 30)

# Create frame queues
lq = rs.frame_queue(capacity=number_of_images, keep_frames=True)
rq = rs.frame_queue(capacity=number_of_images, keep_frames=True)

counter = 0
flags = [False, False]

def cb(frame):
    """
    Callback function to process incoming frames.
    """
    nonlocal counter, flags
    if counter > number_of_images:
        return
    for f in frame.as_frameset():
        p = f.get_profile()
        if p.stream_index() == 1:
            lq.enqueue(f)
            flags[0] = True
        if p.stream_index() == 2:
            rq.enqueue(f)
            flags[1] = True
        if all(flags):
            counter += 1
    flags = [False, False]

# Start pipeline
pipe = rs.pipeline()
pp = pipe.start(cfg, cb)

try:
    print('Starting Focal-Length calibration...')
    print(f'\tTarget Size:\t{target_size}')
    print(f'\tSide Adjustment:\t{adjust_side}')
    stime = time.time()
    while counter < number_of_images:
        time.sleep(0.5)
        if timeout_s < (time.time() - stime):
            raise RuntimeError(f"Failed to capture {number_of_images} frames in {timeout_s} seconds, got only {counter} frames")

    # Run calibration
    pipe = rs.pipeline()
    cfg = rs.config()
    cfg.enable_stream(rs.stream.depth, 256, 144, rs.format.z16, 90)
    dev = pipe.start(cfg).get_device()
    cal = rs.auto_calibrated_device(dev)
    fl_adjust_map = {'right_only': 0, 'both_sides': 1}
    table, ratio, angle = cal.run_focal_length_calibration (lq, rq, target_size[0], target_size[1],  fl_adjust_map[adjust_side], progress_callback)

    print('Focal-Length calibration finished')
    print(f'\tRatio:\t{ratio}')
    print(f'\tAngle:\t{angle}')
    rs.set_calibration_table(table)
    rs.write_calibration()
    return True
except Exception as e:
    print(f"Calibration failed: {str(e)}")
    return False
finally:
    pipe.stop()

success = run_focal_length_calibration(device, (args.target_width, args.target_height), args.focal_adjustment) `

is it correct now? Will it update the focal length?

[MartyG-RealSense]

If there are not any errors then it has likely succeeded, but there is not a way to confirm it.

[Hasnain1997-ai]

I wanted to ask if this is the correct way to do Focal Length Calibration programmatically.

[MartyG-RealSense]

As far as I am aware, you are the first person to write a focal length calibration script for Python. It is not an often used feature, and when it is used it is typically performed by users in the RealSense Viewer rather than with code. So there are not any other focal length calibration scripts to compare it to in order to verify its correctness, unfortunately,

[Hasnain1997-ai]

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/depth_auto_calibration_example.py

I took the idea from this code you can see here the function: run_focal_length_calibration

[MartyG-RealSense]

Thanks very much for the reminder about that official RealSense SDK calibration script for Python, which includes focal length calibration instructions as part of a complete calibration process that includes On-Chip calibration (depth quality improvement), Tare calibration (depth accuracy improvement) and Focal Length calibration (balancing left and right focal lengths).

If your script is a stripped-down version of the example script that only runs the focal length calibration operation and there are no errors then I see no reason why it would not be correct, since the different types of calibration are independent of each other and not running one will not have a negative effect on the others.

[Hasnain1997-ai]

you mentioned that cal.run_focal_length_calibration is the method

While in the official RealSense SDK calibration script for Python I see : adev.run_focal_length_calibration

So which one should I follow now

[MartyG-RealSense]

As the official example script uses adev, I recommend sticking with that, as a full script is a more reliable reference than the couple of example lines in the guide appendex that use cal. Changing how the example script works increases the risk of it going wrong.

[Hasnain1997-ai]

OK Thank you for your assistance