[HOW-TO] Have two cameras with different tunning files

[ilirosmanaj]

Given an RPI that has two cameras (RGB and IR Camera), how can both of these be loaded with different tunning file settings?

Describe what it is that you want to accomplish I have a real time application and needs both of these cameras running simultaneously, the RGB camera and the IR camera. However, I need to apply different exposure ranges for these cameras auto-exposure run.

Currently, whichever camera I initialize first, it sticks with the tunning file of this camera for both cameras.

From the initialization of the Picamera, I see this:

This means we only have one tunning file environment variable being exported for libcamera, which is probably the reason why we cannot have two tunning files.

Additional context

A sample to reproduce is here:

import logging
import time
from enum import Enum
from typing import Optional

from picamera2 import Picamera2
from picamera2.outputs import FileOutput
from picamera2.encoders import H264Encoder

logging.root.setLevel(logging.NOTSET)

class CameraType(Enum):
    RGB = "rgb"
    IR = "ir"

libcamera_base_path = "/usr/share/libcamera/ipa/rpi/vc4/"

def initialize_picamera2(camera_type: CameraType) -> Optional[Picamera2]:
    camera_index = 0 if camera_type == CameraType.IR else 1
    logging.info(f"Initializing the {camera_type} camera on index {camera_index}")

    camera_model_prefix = "imx708_wide"
    if camera_type == CameraType.RGB:
        actual_camera_model = f"{camera_model_prefix}.json"        
        exposure_range = [2500, 3000, 3500, 4000, 4500]
        gain_range = [2.5, 3.0, 4.0, 5.0, 6.0]

    else:
        actual_camera_model = f"{camera_model_prefix}_noir.json"        
        exposure_range = [500, 1000, 1500]
        gain_range = [2.5, 3.0, 4.0]

    try:
        tuning = Picamera2.load_tuning_file(f"{libcamera_base_path}{actual_camera_model}").copy()
        logging.info(f"Loading from full camera path: {libcamera_base_path}{actual_camera_model}")

        ae_settings = Picamera2.find_tuning_algo(tuning, "rpi.agc").copy()

        ae_settings["exposure_modes"]["normal"] = {"shutter": exposure_range, "gain": gain_range}
        logging.info(f"Set the exposure ranges to: {exposure_range}")
        logging.info(f"Set the gain ranges to: {gain_range}")

        camera = Picamera2(camera_index, tuning=tuning)

        camera.set_logging(level=logging.INFO)
        if camera_type == CameraType.RGB:
            video_config = camera.create_video_configuration(
                main={"size": (2304, 1296), "format": "RGB888"}, 
                lores={"size": (128, 72)},
                encode="lores", 
                buffer_count=6,
                queue=False,
                controls={
                    "Brightness": 0,
                    "Contrast": 1.4,
                    "Saturation": 1,
                    "AeConstraintMode": 0,
                    "AeEnable": True,
                    "ExposureTime": 3000,
                    "AnalogueGain": 3,
                    "AeExposureMode": 0,
                    "ExposureValue": 0,
                    "AeMeteringMode": 1,
                    "AwbEnable": False,
                    "AwbMode": 0,
                    "ColourGains": (2.0, 1.5),
                    "AfMetering": 1,
                    "AfMode": 1,
                    "AfRange": 1,
                    "AfSpeed": 1,
                    "FrameRate": 80,
                    "LensPosition": 13,
                }
            )
            camera.configure(video_config)
        else:
            video_ir_config = camera.create_video_configuration(
                main={"size": (2304, 1296), "format": "RGB888"}, 
                lores={"size": (128, 72)},
                encode="lores", 
                buffer_count=6,
                queue=False,
                controls={
                    "Brightness": 0,
                    "Contrast": 2,
                    "Saturation": 1,
                    "AeConstraintMode": 0,
                    "AeEnable": True,
                    "ExposureTime": 3500,
                    "AnalogueGain": 3.0,
                    "AeExposureMode": 0,
                    "ExposureValue": 0,
                    "AeMeteringMode": 1,
                    "AwbEnable": False,
                    "AwbMode": 0,
                    "ColourGains": (1, 1),
                    "AfMetering": 0,
                    "AfMode": 0,
                    "AfRange": 1,
                    "AfSpeed": 1,
                    "LensPosition": 15,
                    "FrameRate": 80,
                },
            )
            camera.configure(video_ir_config)
        camera.stop()
        return camera
    except Exception:
        logging.exception(f"Failed to initialize pi2camera on index {camera_index}", exc_info=True)
        return None

# initialization
IR_CAMERA = initialize_picamera2(camera_type=CameraType.IR)
NORMAL_CAMERA = initialize_picamera2(camera_type=CameraType.RGB)

print("Both cameras are initialized.")

encoder = H264Encoder()
output = FileOutput()
NORMAL_CAMERA.start_recording(encoder, output)
NORMAL_CAMERA.set_controls({"AeEnable": True, "AfMode": 1})

# Now when it's time to start recording the output, including the previous 5 seconds:
output.fileoutput = "file.h264"
output.start()

print("Normal camera started.")

for i in range(5):
    request = NORMAL_CAMERA.capture_request()
    image = request.make_array("main")
    metadata = request.get_metadata()
    request.release()

    exposure_time = metadata['ExposureTime']
    analogue_gain = round(metadata['AnalogueGain'], 3)

    print(f"[RGB Capture][Capture {i}] Exposure: {exposure_time}, Gain: {analogue_gain}")
    time.sleep(0.2)

NORMAL_CAMERA.stop_recording()
output.stop()

IR_CAMERA.start_recording(encoder, output)
output.start()
print("IR camera started.")

for i in range(5):
    request = IR_CAMERA.capture_request()
    image = request.make_array("main")
    metadata = request.get_metadata()
    exposure_time = metadata['ExposureTime']
    analogue_gain = round(metadata['AnalogueGain'], 3)

    print(f"[IR Capture][Capture {i}] Exposure: {exposure_time}, Gain: {analogue_gain}")
    time.sleep(0.1)

IR_CAMERA.stop_recording()
output.stop()

If I use this order of initialization:

IR_CAMERA = initialize_picamera2(camera_type=CameraType.IR)
NORMAL_CAMERA = initialize_picamera2(camera_type=CameraType.RGB)

# output
Normal camera started.
[RGB Capture][Capture 0] Exposure: 1496, Gain: 4.0
[RGB Capture][Capture 1] Exposure: 1496, Gain: 4.0
[RGB Capture][Capture 2] Exposure: 1496, Gain: 4.0
[RGB Capture][Capture 3] Exposure: 1496, Gain: 4.0
[RGB Capture][Capture 4] Exposure: 1496, Gain: 4.0
picamera2.picamera2 INFO: Camera stopped
INFO:picamera2.picamera2:Camera stopped
picamera2.picamera2 INFO: Camera started
INFO:picamera2.picamera2:Camera started
IR camera started.
[IR Capture][Capture 0] Exposure: 1496, Gain: 4.0
[IR Capture][Capture 1] Exposure: 1496, Gain: 4.0
[IR Capture][Capture 2] Exposure: 1496, Gain: 4.0
[IR Capture][Capture 3] Exposure: 1496, Gain: 4.0
[IR Capture][Capture 4] Exposure: 1496, Gain: 4.0
picamera2.picamera2 INFO: Camera stopped
INFO:picamera2.picamera2:Camera stopped

If I use the other order:

NORMAL_CAMERA = initialize_picamera2(camera_type=CameraType.RGB)
IR_CAMERA = initialize_picamera2(camera_type=CameraType.IR)

# output
Normal camera started.
[RGB Capture][Capture 0] Exposure: 4489, Gain: 5.988
[RGB Capture][Capture 1] Exposure: 4489, Gain: 5.988
[RGB Capture][Capture 2] Exposure: 4489, Gain: 5.988
[RGB Capture][Capture 3] Exposure: 4489, Gain: 5.988
[RGB Capture][Capture 4] Exposure: 4489, Gain: 5.988
picamera2.picamera2 INFO: Camera stopped
INFO:picamera2.picamera2:Camera stopped
picamera2.picamera2 INFO: Camera started
INFO:picamera2.picamera2:Camera started
IR camera started.
[IR Capture][Capture 0] Exposure: 4489, Gain: 5.988
[IR Capture][Capture 1] Exposure: 4489, Gain: 5.988
[IR Capture][Capture 2] Exposure: 4489, Gain: 5.988
[IR Capture][Capture 3] Exposure: 4489, Gain: 5.988
[IR Capture][Capture 4] Exposure: 4489, Gain: 5.988
picamera2.picamera2 INFO: Camera stopped
INFO:picamera2.picamera2:Camera stopped

So the exposure range is not being respected in these.

[davidplowman]

Yes, I'm afraid you can only have one tuning file per type of camera. In fact, it's a limitation down in the libcamera API and is something we've long wanted to remove. You may have more luck if you run the cameras in separate processes - might be an option for you?

[ilirosmanaj]

I run the cameras through a Flask API which gets invoked at specific endpoints. Not sure if multiprocessing (created from within the app) would be something that could help us here - need to check how the synchronization between the two cameras would work, since they more or less need to capture at the same time and there are some information exchanged (e.g. we want to use the same Lens position on both cameras, and this lens depends on the user specifications on the request).

[ilirosmanaj]

@davidplowman, from your Yes, I'm afraid you can only have one tuning file per type of camera response. What does type of camera actually mean here? These two are different in hardware and also have different tunning files - but maybe you're referring to something else

[ilirosmanaj]

@davidplowman any updates on the above maybe?

[davidplowman]

Sorry, I thought I'd replied but obviously not. As long as your cameras load different tuning files by default, then you can in principle overwrite them and have your own tuning file for those cameras. In this context, anything with a different sensor counts as different, for example "imx219", "imx477" and so on. I think "imx296" and "imx296 mono" count as different because the camera module is programmed to know which it is. Ditto "imx708" and "imx708 wide", again because I believe it's programmed into the module.

What cameras are you wanting to use? Can you say anything about what you want to tune differently?

[ilirosmanaj]

Thanks for the reply @davidplowman.

I use imx708_wide and imx708_wide_noir cameras. The only setting that I want to tune differently are the exposure ones (the exposure and gain ranges, together with the autoexposure tunning parameters).

[davidplowman]

I'm not sure that "noir" counts as different. But your life may be much easier if you just define two exposure modes which are the ones you want. One is still called "normal" and the other "short", but the name doesn't matter. Simply use set_controls to set one camera to "normal", and the other to "short". Could that work?

[ilirosmanaj]

Potentially yes, that is a very good idea. Let me try that and will get back to you

[ilirosmanaj]

@davidplowman, that worked, thank you very much.

This, however, doesn't let me choose different AutoExposure algorithm parameters per camera type. This is what I have now:

        ae_settings = Picamera2.find_tuning_algo(tuning, "rpi.agc")

        ae_settings["exposure_modes"]["normal"] = {
            "shutter": [2500, 3000, 3500, 4000, 4500], 
            "gain": [2.5, 3.0, 4.0, 5.0, 6.0]
        }
        ae_settings["exposure_modes"]["short"] = {
            "shutter": [2000, 2500, 3000, 3500, 3700], 
            "gain": [2.0, 2.5, 3.0, 3.5, 3.7]
        }

        ae_settings["startup_frames"] = 2
        ae_settings["convergence_frames"] = 3
        ae_settings["desaturate"] = 0
        ae_settings["base_ev"] = 1.5
        ae_settings["speed"] = 1

Any ideas?

[davidplowman]

Hi again, I'm not even sure I'd try to fiddle with the tuning file at all. I'd be tempted to try

from picamera2 import Picamera2
from libcamera import controls
# ...
cam0 = Picamera2(0)
cam1 = Picamera2(1)
# ...
cam0.set_controls({'AeExposureMode': controls.AeExposureModeEnum.Normal})
cam1.set_controls({'AeExposureMode': controls.AeExposureModeEnum.Custom})

where you can simply edit all the tuning files to have the "normal" and "custom" modes that you want.

[ilirosmanaj]

This suggestion helped actually: https://github.com/raspberrypi/picamera2/issues/1094#issuecomment-2291461591

So closing this issue for me.