[OTHER] Colour Correction

[Bra1nsen]

Iam still interested in estimating Solar Irradiance with a Raspberry All Sky Imager.

I would like to know, if the color calibration was already evaluated? How precise is the sRGB colour spaced projected?

I know the solar spectral distribution:

The following Diagram shows the sRGB field:

Its possible to calculate the wavelengths corrections for the sRGB colorspace from CIE 1931 Space:

So when we capture images with the raspberry hq cam, the images should be sRGB or? Since then it should be possible to estimate the irradiance for the visible spectrum with a physical model.

Iam just interested in learning more about the color calibration described on raspberry pi tuning guide, and if its already evaluated.

[davidplowman]

The images that you get from the Pi are indeed sRGB, but with some caveats.

They do use the sRGB primaries, but we don't use the standard sRGB transfer function in our gamma block. That's because it produces rather pale washed out images. So far as I know, all camera vendors tweak this function for aesthetic reasons, and we do the same.

The YUV values that we use to create JPEG images are created using the BT.601 YCbCr conversion matrix.

The colour matrices that we use in our pipeline are fitted using a rather simple procedure, directly in RGB space. I've always wanted to go back an improve this, perhaps by fitting in Lab space to reduce colour errors overall, but have never got round to it.

In your case, I wonder if the capture conditions are sufficiently fixed that you could use the raw sensor values directly (after some calibration perhaps). My worry would be that both AWB and the colour matrices potentially introduce quite a lot of uncertainly - but that's just me thinking out loud!

[Bra1nsen]

Thank you for the assessment David. I appreciate your knowledge! And thanks for thinking out loud, that helps immensely.

Such a lab for color correction is certainly quite elaborate, at least if you want to be very precise, since the exposure of the scene must be adjusted exactly.

• Ok interesting, the sRGB primaries are set, but in order for this color space to actually be mapped, the following parameters need to be calibrated accordingly:

1. White balance: Adjusts the overall color temperature of an image to ensure accurate color reproduction under different lighting conditions.

2. Color space: A defined range of colors that a camera can reproduce, such as sRGB, Adobe RGB, or ProPhoto RGB.

3. Hue: Controls the dominant color in the image and can be adjusted for more accurate color representation.

4. Saturation: Refers to the intensity or vibrancy of colors in an image. Adjusting saturation can help achieve more accurate color reproduction.

5. Luminance: Controls the brightness of colors in the image, and adjusting it can help maintain color accuracy.

6. Color gain (chroma gain): A parameter that controls the strength or intensity of colors in an image by amplifying or attenuating the color signals.

7. Gamma correction: Adjusts the relationship between the input signal and the brightness of the output image, ensuring appropriate brightness and contrast levels.

8. Tone curve: Adjusts the distribution of tones in the image, which can influence color reproduction and contrast.

9. Color matrix: A matrix used to transform color values from the camera's sensor to a specific color space. Adjusting the color matrix can help achieve better color accuracy.

10. Color profile: A custom profile created based on the analysis of a reference image, used to correct discrepancies in color reproduction.

11. Camera sensor characteristics: Specific features of the camera sensor that may influence color reproduction, such as sensor sensitivity, dynamic range, and noise performance.

12. Image processing pipeline: The various stages of processing that the image undergoes within the camera, such as demosaicing, noise reduction, and sharpening, which can affect color accuracy.

13. Illuminant and observer profiles: These describe the characteristics of the light source and the observer's perception of color, which are essential for color calibration in specific contexts, such as display devices and printing.

right? (researched a bit)

I would be interested to know how the plot 2 (CIE sRGB) changes when, for example, the gamma block is changed.

• So the YUV color space is more accurate than the sRGB color space?

I think the approach via the raw pixel intensities will be the best. But for that I would need to know exactly these 3 functions and espically their borders:

Do you actually have the entire Sony IMX477 datasheet? I know that it canbe requested..

Kind regards

[Bra1nsen]

What happend?

[davidplowman]

Hi, there's been a code re-organisation. The code is all still there, in this case raspberrypi was renamed rpi (some folks got fed up of typing the longer string).

[Bra1nsen]

Ah okay thanks David. I know you have great impact in Raspberry Cameras Group, if Raspberry decides to develope software for new sony sensor, I would highly recommend the IMX990 :D

[Bra1nsen]

Iam wondering if your Team is working on some kind of auto exposure setting..

With the idea to "capture" (not saving) over the whole exposure timeline (50μs ~ 120s) with a rough resolution. In Example:

rough_exp_range = [50, 100, 200, 400, 800, 1_600, 3_200, 6_400, 12_800, 25_600, 50_000, 100_000, 200_000, 400_00, 800_000, 1_600_000, 3_200_000, 6_250_000, 12_500_000, 25_000_000, 75_000_000, 120_000_000]

only analysing the histograms, refining the range again within the valid limits (sorting by total under- and over exposed)

refined_exp_range = [1_600, 2_100, 2_600, 3_200, 6_400, 8_300, 12_800, 15_700, 25_600]

-->these ldr frames could then be fused to hdr images

Maybe you already did that David, would you share it with me? It would be great, if one could access the histogramm metas for such a algorithmn, since that would ease the computational effort I guess:)

Kind regards Paul

[davidplowman]

Thanks for the info on the imx990. Have you used this sensor? It looks like a global shutter sensor (so large pixels, low resolution). Am I right in thinking it has a frequency response that includes quite a lot of IR, so that it's a bit like our "NoIR" modules (ones with the IR filter removed)?

On the other question, we're not doing anything like this at the moment. The image histograms, unfortunately, don't get passed out to you so I'd be inclined to capture low resolution images, and then create and analyse the histograms in software. Based on the results, you could then capture higher resolution images and decide which ones to fuse. But I'm only thinking out loud, I don't have any code like that!

[Bra1nsen]

Not used yet :)

Ok I will try capturing low resolutions images as histogram indicators, but the downscaling costs also time doenst it? I mean capturing 20 frames .... But lets try :)

[davidplowman]

If you can create your histograms from processed (not raw) images, then you can ask the imaging pipeline to downscale them, which is free.

[Bra1nsen]

Hej David, Iam currently working on the auto exposure settings for exposure timeseries.

I use currently:

config_highres = picam2.create_still_configuration(main={"size": (1014, 760), "format": "RGB888"},
                                               raw={"format": "SRGGB12", "size": (2028, 1520)})

so I concluded these settings should be valid:

{'bit_depth': 12,
  'crop_limits': (0, 0, 4056, 3040),
  'exposure_limits': (60, 127156999),
  'format': SRGGB12_CSI2P,
  'fps': 40.01,
  'size': (2028, 1520),
  'unpacked': 'SRGGB12'},

so working with should be valid: ?

rough_exp_range = [50, 100, 200, 400, 800, 1_600, 3_200, 6_400, 12_800, 25_600, 50_000, 100_000, 200_000, 400_00, 800_000, 1_600_000, 3_200_000, 6_250_000, 12_500_000, 25_000_000, 75_000_000, 120_000_000]

Because I thought using 40fps mode maximal exposure time would be 1/40th of a second, which is approximately 0.025 seconds or 25 milliseconds...

[davidplowman]

Hi, that all looks reasonable to me. I might be tempted to include a higher buffer_count in your config_highres because the default for create_still_coniguration will be just 1. This will cause you to drop frames, and your configuration isn't actually using that much memory so you can easily allocate more buffers.

With that change I'd expect you to be able to get up to 40fps. You may recall (I've mentioned it in the past) that there are some issues with the imx477 driver. If you let the framerate vary dynamically then it doesn't always change when we expect, so you can end up with lower exposures than you wanted.

[Bra1nsen]

Hey David - thanks for reply. Thats the current test script:

#!/usr/bin/python3

from picamera2 import Picamera2

def capture_multiple_exposures(picam2, exp_list, callback):

    def match_exp(metadata, indexed_list):
        err_factor = 0.05
        err_exp_offset = 50
        exp = metadata["ExposureTime"]
        gain = metadata["AnalogueGain"]
        for want in indexed_list:
            want_exp, _ = want
            if abs(gain - 1.0) < err_factor and abs(exp - want_exp) < want_exp * err_factor + err_exp_offset:
                return want
        return None

    indexed_list = [(exp, i) for i, exp in enumerate(exp_list)]
    while indexed_list:
        request = picam2.capture_request()
        match = match_exp(request.get_metadata(), indexed_list)
        if match is not None:
            indexed_list.remove(match)
            exp, i = match
            callback(i, exp, request)
        if indexed_list:
            exp, _ = indexed_list[0]
            picam2.set_controls({"ExposureTime": exp, "AnalogueGain": 1.0})
            indexed_list.append(indexed_list.pop(0))
        request.release()

def callback_func(i, wanted_exp, request):
    print(i, "wanted", wanted_exp, "got", request.get_metadata()["ExposureTime"])
    images.append((i, request.make_array("main"))) #"raw"

with Picamera2() as picam2:
    config = picam2.create_preview_configuration(main={"size": (1014, 760), "format": "RGB888"}, raw={"format": "SRGGB12", "size": (2028, 1520)}, buffer_count=3)
    picam2.configure(config)
    picam2.start()

    images = []
    exposure_list = [60, 120, 240, 480, 960, 1_600, 3_200, 6_400, 12_800, 25_600, 50_000, 100_000, 200_000, 400_00,
                   800_000, 1_600_000, 3_200_000, 6_250_000, 12_500_000, 25_000_000, 75_000_000, 120_000_000]
    capture_multiple_exposures(picam2, exposure_list, callback_func)

    picam2.stop()

So first Iam wondering how we get to 50ms, any idea? Since I thought we are in the 40fps mode:

Could you give the imx477 driver a new try, to find out, how to change the fps settings, so that it would be possible to run through all exposure times in the list without stopping the camera?

Kind regards Paul

[davidplowman]

That looks OK to me, at least as far as it goes. Is the problem that it's not producing the frames with the long exposure times?

By default create_preview_configuration won't let the frame duration get very long, so you should add something like

controls={'FrameDurationLimits': (50, 120000000)}

However, I'm always a bit nervous about running the imx477 in a variable framerate mode like this, I'm not sure it will behave quite right - but you can try it and see.

[Bra1nsen]

interesting - that works! But why? I thought that would not be possible :)

I will now try to add a new method analyze_histogram(), I will keep ypu up2date.

[Bra1nsen]

Ahhh David, since I want to stopp capturing after the first overexposed image frame arrives, it would be great to manipulate the capturing script to catch in order, at least for exposures > 1s.

Any idea how to do that?

[davidplowman]

Hmm, you're going to have to fiddle with the code a bit.

I would ask for a very low resolution image, so maybe add something like this to your config: lores={'size': (64, 48)}.

Then you can compute the mean Y value every time an image is captured, like this:

Y_mean = request.make_array("lores")[:48, :64].mean()

And if Y_mean is greater than some threshold, remove every entry from indexed_list where the exposure is greater than than the image just captured.

[Bra1nsen]

I modified the original script a bit to get the exposures in order, I thought we would get a more significant drop in the processing time performance, but it seems to work well for the first few frames. I thought it would make sense to increase tolerance for increasing exposure time but somehow iam not able to implement it. What iam doing wrong? Could you please take a look and check since I increased the err_exp_offset immensely with basically no effect

# !/usr/bin/python3

from picamera2 import Picamera2
import time
from collections import deque
import math

def capture_multiple_exposures(picam2, exp_list, callback):
    def match_exp(metadata, exp):
        err_factor_gain = 0.05
        err_factor = 0.05 
        err_exp_offset = 20 + (exp*0.1)
        actual_exp = metadata["ExposureTime"]
        gain = metadata["AnalogueGain"]
        return abs(gain - 1.0) < err_factor_gain and abs(actual_exp - exp) < exp * err_factor + err_exp_offset

    indexed_queue = deque([(exp, i) for i, exp in enumerate(exp_list)])
    while indexed_queue:
        exp, i = indexed_queue[0]
        picam2.set_controls({"ExposureTime": exp, "AnalogueGain": 1.0, "FrameDurationLimits": (50, 120000000)})
        request = picam2.capture_request()
        if match_exp(request.get_metadata(), exp):
            callback(i, exp, request)
            indexed_queue.popleft()  # Remove the matched exposure time from the front of the queue
        request.release()

def callback_func(i, wanted_exp, request):
    print(i, "wanted", wanted_exp, "got", request.get_metadata()["ExposureTime"])
    images.append((i, request.make_array("main")))  # "raw"

with Picamera2() as picam2:
    config = picam2.create_preview_configuration(main={"size": (1014, 760), "format": "RGB888"},
                                                 raw={"format": "SRGGB12", "size": (2028, 1520)}, buffer_count=5)
    picam2.configure(config)
    picam2.start()

    images = []
    exposure_list = [60, 120, 240, 480, 960, 1_600, 3_200, 6_400, 12_800, 25_600, 50_000, 100_000, 200_000, 400_000,
                     800_000, 1_600_000, 3_200_000, 6_250_000, 12_500_000, 25_000_000, 75_000_000, 120_000_000]

    start_time = time.time()  # Start timer

    capture_multiple_exposures(picam2, exposure_list, callback_func)

    end_time = time.time()  # Stop timer
    execution_time = end_time - start_time
    sum_of_list = sum(exposure_list)
    print("Sum of the list (Ideal Time):", sum_of_list)

    print("Execution Time:", execution_time, "seconds")

    picam2.stop()

for the analyse_histogram() method I considered to use something like an average pixel value as threshold. We could add that in callback_func()

# Generate histogram
hist, bin_edges = np.histogram(image_array.flatten(), bins=256, range=[0, 256])

and do smthing like this:

def analyze_histogram(histogram, min_avg_intensity=15, max_avg_intensity=240):
    """
    Analyze histogram and decide if we need to continue capturing.
    """

    # Calculate average intensity of the histogram
    avg_intensity = np.average(np.arange(256), weights=histogram)

    # Check for overexposure
    overexposed_pixels = histogram[max_avg_intensity:].sum()
    total_pixels = histogram.sum()
    if overexposed_pixels / total_pixels > overexposed_threshold:
        return False  # stop capturing if the image is overexposed

    # Check if exposure is within the valid range
    if min_avg_intensity <= avg_intensity <= max_avg_intensity:
        return True

    return False

But your approach sounds very interesting to me! Why do you suggest using lores and can you give more context about the implementation you had in mind David?

Kind regards Paul

[davidplowman]

The idea behind using the low resolution image to get the mean is that it will be more efficient. For a 64x48 low res image, that will be only 3072 pixel values to add up, whereas making a histogram from a larger image will be more expensive. But you should get much the same result.

What I think you need to do is edit the while indexed_list: loop in capture_multiple_exposures. Somewhere in there you need to calculate the mean pixel value, and then delete from indexed_list every exposure larger than the exposure of the image that gave you the saturated mean value. Does that make sense?

[Bra1nsen]

yes that makes totally sense, I implemented a full working auto exposure timeseries script and will test it now for the next couple days. But Iam still wondering how to increase the exposure time tolerance for longer shutter speeds, because Iam currently thinking that it would be easier for the system to catch on the wanted exposure time with higher tolerance. Or does that make no sense?

[davidplowman]

I'm not quite sure what you mean by "the exposure time tolerance for longer shutter speeds". The exposure time doesn't affect the number of frames you have to wait for an exposure change to happen. Though if the exposure times are long, then obviously each frame will take longer. But it's pretty much always "n frames" of delay, for some value of n, for any exposure change.

In your latest version of capture_multiple_exposures just above, I notice that you pop the exposure value only when it matches the frame that's just arrived. This is OK too, but it will give you a different kind of behaviour. Maybe print out the exposure of each frame that arrives (just after request = picam2.capture_request()), to see what happens.

In this version, I think you'll see: wait n frames for exposure index 0 to arrive, the wait n frames for exposure index 1 to arrive and so on.

In the original version, you should see: wait n frames for exposure index 0 to arrive, but then most of the others will arrive back-to-back (but occasionally some get skipped and we have to wait a bit longer for them).

[davidplowman]

Hi, just to say that I'm going to be away for ~10 days or so, so there will be a period of radio silence!

[Bra1nsen]

Thanks for telling me, I will work further on the script and share soon if you like. WIsh a few relaxing days David.

[Bra1nsen]

that the current script:

#!/usr/bin/python3

from picamera2 import Picamera2
import time
from collections import deque
import numpy as np

def capture_multiple_exposures(picam2, exp_list, callback, min_avg_intensity=15):

    def match_exp(metadata, exp):
        err_factor = 0.05
        err_exp_offset = 20
        actual_exp = metadata["ExposureTime"]
        gain = metadata["AnalogueGain"]
        return abs(gain - 1.0) < err_factor and abs(actual_exp - exp) < exp * err_factor + err_exp_offset

    min_exp_time = None
    max_exp_time = None

    indexed_queue = deque([(exp, i) for i, exp in enumerate(exp_list)])
    while indexed_queue:
        exp, i = indexed_queue[0]
        picam2.set_controls({"ExposureTime": exp, "AnalogueGain": 1.0, "FrameDurationLimits": (50, 120000000)})
        request = picam2.capture_request()
        if match_exp(request.get_metadata(), exp):
            histogram, image_array = callback(i, exp, request)

            # Calculate average intensity of the histogram
            avg_intensity = np.average(np.arange(256), weights=histogram)

            if avg_intensity > min_avg_intensity and min_exp_time is None:
                min_exp_time = exp

            if analyze_histogram(histogram):
                max_exp_time = exp
                break  # Stop capturing if the image is overexposed
            indexed_queue.popleft()  # Remove the matched exposure time from the front of the queue
        request.release()

    return min_exp_time, max_exp_time

def callback_func(i, wanted_exp, request):
    print(i, "wanted", wanted_exp, "got", request.get_metadata()["ExposureTime"])
    image_array = request.make_array("main")

    # Generate histogram
    hist, bin_edges = np.histogram(image_array.flatten(), bins=256, range=[0, 256])

    # Return histogram and image array
    return hist, image_array

def analyze_histogram(histogram, max_avg_intensity=240, overexposed_threshold=0.85):
    """
    Analyze histogram and decide if we need to continue capturing.
    """

    # Calculate average intensity of the histogram
    avg_intensity = np.average(np.arange(256), weights=histogram)
    print(f"avg_intensity: {avg_intensity}")

    # Check for overexposure
    overexposed_pixels = histogram[max_avg_intensity:].sum()
    total_pixels = histogram.sum()
    print(f"overexposed_pixels / total_pixels : {overexposed_pixels / total_pixels }")
    if overexposed_pixels / total_pixels > overexposed_threshold:
        return True  # stop capturing if the image is overexposed

    # Check if exposure is within the valid range
    if max_avg_intensity <= avg_intensity:
        return True

    return False

def fibonacci_bracketing(min_exp, max_exp, num_frames):
    fib_seq = [0, 1]
    for _ in range(num_frames - 1):
        fib_seq.append(fib_seq[-1] + fib_seq[-2])
    normalized_fib_seq = [x / fib_seq[-1] for x in fib_seq[1:]]
    return [min_exp + (max_exp - min_exp) * x for x in normalized_fib_seq]

def logarithmic_bracketing(min_exp, max_exp, num_frames):
    log_min = np.log(min_exp)
    log_max = np.log(max_exp)
    log_seq = np.linspace(log_min, log_max, num=num_frames)
    return np.exp(log_seq).tolist()

def generate_exposure_time_list(min_exp, max_exp, method='log'):
    """
    Generate a new exposure time list between min_exp and max_exp.
    """
    distance = max_exp - min_exp
    num_frames = max(4, min(8, int(distance / min_exp)))

    if method == 'fibonacci':
        return [round(x) for x in fibonacci_bracketing(min_exp, max_exp, num_frames)]
    elif method == 'log':
        return [round(x) for x in logarithmic_bracketing(min_exp, max_exp, num_frames)]
    else:
        raise ValueError("Invalid method. Choose 'fibonacci' or 'log'.")

with Picamera2() as picam2:
    config = picam2.create_preview_configuration(main={"size": (1014, 760), "format": "RGB888"}, raw={"format": "SRGGB12", "size": (2028, 1520)}, buffer_count=5)
    picam2.configure(config)
    picam2.start()

    exposure_list = [60, 120, 240, 480, 960, 1_600, 3_200, 4_800, 6_400, 9600, 12_800, 19_200, 25_600, 37_900, 50_000, 75_000, 100_000, 200_000, 400_000, 800_000, 1_600_000, 3_200_000, 6_250_000, 12_500_000, 25_000_000, 75_000_000, 120_000_000]

    start_time = time.time()  # Start timer

    min_exp_time, max_exp_time = capture_multiple_exposures(picam2, exposure_list, callback_func)

    print("Minimum Exposure Time:", min_exp_time)
    print("Maximum Exposure Time:", max_exp_time)

    refined_exp_list = generate_exposure_time_list(min_exp_time, max_exp_time, method='log')

    print("Refined Exposure Time List:", refined_exp_list)

    end_time = time.time()  # Stop timer
    execution_time = end_time - start_time
    sum_of_list = sum(exposure_list)
    print("Sum of the list (Ideal Time):", sum_of_list)

    print("Execution Time:", execution_time, "seconds")

    picam2.stop()

my concern was that the longer exposures are harder to hit if the maximal tolerance is like 20ms... Try the script David and give feedback for optimization :) !

[Bra1nsen]

Does it make sense to work with buffer_cont of 5 when we are expecting exposuretimes of more than 50 seconds for example?

[davidplowman]

I don't think that would make much difference, but try it and see. In general, a higher buffer count prevents frame drops, as follows:

• buffer_count = 1 - you're guaranteed to drop at least every other frame, and possibly worse.
• buffer_count = 2 - you'll probably still get frame drops if you're doing any processing at all on the frames. Could be as high as one frame in two, again.
• buffer_count = 3 - frames drops should be fairly rare, unless the system is generally busy doing other stuff (such as doing lots of computation, running video encoders etc.)
• After that, increasing buffer_count will tend to make frame drops rarer, but with diminishing returns. Always be aware that if the average time for which you hold a request exceeds the frame to frame interval, then you are of course doomed and will drop camera frames that arrive faster than you are processing them.

[Bra1nsen]

Thanks for the explanation. -Understood!

[Bra1nsen]

Hey David, any idea why its not possible to dynamically change the gain parameter? The script works fine if both gain values are equal, but if they are different it dont ...

    def capture_multiple_exposures(self, picam2, exp_list, callback, gain=1.0):
        timestamp = datetime.now().strftime("%Y%m%d%H%M%S")  # Define timestamp here

        def match_exp(metadata, indexed_list):
            err_factor = 0.05
            err_exp_offset = 50
            exp = metadata["ExposureTime"]
            gain = metadata["AnalogueGain"]
            for want in indexed_list:
                want_exp, _ = want
                if abs(gain - 1.0) < err_factor and abs(exp - want_exp) < want_exp * err_factor + err_exp_offset:
                    return want
            return None

        indexed_list = [(exp, i) for i, exp in enumerate(exp_list)]
        while indexed_list:
            request = picam2.capture_request()
            match = match_exp(request.get_metadata(), indexed_list)
            if match is not None:
                indexed_list.remove(match)
                exp, i = match
                callback(i, exp, request, timestamp)  # Pass timestamp to the callback
            if indexed_list:
                exp, _ = indexed_list[0]
                picam2.set_controls({"ExposureTime": exp, "AnalogueGain": gain, "FrameDurationLimits": (50, 120000000)})
                indexed_list.append(indexed_list.pop(0))
            request.release()

if __name__ == "__main__":
    # Define the city
    city = LocationInfo("Almería", "Spain")

    # Initialize the classes
    camera_name = "Cloud_Cam_Eagle"
    camera_name_raw = "Cloud_Cam_Eagle_raw"
    tuning = Picamera2.load_tuning_file("imx477.json")

    night_exp_list = [3_200_000, 6_250_000, 12_500_000, 25_000_000, 50_000_000, 75_000_000, 115_000_000]
    night_gain = 8

    day_exp_list = [60, 120, 240, 480, 960, 1_600, 3_200, 6_400]
    day_gain = 1

    capturer = Capturing(camera_name, camera_name_raw, city, tuning)

    while True:
        try:
            if is_daytime(city):
                exposure_list = day_exp_list
                gain = day_gain
                time_sleep = 180
            else:
                exposure_list = night_exp_list
                gain = night_gain
                time_sleep = 1800

            # Only start a new capturing process if the previous one has finished
            if not capturer.status:
                # Run the capturing with the refined exposure list
                capturer.run(exposure_list, gain)
                time.sleep(time_sleep)  # Sleep for a longer period if capturing process has started
            else:
                print("Camera busy, retry in 15s")
                time.sleep(15)  # Sleep for a shorter period and then retry if capturing process has not started
        except Exception as e:
            print(f"An error occurred: {e}")
            time.sleep(15)  # Sleep for a short period and then retry if an error occurred

[Bra1nsen]

Moreover I experience very strange behaviour:

if i set FrameDurationLimits within picam2.set_controls its not able to catch "raw" arrays..., commenting FrameDurationLimits out, they are saved.

[Bra1nsen]

from picamera2 import Picamera2
import numpy as np
import os
from datetime import datetime
import atexit

class Capturing:
    def __init__(self, camera_name):
        self.camera_name = camera_name
        self.images_raw = {}
        self.metadata_raw = []

    def create_save_path(self):
        base_path = f"/media/dlr/usbstick/Dataset/{self.camera_name}"
        os.makedirs(base_path, exist_ok=True)
        return base_path

    def capture_multiple_exposures(self, picam2, exp_list, callback, gain=1.0):
        timestamp = datetime.now().strftime("%Y%m%d%H%M%S")

        def match_exp(metadata, indexed_list):
            err_factor = 0.05
            err_exp_offset = 50
            exp = metadata["ExposureTime"]
            gain = metadata["AnalogueGain"]
            for want in indexed_list:
                want_exp, _ = want
                if abs(gain - 1.0) < err_factor and abs(exp - want_exp) < want_exp * err_factor + err_exp_offset:
                    return want
            return None

        indexed_list = [(exp, i) for i, exp in enumerate(exp_list)]
        while indexed_list:
            request = picam2.capture_request()
            match = match_exp(request.get_metadata(), indexed_list)
            if match is not None:
                indexed_list.remove(match)
                exp, i = match
                callback(i, exp, request, timestamp)  
            if indexed_list:
                exp, _ = indexed_list[0]
                picam2.set_controls({"ExposureTime": exp, "AnalogueGain": gain, "FrameDurationLimits": (50, 120000000)})
                indexed_list.append(indexed_list.pop(0))
            request.release()

    def callback_func(self, i, wanted_exp, request, timestamp):
        image_raw = request.make_array("raw")
        self.images_raw[f'{timestamp}_{wanted_exp}'] = image_raw
        self.metadata_raw.append((f'{timestamp}_{wanted_exp}', request.get_metadata()))

        if len(self.images_raw) == len(self.exposure_list):
            dtypes = [('timestamp_exp', 'U20')] + [(key.replace(' ', '_'), 'U100') for key in request.get_metadata().keys()]
            metadata_array = np.zeros(len(self.metadata_raw), dtype=dtypes)

            for i, (timestamp_exp, meta) in enumerate(self.metadata_raw):
                metadata_array[i]['timestamp_exp'] = timestamp_exp
                for key in meta.keys():
                    metadata_array[i][key.replace(' ', '_')] = str(meta[key])

            save_path = self.create_save_path()
            np.savez(f'{save_path}/{timestamp}.npz', **self.images_raw, **{f'{timestamp}_metadata': metadata_array})

            self.images_raw = {}
            self.metadata_raw = []

    def run(self, exposure_list, gain):
        self.status = True
        self.exposure_list = exposure_list  

        try:
            with Picamera2() as picam2:
                self.picam2 = picam2
                atexit.register(self.picam2.stop)

                config = picam2.create_preview_configuration(main={"size": (1014, 760), "format": "RGB888"},
                                                             raw={"format": "SRGGB12", "size": (2028, 1520)},
                                                             buffer_count=3)
                picam2.configure(config)
                picam2.start()

                self.capture_multiple_exposures(picam2, exposure_list, self.callback_func, gain)
        except Exception as e:
            print(f"An error occurred during capture: {e}")
        finally:
            if self.picam2:
                self.picam2.stop()
            self.status = False

[davidplowman]

Firstly, it should be possible to change the gain dynamically. Can you give some examples of the (exposure, gain) pairs that you are setting and show what you get back? You should get the values you asked for, synchronised on the same frame.

As I've said before, there are some problems with the imx477 when the framerate (frame duration) is variable. The frame duration doesn't always increase as we expect, and the system would apply digital gain to hide the problem. So it's often worth testing the behaviour with a fixed framerate.

It might be worth adding some debug to the code so that you can share what it's doing. Maybe print out the controls you send in picam2.set_controls every time, and then print out the exposure/gain on every request that comes back. That might make it easier to see what's going on. Thanks!

[Bra1nsen]

ah the gain issue was caused here:

        def match_exp(metadata, indexed_list):
            err_factor = 0.05
            err_exp_offset = 50
            exp = metadata["ExposureTime"]
            gain = metadata["AnalogueGain"]
            for want in indexed_list:
                want_exp, _ = want
                if abs(gain - 1.0) < err_factor and abs(exp - want_exp) < want_exp * err_factor + err_exp_offset:
                    return want
            return None

so just changed it to

if abs(metadata_gain - gain) < err_factor and abs(exp - want_exp) < want_exp * err_factor + err_exp_offset:

ok I will try fix FrameDuration..Just wondering why it worked for isp but not for raw :)

[davidplowman]

I assume metadata_gain is tha "AnalogueGain" from the metadata, and gain is a value that you pass in (and which you want)? That looks OK to me.

[Bra1nsen]

Please David you have to test that:

#!/usr/bin/python3

import piexif
import json
import os
import time
import pytz
from datetime import timezone, timedelta, datetime
from picamera2 import *
from astral.sun import sun
from astral import LocationInfo
from PIL import Image

tuning = Picamera2.load_tuning_file("imx477.json")

city = LocationInfo("Almería", "Spain")

exposures = [50, 150, 250, 400, 650, 1050, 1700, 2750, 5250]
capture_interval = 180  # 10 minutes in seconds
camera_name = "Cloud_Cam_Eagle"

save_path = "/home/dlr/"

def save_image_with_custom_exif(image_path, metadata):
    # Load the Exif data
    exif_dict = piexif.load(image_path)

    # Add custom Exif tags using the Exiftool format
    exif_dict["0th"][piexif.ImageIFD.Make] = "Raspberry Pi"
    exif_dict["0th"][piexif.ImageIFD.Model] = "Raspberry Pi High Quality Camera"
    exif_dict["Exif"][piexif.ExifIFD.DateTimeOriginal] = datetime.now().strftime("%Y:%m:%d %H:%M:%S")

    # Add custom metadata to the Exif UserComment
    exif_dict["Exif"][piexif.ExifIFD.UserComment] = json.dumps(metadata).encode('utf-8')

    # Dump the modified Exif data
    exif_bytes = piexif.dump(exif_dict)

    # Save the image with the new Exif data
    piexif.insert(exif_bytes, image_path)

def exposure_bracket():
    if True:
        with picamera2.Picamera2(tuning=tuning) as camera:
            config = camera.create_still_configuration(main={"size": (1014, 760), "format": "RGB888"},
                                                       raw={"format": "SRGGB12", "size": (2028, 1520)})
            camera.configure(config)

            if True:
                start = time.monotonic()
                for exp in exposures:
                    camera.set_controls({"ExposureTime": exp, "AnalogueGain": 1, "ColourGains": (1.0, 1.0), "FrameDurationLimits": (50, 120_000_000)})
                    camera.start()

                    timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
                    file_name = f"{timestamp}_{exp}.jpg"
                    full_path = os.path.join(save_path, file_name)
                    metadata = camera.capture_file(full_path)
                    lux = metadata["Lux"]
                    print(lux)
                    save_image_with_custom_exif(full_path, metadata)

                    # Save the main array image
                    image_array = camera.capture_request().make_array("main")
                    main_image_path = os.path.join(save_path, f"{timestamp}_{exp}_main.jpg")
                    Image.fromarray(image_array).save(main_image_path)

                    end = time.monotonic()
                    execution_time = end - start
                    print(f"Exposure Time: {exp} us")
                    print(f"Execution Time: {execution_time:.2f} seconds")
                    print(f"Metadata: {metadata}")
                    print(f"Saved to: {full_path}\n")

                    camera.stop()

if __name__ == "__main__":
    exposure_bracket()

I expected to get the same image, but they are different!! Please explain!

[davidplowman]

I can't see an obvious reason in the code. You might have to investigate a bit. Some things I would check:

• If you capture the first image just as an array, does it match the second image_array, or are there differences (like in your pictures)?
• If they are different, does comparing the metadata for the two image give any clues as to what it might be?

[Bra1nsen]

Metadata capture_file: {'ScalerCrop': (0, 0, 4056, 3040), 'SensorBlackLevels': (4096, 4096, 4096, 4096), 'ColourTemperature': 5932, 'DigitalGain': 1.0, 'ColourGains': (1.0, 1.0), 'AeLocked': False, 'Lux': 400.0, 'FrameDuration': 24994, 'ColourCorrectionMatrix': (1.9874064922332764, -0.8473411798477173, -0.1400553286075592, -0.2653316855430603, 1.7090944051742554, -0.44377270340919495, -0.096389040350914, -0.47202178835868835, 1.568410873413086), 'SensorTimestamp': 695233200264000, 'AnalogueGain': 1.0, 'SensorTemperature': 0.0, 'ExposureTime': 60}

Metadata capture_request ("main" array): {'FocusFoM': 433, 'ExposureTime': 60, 'SensorTemperature': 0.0, 'SensorTimestamp': 695233300247000, 'ScalerCrop': (0, 0, 4056, 3040), 'ColourTemperature': 5932, 'DigitalGain': 1.0, 'AnalogueGain': 1.0, 'ColourGains': (1.0, 1.0), 'AeLocked': False, 'SensorBlackLevels': (4096, 4096, 4096, 4096), 'Lux': 34410.41796875, 'FrameDuration': 24994, 'ColourCorrectionMatrix': (1.9874064922332764, -0.8473411798477173, -0.1400553286075592, -0.2653316855430603, 1.7090944051742554, -0.44377270340919495, -0.096389040350914, -0.47202178835868835, 1.568410873413086)}

[Bra1nsen]

thats the code i use for printing, interesting that they have a different order in terms of printing metadata parameter...

def exposure_bracket():
    if True:
        with picamera2.Picamera2(tuning=tuning) as camera:
            config = camera.create_still_configuration(main={"size": (1014, 760), "format": "RGB888"},
                                                       raw={"format": "SRGGB12", "size": (2028, 1520)})
            camera.configure(config)

            if True:
                start = time.monotonic()
                for exp in exposures:
                    camera.set_controls({"ExposureTime": exp, "AnalogueGain": 1, "ColourGains": (1.0, 1.0), "FrameDurationLimits": (50, 120_000_000)})
                    camera.start()

                    timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
                    file_name = f"{timestamp}_{exp}.jpg"
                    full_path = os.path.join(save_path, file_name)
                    metadata = camera.capture_file(full_path)
                    lux = metadata["Lux"]
                    print(lux)
                    save_image_with_custom_exif(full_path, metadata)

                    # Save the main array image
                    capture_request = camera.capture_request()
                    image_array = capture_request.make_array("main")
                    main_image_path = os.path.join(save_path, f"{timestamp}_{exp}_main.jpg")
                    Image.fromarray(image_array).save(main_image_path)

                    # Print the metadata of the main image
                    main_metadata = capture_request.get_metadata()
                    print(f"Main Metadata -main-: {main_metadata}")

                    end = time.monotonic()
                    execution_time = end - start
                    print(f"Exposure Time: {exp} us")
                    print(f"Execution Time: {execution_time:.2f} seconds")
                    print(f"Metadata capture_file: {metadata}")
                    print(f"Saved to: {full_path}\n")

                    camera.stop()

[davidplowman]

I don't think I see any real differences in the metadata. What if you capture both as arrays and subtract them, are there significant differences?

[Bra1nsen]

If you capture the first image just as an array, does it match the second image_array, or are there differences (like in your pictures)?

like this?

def exposure_bracket():
    if True:
        with picamera2.Picamera2(tuning=tuning) as camera:
            config = camera.create_still_configuration(main={"size": (1014, 760), "format": "RGB888"},
                                                       raw={"format": "SRGGB12", "size": (2028, 1520)})
            camera.configure(config)

            if True:
                start = time.monotonic()
                for exp in exposures:
                    camera.set_controls({"ExposureTime": exp, "AnalogueGain": 1, "ColourGains": (1.0, 1.0), "FrameDurationLimits": (50, 120_000_000)})
                    camera.start()

                    timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
                    file_name = f"{timestamp}_{exp}.jpg"
                    full_path = os.path.join(save_path, file_name)
                    metadata = camera.capture_file(full_path)
                    lux = metadata["Lux"]
                    print(lux)
                    save_image_with_custom_exif(full_path, metadata)

                    # Save the main array image
                    capture_request = camera.capture_request()
                    image_array = capture_request.make_array("main")
                    main_image_path = os.path.join(save_path, f"{timestamp}_{exp}_main.jpg")
                    Image.fromarray(image_array).save(main_image_path)

                    # Save the image array as a separate JPG file
                    array_image_path = os.path.join(save_path, f"{timestamp}_{exp}_array.jpg")
                    Image.fromarray(image_array).save(array_image_path)

                    # Print the metadata of the main image
                    main_metadata = capture_request.get_metadata()
                    print(f"Main Metadata -main-: {main_metadata}")

                    end = time.monotonic()
                    execution_time = end - start
                    print(f"Exposure Time: {exp} us")
                    print(f"Execution Time: {execution_time:.2f} seconds")
                    print(f"Metadata capture_file: {metadata}")
                    print(f"Saved to: {full_path}\n")

                    camera.stop()

[Bra1nsen]

How is that possible?

[davidplowman]

Ah, do yout have a red/blue swap when you convert it into an Image? I'm sure that the Requset.make_image function does this, there are probably some parameters to control this.

[Bra1nsen]

Ok thanks (so I guess the "normal" jpg, the image below, is the correct one), would you please show which settings to use? And wouldnt it be nice, if its default? Kind regards

[davidplowman]

I think maybe here. The mode parameter looks interesting. I expect the image from capture_file is probably the correct one.

[Bra1nsen]

is that normal that one has to convert from bgr to rgb first? now it looks fine.

              # Save the main array image
                    capture_request = camera.capture_request()
                    image_array = capture_request.make_array("main")
                    # Convert BGR to RGB
                    image_array = image_array[:, :, ::-1]

[davidplowman]

The trouble with OpenCV in Python is that you can't tell from a numpy array which way round R and B are and, as far as I can tell, all these different packages make random choices. I normally just accept that I swap R and B round if they look wrong. Mostly you can do it by changing the format you request from the camera so you can generally avoid extra software conversions.

[Bra1nsen]

even tho they look quite equal, they are not...

which is probably for the most people no big deal, but for me, unfortunaltely, it is, since iam interested in estimating solar irradiance based on pixel intentsities.... Any idea how to get accurate results with make.array("main")?

[davidplowman]

So you're setting the same colour gains, exposure and gain. We should rule out any other camera algorithms first, so it might be worth disabling the lens shading (in the tuning flle, find "rpi.alsc" and change it to something like "x.rpi.alsc"). Also disable any adaptive contrast manipulation. Find the "rpi.contrast" section in the tuning file and change "ce_enable": 1 to "ce_enable": 0. Then let's see how the numbers look.

[Bra1nsen]

It was already 0. I changed it to 1 captured again and compared values, no improving..

ce_eanle: 0

ce_enable: 1

Here is my complete tuning file:

https://drive.google.com/file/d/1Fqd73_VmiuLqZfsjXMGUtR9vgkZd6PmD/view?usp=sharing

[Bra1nsen]

I also changed the name "rpi.alsc" to "x.rpi.alsc" and set ce_enable: 0 .. no success:

[davidplowman]

Can't think of an obvious reason. Might be worth saving a "difference" image between the two, and seeing if there's any kind of pattern. Also, if you drop the first frame from the camera completely, so that you capture the 2nd and 3rd, do those match better?

[Bra1nsen]

Currently Iam only able to compare the frist frame, because after using capture_request(), the brackting stops. Could you please reproduce the error:

# !/usr/bin/python3

import piexif
import json
import os
import time
import pytz
from datetime import timezone, timedelta, datetime
from picamera2 import *
from astral.sun import sun
from astral import LocationInfo
from PIL import Image

tuning = Picamera2.load_tuning_file("imx477.json")

city = LocationInfo("Almería", "Spain")

exposures = [50, 150, 250, 400, 650, 1050, 1700, 2750, 5250]
capture_interval = 180  # 10 minutes in seconds
camera_name = "Cloud_Cam_Eagle"

save_path = "/home/pi/test"

def save_image_with_custom_exif(image_path, metadata):
    # Load the Exif data
    exif_dict = piexif.load(image_path)

    # Add custom Exif tags using the Exiftool format
    exif_dict["0th"][piexif.ImageIFD.Make] = "Raspberry Pi"
    exif_dict["0th"][piexif.ImageIFD.Model] = "Raspberry Pi High Quality Camera"
    exif_dict["Exif"][piexif.ExifIFD.DateTimeOriginal] = datetime.now().strftime("%Y:%m:%d %H:%M:%S")

    # Add custom metadata to the Exif UserComment
    exif_dict["Exif"][piexif.ExifIFD.UserComment] = json.dumps(metadata).encode('utf-8')

    # Dump the modified Exif data
    exif_bytes = piexif.dump(exif_dict)

    # Save the image with the new Exif data
    piexif.insert(exif_bytes, image_path)

def exposure_bracket():
    if True:
        with picamera2.Picamera2(tuning=tuning) as camera:
            config = camera.create_still_configuration(main={"size": (1014, 760), "format": "RGB888"},
                                                       raw={"format": "SRGGB12", "size": (2028, 1520)})
            camera.configure(config)

            if True:
                for exp in exposures:
                    camera.set_controls({"ExposureTime": exp, "AnalogueGain": 1, "ColourGains": (1.0, 1.0), "FrameDurationLimits": (50, 120_000_000)})
                    print(f"exp: {exp}")
                    camera.start()

                    timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
                    file_name = f"{timestamp}_{exp}.jpg"
                    full_path = os.path.join(save_path, file_name)
                    metadata = camera.capture_file(full_path)
                    save_image_with_custom_exif(full_path, metadata)

                    # Save the main array image
                    capture_request = camera.capture_request()
                    image_array = capture_request.make_array("main")
                    #image_array = image_array[:, :, ::-1]
                    main_image_path = os.path.join(save_path, f"{timestamp}_{exp}_main.jpg")
                    pil_img = Image.fromarray(image_array)
                    pil_img.save(main_image_path)

                    camera.stop()

if __name__ == "__main__":
    exposure_bracket()

afterwards I load the images like that:

from PIL import Image
import numpy as np

# List of image paths
image_paths = [
    "20230619162506_50.jpg",
    "20230619162506_50_main.jpg"
]

for image_path in image_paths:
    # Load the image
    image = Image.open(image_path)

    # Convert the image to a numpy array
    image_array = np.array(image)

    # Calculate the sum of each channel
    sum_r = np.sum(image_array[:, :, 0])
    sum_g = np.sum(image_array[:, :, 1])
    sum_b = np.sum(image_array[:, :, 2])

    # Print the results
    print(f"Image: {image_path}")
    print(f"Sum of R channel: {sum_r}")
    print(f"Sum of G channel: {sum_g}")
    print(f"Sum of B channel: {sum_b}\n")

Kind regards