[OTHER] Colour Correction

[Bra1nsen]

Iam still interested in estimating Solar Irradiance with a Raspberry All Sky Imager.

I would like to know, if the color calibration was already evaluated? How precise is the sRGB colour spaced projected?

I know the solar spectral distribution:

The following Diagram shows the sRGB field:

Its possible to calculate the wavelengths corrections for the sRGB colorspace from CIE 1931 Space:

So when we capture images with the raspberry hq cam, the images should be sRGB or? Since then it should be possible to estimate the irradiance for the visible spectrum with a physical model.

Iam just interested in learning more about the color calibration described on raspberry pi tuning guide, and if its already evaluated.

[davidplowman]

I wonder if it's just that your images are dark and you're getting some severe JPEG quantisation. Try replacing .jpg by .png (you might need format "BGR888", I wouldn't trust the reds and blues not to swap round randomly again).

[Bra1nsen]

The current pipeline saves raw and jpg within one capturing process..., the "raws" will still be SRGGB12 right?

interesting, i changed the mode to BGR888, and compared results btw jpg and png.

[Bra1nsen]

why is JPEG quantisation different btw both jpg savings?

import numpy as np
from PIL import Image
import os
from datetime import datetime

def exposure_bracket():
    with picamera2.Picamera2(tuning=tuning) as camera:
        config = camera.create_still_configuration(main={"size": (1014, 760), "format": "BGR888"},
                                                   raw={"format": "SRGGB12", "size": (2028, 1520)})
        camera.configure(config)

        for exp in exposures:
            camera.set_controls({"ExposureTime": exp, "AnalogueGain": 1, "ColourGains": (1.0, 1.0), "FrameDurationLimits": (50, 120_000_000)})
            print(f"exp: {exp}")
            camera.start()

            timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
            file_name = f"{timestamp}_{exp}.jpg"
            full_path = os.path.join(save_path, file_name)
            camera.capture_file(full_path)

            # Save the main array image
            capture_request = camera.capture_request()
            image_array = capture_request.make_array("main")
            main_image_path = os.path.join(save_path, f"{timestamp}_{exp}_main.jpg")
            pil_img = Image.fromarray(image_array)
            pil_img.save(main_image_path)

            camera.stop()

if __name__ == "__main__":
    exposure_bracket()

[Bra1nsen]

currently one frame (random) is captured not using tuning file I guess. Sometimes its the 120, sometimes the 240, sometimes the 960 ... here an example:

any idea how I can pretend this from occuring?

def capture_multiple_exposures(self, picam2, exp_list, frame_duration, callback, gain=1.0):
        timestamp = datetime.now().strftime("%Y%m%d%H%M%S")  # Define timestamp here

        def match_exp(metadata, indexed_queue):
            err_factor = 0.05

            err_gain = 0.1
            err_exp_offset = 50
            exp = metadata["ExposureTime"]
            metadata_gain = metadata["AnalogueGain"]
            for want in list(indexed_queue):  # Convert deque to list for iteration
                want_exp, _ = want
                if abs(metadata_gain - gain) < err_gain and abs(
                        exp - want_exp) < want_exp * err_factor + err_exp_offset:
                    return want
            return None

        indexed_queue = deque([(exp, i) for i, exp in enumerate(exp_list)])
        while indexed_queue:
            request = picam2.capture_request()
            match = match_exp(request.get_metadata(), indexed_queue)
            if match is not None:
                indexed_queue.remove(match)
                exp, i = match
                callback(i, exp, request, timestamp, capture_time)  # Pass capture_time to the callback
            if indexed_queue:
                exp, _ = indexed_queue[0]
                picam2.set_controls({"ExposureTime": exp, "AnalogueGain": gain, "ColourGains": (1.0, 1.0), "FrameDurationLimits": frame_duration})
                indexed_queue.append(indexed_queue.popleft())  # Use deque.popleft() and deque.append()
            request.release()

    def callback_func(self, i, wanted_exp, request, timestamp, capture_time):
        image_array = request.make_array("main")

        # Save the image using PIL
        file_name = f"{timestamp}_{wanted_exp}.png"
        full_path = os.path.join(self.create_save_path(self.camera_name), file_name)
        image = Image.fromarray(image_array)
        image.save(full_path)

        # Now that the image is saved, you can modify the Exif data
        metadata = request.get_metadata()
        lux = metadata["Lux"]
        # print(f"estimated lux: {lux})
        self.add_exif_data(full_path, metadata)

    def run(self, exposure_list, frame_duration, gain):
        self.status = True
        self.exposure_list = exposure_list

        # Check for available memory before starting the capturing process
        self.create_save_path(self.camera_name)
        self.create_save_path(self.camera_name_raw)

        try:
            with Picamera2() as picam2:
                self.picam2 = picam2
                atexit.register(self.picam2.stop)

                config = picam2.create_preview_configuration(main={"size": (1014, 760), "format": "BGR888"},
                                                             raw={"format": "SRGGB12", "size": (2028, 1520)},
                                                             buffer_count=5)
                picam2.configure(config)
                picam2.start()

                self.capture_multiple_exposures(picam2, exposure_list, frame_duration, self.callback_func, gain)
        except Exception as e:
            logging.error(f"An error occurred during capture: {e}", exc_info=True)
        finally:
            if self.picam2:
                self.picam2.stop()
            self.status = False

Thanks..

[davidplowman]

I wonder if one of the exposures occasionally matches by "luck" before the colour gains (and everything else) have been applied. Maybe try doing

picam2.set_controls({'ColourGains': (1.0, 1.0)})

after picam2.configure(...) but before picam2.start().

[Bra1nsen]

Iam not sure, I set_controls within the capture_multiple_exposures() method... Why shoud it make a difference tho?

[Bra1nsen]

Any other way to check if they match by "luck", or how to pretend that from occuring?

[davidplowman]

When you use set_controls, it normally takes "several frames" for the controls to be applied (which is why this whole multiple exposures thing is difficult). But if you do it before starting the camera, then it's guaranteed to be applied for the very first frame.

When you start monitoring the first frames for the exposures you want, I'm suspicious that one of those first frames matches one of your exposures "by luck", even though the first set_controls, with the colour gains, has not taken effect yet. The easiest fix would be just to set those colour gains before the camera starts and, in theory, this problem cannot then happen.

[Bra1nsen]

Ok i set it twice and it seems to work. Maybe its also worth to set frame duration limits at that point..

[davidplowman]

Setting the colour gains once (before start()) should be enough. In fact, anything that doesn't change is probably best set before you start the camera, because once it's running you get these annoying delays for things to happen.

[Bra1nsen]

Iam interested in your opionion, because iam not sure what to think of it. We are capturing png and raw RGGB arrays within one run. heres an example result:

and then i was wondering why the raw arrays contain values beyond 4095, since I thougth they are 12bit...

Any Idea why values beyond 4095, since I thougth they are 12bit ? I just take the arrays resize them and sort them...

 # Save raw image
        image_raw = request.make_array("raw")
        # print(f"shape raw arraw: {image_raw.shape}")
        resized_image_raw = cv2.resize(image_raw, (
            image_raw.shape[1] // 2, image_raw.shape[0] // 2))  # Reduces the resolution by 50

        self.images_raw[f'{timestamp}_{wanted_exp}'] = resized_image_raw
        self.metadata_raw.append((f'{timestamp}_{wanted_exp}', metadata))

        if len(self.images_raw) == len(self.exposure_list):  # Once we have all exposures for the series
            dtypes = [('timestamp_exp', 'U20')] + [(key.replace(' ', '_'), 'U100') for key in metadata.keys()]
            metadata_array = np.zeros(len(self.metadata_raw), dtype=dtypes)

            # Sort the images and metadata by the keys (which include the exposure time)
            sorted_keys = natsorted(self.images_raw.keys())
            images_raw_sorted = {key: self.images_raw[key] for key in sorted_keys}
            metadata_raw_sorted = sorted(self.metadata_raw, key=lambda x: x[0])

            for i, (timestamp_exp, meta) in enumerate(metadata_raw_sorted):
                metadata_array[i]['timestamp_exp'] = timestamp_exp
                for key in meta.keys():
                    if key.replace(' ', '_') in metadata_array.dtype.names:
                        metadata_array[i][key.replace(' ', '_')] = str(meta[key])
                    else:
                        logging.warning(f"Key '{key}' not found in metadata_array dtype names")

            save_path = self.create_save_path(self.camera_name_raw)
            try:
                np.savez(f'{save_path}/{timestamp}.npz', **images_raw_sorted,
                         **{f'{timestamp}_metadata': metadata_array})
            except Exception as e:
                logging.error(f"An error occurred while saving .npz file: {e}", exc_info=True)

            # Reset for next series
            self.images_raw = {}
            self.metadata_raw = []

Somehow the raw arrays look pretty interesting, not sure if its normal that raw arrays look so different from isp frames.. Kind regards

[Bra1nsen]

Max Value of 34695 makes no sense to me, any idea David?

[davidplowman]

Are you reading back DNG files? In those I believe the values are shifted up to 16-bits.

[Bra1nsen]

No I use

image_raw = request.make_array("raw")

and save the rggb array as numpy file

[davidplowman]

You're sure you're using the unpacked raw format, and that we're interpreting the bytes in the correct order? Maybe have a look at the raw image just as bytes first, and make sure you see the alternating pattern of 8 low bits in the first byte, and then the 4 high bits in the next byte. Also check the values before and after saving, just to be sure there's nothing strange happening there.

[Bra1nsen]

config = picam2.create_preview_configuration(main={"size": size_main, "format": "BGR888"},
                                                             raw={"format": "SRGGB12", "size": size_raw},
                                                             buffer_count=5)

should be unpacked right?

Does that look correct?

[davidplowman]

Yes, that's unpacked and should be fine. I would check that the data look sensible straight after you've created it (with make_array) first. Presumably that must be right...? In which case it has to be down to the save/load process.

[Bra1nsen]

i did:

        # Save raw image
        image_raw = request.make_array("raw")
        print(image_raw.dtype)

and it printed uint8 ...

[Bra1nsen]

I just saved the "raw" arrays, how can I read them correctly?

[davidplowman]

I would have a look at them before you save them, to be 100% certain what you're starting with!

[Bra1nsen]

I dont understand that to be honest. I use capture_multiple_exposures()using request = picam2.capture_request() and within callback_func() i userequest.make_array("raw") and request.make_array("main") to save both.

    def capture_multiple_exposures(self, picam2, exp_list, frame_duration, callback, gain=1.0):
        timestamp = datetime.now().strftime("%Y%m%d%H%M%S")  # Define timestamp here

        def match_exp(metadata, indexed_queue):
            err_factor = 0.05

            err_gain = 0.1
            err_exp_offset = 50
            exp = metadata["ExposureTime"]
            metadata_gain = metadata["AnalogueGain"]
            for want in list(indexed_queue):  # Convert deque to list for iteration
                want_exp, _ = want
                if abs(metadata_gain - gain) < err_gain and abs(
                        exp - want_exp) < want_exp * err_factor + err_exp_offset:
                    return want
            return None

        indexed_queue = deque([(exp, i) for i, exp in enumerate(exp_list)])
        while indexed_queue:
            start_time = time.time()  # Record the start time
            request = picam2.capture_request()
            end_time = time.time()  # Record the end time
            capture_time = end_time - start_time  # Calculate the capture time
            match = match_exp(request.get_metadata(), indexed_queue)
            if match is not None:
                indexed_queue.remove(match)
                exp, i = match
                callback(i, exp, request, timestamp, capture_time)  # Pass capture_time to the callback
            if indexed_queue:
                exp, _ = indexed_queue[0]
                picam2.set_controls({"ExposureTime": exp, "AnalogueGain": gain, "ColourGains": (1.0, 1.0),
                                     "FrameDurationLimits": frame_duration})
                indexed_queue.append(indexed_queue.popleft())  # Use deque.popleft() and deque.append()
            request.release()

    def callback_func(self, i, wanted_exp, request, timestamp, capture_time):

        image_array = request.make_array("main")

        image_raw = request.make_array("raw")
        print(image_raw.dtype)

[davidplowman]

Well, I don't really know why you see these values larger than 4095 so we have to work by a process of elimination. Do you see values larger than 4095 in image_raw (when viewed as 16-bit numbers)? I'm guessing that you won't, so the problem would have to be in the saving/loading procedure. But let's check that first, and then we can move on to look at the next thing.

[Bra1nsen]

Its the cv2 resize.

        # Save raw image
        image_raw = request.make_array("raw")
        print(image_raw.dtype)
        max_valu = np.max(image_raw.view(np.uint16))
        print('Max value no resize:', max_valu)

        resized_image_raw = cv2.resize(image_raw, (
            image_raw.shape[1] // 2, image_raw.shape[0] // 2))  # Reduces the resolution by 50

        print(resized_image_raw.dtype)
        # Print the maximum value.
        max_value = np.max(resized_image_raw.view(np.uint16))
        print('Max value:', max_value)

Max value no resize: 4095
Max value after resize: 34695

Iam sorry, I did not know it would change the pixel values. How would you resize the raw arrays)

[Bra1nsen]

ahh okay using:

        resized_image_raw = cv2.resize(image_raw.view(np.uint16), (
            image_raw.shape[1] // 2, image_raw.shape[0] // 2)) 

worked. thanks for helping and sorry for my blindness :/

[Bra1nsen]

Hello David,

we are considering building a small series of all sky imagers (~25p) and it would be decent to use Picamera2 Python library for capturing algorithm.

But the Raspberry Pi 4B is pretty rarecurrently and a bit overstacked. Basically we just need a single board computer (optimal with gpu power) with a csi, gpio and rj45 ethernet port...

Moreover I was asking my self if its possible to use Picamera2 with 3rd partie hardware.

e.g.: Odroid-XU4 Asus Tinker Board Pine64 RockPro64 Banana Pi M3 NanoPi M4 Jetson Nano

I also thought about the cm400, but seems tough to get csi connection. Thanks.

Kind regards Paul

[davidplowman]

Picamera2 will only have a chance of working if the platform in question supports libcamera. There is some support for one or maybe two different RockChip ISPs, and I think also an Intel one. But you might still find that the support there is different. For example, different formats, different numbers or types of stream, different allowable scaling. You may also find controls are handled differently, it's quite likely other platforms won't pipeline the exposure updates in the way that the Pi does.

Basically I think you'd have to find a board with libcamera support and buy one for prototyping and feasibility testing. I wouldn't rate the chances of success as being terribly high (though that's just my opinion), and I also wouldn't be able to help - my employer certainly wouldn't approve of me getting things to work on other platforms!!

Another thing to consider might be a Pi 3. CPU processing will be slower but I would expect you'd mostly get things to run at similar framerates. A CM4 (Compute Module 4) should work pretty much identically to a Pi 4, and in both cases there'd be advice available on the forums or here. A Pi 400 (the Pi-in-a-keyboard) is a no-go, because there's no CSI-2 connector. Getting one of these devices and doing some prototyping doesn't sound like a bad idea.

On the other hand, we're being told that Pi 4 availability is supposed to be improving, but it may well be that it's still happening quite slowly - I'm not really up to date on that.

[Bra1nsen]

Thanks for the hint with the Raspberry Pi CM4 Module. I found a very interesting carrier board, but damn very expensive and not available. Currently considering to design a carrier board by myself:

Moreover I have a short question about the tuning file you provided me long time ago. If you remember, we tried to make the pixel intensities as raw as possible.

So we set Color Gains to (1.0,1.0) as well as AnalogGain to 1.0.

Tuning File: https://drive.google.com/file/d/1p5URc-tW4hgrWwkI0AFTh_W_DPN0rfYh/view?usp=sharing

When checking the metadata:

Colour Correction Matrix : 
(1.9874064922332764, -0.8473411798477173, -0.1400553286075592, -0.2653316855430603, 1.7090944051742554, -0.44377270340919495, -0.096389040350914, -0.47202178835868835, 1.568410873413086)

Colour Temperature : 5932

So we always have a constant color temperature and color correction matrix. Why 5932? And would it even matter if choose e.g. 10.000K for color temperature? I guess just the correction matrix would change, so that it reverts the color correction for the certain white balancing applied by the isp, or?

[Bra1nsen]

So its not possible to "turn off" White Balancing, one can just define it, right (at least for the isp)?

[davidplowman]

That's quite an expensive carrier board, but then it looks like it has a Google Coral AI processor on it, which would explain it! It depends whether you need to run TensorFlow models at 30fps or 5fps (not sure of the numbers there, but probably something like that). You can certainly find much cheaper carrier boards around, and which are in stock, but the CM4s themselves may be more of a problem...

When you set fixed colour gains, those are indeed what it uses. But it also calculates, from the calibrated CT curve, what colour temperature "looks most like" the numbers you've entered. For many folks, who are entering values near the calibrated curve, that's useful, but in your case - where the numbers are miles away - it probably doesn't mean much.

It does use that colour temperature to select the colour matrix, however. The question of what colour matrix you should really be using is not clear to me. Maybe you don't really care? Maybe you would do better with a matrix of "all 1s" everywhere? Don't know. If you decide there's a particular colour matrix that you want, you should edit the tuning file and change all the matrices in it to be your fixed matrix.

[Bra1nsen]

I found these lines within the tuning file:

            "rpi.ccm":
            {
                "ccms": [
                    {
                        "ct": 2360,
                        "ccm":
                        [
                            1.0, 0.0, 0.0,
                0.0, 1.0, 0.0,
                0.0, 0.0, 1.0
                        ]

but since the metadata shows a different ccm, I probably have to apply the reverse ccm to get the raw intensities or? Since we are only interested in the raw intensities, not really in color, we dont care about color..

Gamma Transformation is already linear, which is great. So the only thing to do is to define a constant color temperature and set "all 1s" for the cmm, then it should be pretty raw or?

[davidplowman]

The identity matrix there probably makes sense, that way you're getting exactly what the camera is seeing in terms of red, green and blue.

What I don't understand is why you aren't seeing that reported as the colour matrix. Are all the matrices in the file set to the identity? Also maybe worth checking you're loading the right tuning file. I find the easiest way to check is to insert a line of garbage at the top of the file and verify that the camera fails to start!

[Bra1nsen]

Yes that file gets loaded: please just check the preview:

Tuning File: https://drive.google.com/file/d/1p5URc-tW4hgrWwkI0AFTh_W_DPN0rfYh/view?usp=sharing

So it should report the identity matrix for the ccm? Any Idea why its not?

[davidplowman]

I've just tried it and the metadata is reporting the identity matrix as I expect. (Actually it's few hundredths of a percent different, but that's because there are some matrix manipulations happening in the code.)

So it's all a bit baffling. Maybe try running Python like this: LIBCAMERA_LOG_LEVELS=RPiCcm:0 python and see what debug comes out once you start the camera.

[Bra1nsen]

The reason it failed was caused by naming/path.

I called the file: "imx477.json" . Its loading the default json file, even if the file is edited. Now I renamed it, changed from filename to full path and now its getting the right one...

why did we set ct: 2360?

            "rpi.ccm":
            {
                "ccms": [
                    {
                        "ct": 2360,
                        "ccm":
                        [
                            1.0, 0.0, 0.0,
                0.0, 1.0, 0.0,
                0.0, 0.0, 1.0

[Bra1nsen]

Hey David, Iam very curios how the exactly the color temperature of the image is determined. I know you guys used a colorimeter for calibration process.

void Awb::calculateCct(const struct CamMetadata &metadata)
{
    const double u = metadata.color_gains[0];
    const double v = metadata.color_gains[2];

    const double n = (u - 0.3320) / (v - 0.1858);

    metadata.cct = 449.0 * pow(n, 3) + 3525.0 * pow(n, 2) + 6823.3 * n + 5520.33;
}

So it looks like you just used the ratios of:

r = R/G and b = B/G , is that correct? I thought exposure time and lux would also play a role.

[davidplowman]

Yes, we use a colorimeter to measure the colour temperature of every calibration picture. Then we measure the R, G, B values for the neutral squares of the Macbeth chart, and these give us the normalised r and b values (R/G and B/G) for that illuminant. These are what you see reported as the "as shot neutral" values in a DNG file, and the reciprocals of these normalised r and b values are the colour gains that will white-balance the image.

The colour matrix is chosen purely based on the colour temperature reported by AWB. Though if you've set fixed colour gains, then the reported colour temperature will be fixed too.

I don't recognise the code above (I don't think it's ours?), but you're right that the light level (lux) plays a role when the AWB is operating. We define "priors" which give an a priori likelihood of the colour temperature for different lux levels. In very bright light, daylight illuminants are more likey for example. There's quite a detailed description in the tuning guide, section 5.7.

[Bra1nsen]

I cant thank you enough, for taking the time, answering me!

So in consequence the Colortemperature is solely estimated by the raw R G B Values, since known rgb ratios are calibrated with a Colorimeter.

Then the Color Gain is applied based on known calibrated frames to get a first rough correction, the fine tuning is done with the Color Correction Matrix, which considers more parameters as the color gain, values such as lux, exposuretimes, saturation,.. Is that correct?

Why is the ccm after the awb?

[davidplowman]

Yes, that's broadly correct though I would prefer to be more precise!

• Colour temperature is estimated by the AWB algorithm, which uses a set of calibrated images (using a Macbeth chart and colorimeter) to associate R, G and B levels with the colour temperature.
• White balance applies the red and blue gains so as to make things that should be grey actually be grey. I guess you could view this as a "rough correction", but it's a very precise thing. Grey stuff will by grey, but other colours will still be incorrect.
• The CCM is chosen purely from the colour temperature estimated by the AWB algorithm. It has the property that anything made grey by white balancing will remain grey (because the rows of the matrix always sum to 1). It adjusts all the colours, as best it can, so that they match sRGB (our standard colour space).

Hope that helps!

[Bra1nsen]

Ok but why are there two different operations? I mean we could just integrate the color gains within the ccm theoretically? Both are part of the colour correction, while the colour gains just fix the neutral white pixels, the ccm fixes all colours. why not fixing white and all colours within one operation, whats the reason for seperating these two operations?

[davidplowman]

Well, there are some practical considerations. De-Bayering works better when pixels have been white-balanced. These algorithms normally look at pixel differences to "guess" where the edges are, so if the colours all have roughly the "right" order of magnitude, then that works better. So we prefer to white-balance before de-Bayering.

But for the CCM you need all three colours at every pixel location (you wouldn't want to mix together pixels at different locations!), so that has to happen after de-Bayering.

[Bra1nsen]

Ok very intersting. So the color gains are applied to raw RGGB array? Somehow the rgb ratios need to be defined for the estimation of the color temperature, so some kind of pre debayering need to happen there?

So I guess that are bad news for me, since it sounds like that the demosaicing introduces non-linearity-characteristics to the pixel intenties, which affects the proportionality of pixel intensity and irradiance estimation. At least for a physical model that algorithmus need to be known.

I guess for regression techniques it should not matter that much, if enough data is available. Nevertheless, I know you can't share the exact De-Bayer Algorithm, but speaking generally, could you point me towards an open-source demosaicing code that might be similar?

[davidplowman]

Yes, colour gains are applied to the raw RGGB array. There's no need for any de-Bayering there, the gains are calculated from the image statistics which are just R/G/B averages (which you can get fine from the Bayer array).

Demosaic does not introduce any non-linearity. The R, G and B values coming out match the values that you have going in, it's just that all the missing values (the R and B at the green pixels in the raw array, and so on) are now present.

The algorithm in the Pi is more complicated, but in essence it is not unlike this: https://www.cse.iitb.ac.in/~ajitvr/CS663_Fall2016/demosaicing.pdf

[Pepslee]

Can anybody helps ? How to unpack SRGGB12 correctly ?

[davidplowman]

Can anybody helps ? How to unpack SRGGB12 correctly ?

If you're already asking for SRGGB12 explicitly (rather than the packed version SRGGB12_CSI2P) then there's no unpacking to do. In numpy you can view it as type np.uint16 and it should be correct.

[Pepslee]

@davidplowman Thank you. One more question, why do I get 3040x6112 array, instead of 3040x4056 , and how can I convert SRGGB12 to RGB explicitly ?

[davidplowman]

Sounds like you're getting SRGGB12_CSI2P. Each pixel has 12 bits so you have 4056 * 12 / 8 bytes plus some padding. As I said, if you asked explicitly for SRGGB12 then things would probably be simpler (you'd just have 2 bytes per pixel).

Converting that to regular RGB is complicated. That's what the hardware ISP does, so you can get those images on the "main" or "lores" stream. There are some shortcuts you could use to convert the raw image into a kind of RGB in software, but it all depends on your application, what compromises you can put up with, and how much computation you can do.

[Pepslee]

I use data = capture.capture_array(name='raw', wait=True) to read an array. 4056*12/8 = 6048 but capture_array returns 3040x6112. I want to make screenshot with full 4K resolution, send it via network and convert to RGB on the other side, I think the compute capability of the client side is enough for converting to RGB. But I want to read FullHD video at the same time, so I can't use main stream or lores stream to get such 4K snapshot.

[davidplowman]

Yes, there are alignment requirements on the layout of the image in memory, namely that every line has to be a multiple of 32 bytes. That's why 4056*12/8 = 6084 (which isn't a multiple of 32) gets rounded up to 6112.

To take 12MP captures you need to run the sensor in full resolution mode. But the maximum framerate will be limited to 10fps which probably isn't what you want for video, so you would have to pick what is more important to you - you can't have 12MP at 30fps. I also wanted to check that you understand the the "raw" stream is in Bayer format. You can't encode it, you can't really view it without doing lots of processing to it.

[Pepslee]

@davidplowman thank you very much, now I can read raw image and convert it to human readable RGB. Yes, it still needs a lot of work with color, like color correction, white balance, but even without these steps it looks good enough for me.

[Pepslee]

Yes, there are alignment requirements on the layout of the image in memory, namely that every line has to be a multiple of 32 bytes. That's why 4056*12/8 = 6084 (which isn't a multiple of 32) gets rounded up to 6112.

To take 12MP captures you need to run the sensor in full resolution mode. But the maximum framerate will be limited to 10fps which probably isn't what you want for video, so you would have to pick what is more important to you - you can't have 12MP at 30fps. I also wanted to check that you understand the the "raw" stream is in Bayer format. You can't encode it, you can't really view it without doing lots of processing to it.

I don't wont 30 fps 12MP. I want FullHD video 25 fps, but sometimes (one time per several minutes) to make a screenshot of 12MP image. So I want to use main stream for 1920x1080 25fps video, and raw stream to capture 4K image one time per minute.