Creating and reading HQ camera images with picamera2/pidng/libraw/rawpy

[cpixip]

Hi, not sure if this is the right place to ask; I did post a similar question to the picamera2 repo.

The thing is rather simple. Once I have created a raw .dng-file with the picamera2 lib, I want to read it back for further processing. I use the standard libraw in its python disguise, ie., rawpy for reading in the raw image, postprocessing it to a linear 16-bit which is then converted into a float image with values in [0:1], like so:

with rawpy.imread(path) as raw:
    rgb  = raw.postprocess(gamma=(1,1), 
                           no_auto_bright=True, 
                           half_size=True,
                           output_bps=16)
imgA  = rgb/0xFFFF

An image of a color checker taken in daylight illumination (around D65) shows however rather large color differences to the reference;

Here, the small rectangles visible within each patch show the colors these patches should actually have.

Jugding from this test, the colors come out too vivid (more saturation) and especially the blue color patch looks weird. While green colors seem to be ok, there are certainly also deviations in the yellow-red zone. Overall, the image looks too blueish for my taste.

I think this is related to the color matrix embedded in the .dng; I get similar results when I open this image for example in darktable using the embedded matrix as input profile.

Now, creating an optimized matrix on the basis of the color checker, I end up with the following image,

which shows some improvement in most colors. Mostly noticable is a reduced color saturation.

So, here's my question: where do the color matrices embedded in the .dng-file come from? It seems that the .dng's (at least the ones I captured with picamera2) have only a single Color Matrix 1 embedded in their EXIF-data, plus two (?) Camera Calibration Matrices (which would encode unit-specific variations - but only one would be needed here, and it is unused anyway).

It is tempting to assume that the single matrix embedded was derived from the libcamera's choice of the ccm at the time of capture. If so, it would be a matrix interpolated between two matrices selected from the tuning file, namely the two closest ones in estimated color temperature. If that is the case, the slightly wrong colors noted above might give a hint towards a not so perfect data set in the tuning file. A better tuning file could than in principle resolve this issue.

Interestingly, in the camdefs.py file of this repo are actually two ccm's plus two forward matrices listed in the RaspberryPiHqCamera class - if these matrices are corresponding to the classical illuminant A and D65 data usually found in .dng-files, this would open up a .dng-path independent from libcamera (which I would prefer) and the uncertainties related to the tuning file. Are these matrices used in any way? I couldn't find anything in the code. Besides, it would be interesting to know where these matrices are coming from.

[cpixip]

Libcamera/picamera2 writes the estimate of the current color temperature and the compromise color matrix used to create the embedded preview-jpg into the DNG-files. Generally better results can be achieved by by-passing this data. One way to do this is to use DCP input profiles in the raw converter. For the HQ camera/IMX477 sensor, these can files can be downloaded from here.

Once this input profile is selected in the raw converter, the embedded color matrix is no longer used. A second necessary step is to adjust the white balance manually, for example by sampling a neutral colored patch with the white balance picker. This second step discards the embedded white-balance parameter. The resulting image

is much closer to ground truth than the image created with the data embedded in the DNG: