Feature request: generating a profile / color adjustments file by comparing to TIFF files

[assaft]

I'm not sure whether this is feasible but I think it can be helpful. The goal is to make it easier to reproduce, in a RAW workflow, the color adjustments that the manufacturers do in their jpeg engine. Currently the only straightforward way to do that is by using Adobe's color profiles, but I found them not so accurate, at least for my cameras.

So I'm thinking of an alternative way of reproducing the color signature of the manufacturer RAW converter. This is the idea:

1. Take a picture of a color checker.
2. Develop the RAW file using the manufacturer proprietary RAW converter. Obtain a TIFF file that incorporates all the color adjustments that the manufacturer does (by default).
3. Develop the RAW using the most basic profile, or no profile at all, in RT and obtain a TIFF file.
4. Run some sort of 'diff' between the TIFF created by the basic/no profile in RT and the TIFF created by the manufacturer's RAW converter. The result of this 'diff' (i.e. the delta between the two TIFF files) should be saved as a profile or a set of color adjustments.

Then, when a new RAW is developed, it should be possible to open it with the basic/no profile, and subsequently to apply the delta that was previously created. This will simulate the manufacturer's rendering while keeping the workflow in the RAW converter.

In some way this is similar to the algorithm that automatically corrects distortion based on the embedded JPEG. The algorithm checks how the JPEG was corrected in relation to the RAW file being developed and tries to infer the correction parameters. So What I'm asking about here is whether a similar algorithm is possible with respect to color, based on an image of a color checker.

[Beep6581]

This is what the Film Simulation tool does, the problem is it uses a raster image instead of a raw image, so the target raw development program probably won't do the same things to it as it would to a raw file... unless you trick it into treating it as a raw file. Have you tried that? Convert the identity Hald CLUT to a linear DNG and open it in the target raw converter? http://rawpedia.rawtherapee.com/Film_Simulation

[assaft]

Thanks. Sounds good, and I actually used Film Simulation in the past for some images, but I don't really know how it works internally. I'll need to read about it and to understand how they are created. One problem I expect with the original suggestion is regarding WB - if I'm relying on TIFF files then WB has already been applied. I might get a different diff between the two TIFF files if there is a change in the WB. Could you please elaborate about the purpose of converting the identity Hald CLUT to a linear DNG?

[assaft]

OK, I got what you mean. I'm afraid I can't trick Olympus's RAW converter to open the identity Hald CLUT as if it was a RAW file. I'll check what's possible; I think it works only with ORF files.

[Beep6581]

Yes, your request would be very limited in use as it would depend on many things being the same, and anyway replicating the camera look is not what RawTherapee is about. I'm just pondering the idea for curiosity's sake.

I'm trying to figure out how to convert a TIFF to DNG using FOSS. Do you have LR?

[assaft]

Thanks, but no, I don't have LR. How a DNG would help here?

[Beep6581]

I worked out how to make a DNG of the Hald CLUT image so that hopefully it can be used to emulate the tone processing of other software (and equipment) which requires raw files.

convert hald:12 -depth 16 -colorspace RGB -gravity NorthWest -splice 4x4 -gravity SouthEast -splice 4x4 foo.tif
exiftool -DNGVersion=1.4.0.0 -PhotometricInterpretation='Linear Raw' foo.tif
mv -v foo.tif Hald_CLUT_Identity_12.dng

The program needs to support DNG files, and demosaiced ones at that, what Adobe DNG Converter refers to as "Linear (Demosaiced)". Any modern program should.

The user must figure out whether any raw cropping is performed by that program when reading demosaiced DNG files, and adjust the borders accordingly. Open the DNG in the target program, zoom in to each corner and work out whether it does any cropping and if so, how much, then adjust the command accordingly. RawTherapee cuts off 4 pixels all around when reading demosaiced DNG files, so my command above first adds a 4 pixel row and column to the bottom and right edges, then another 4 pixel row and column to the top and left edges. The 12-level CLUT image's size as reported by the target program must be 1728x1728.

Once you got the borders figured out, merely open this DNG in the target program, apply whatever color settings you want (remember, nothing local, no tone-mapping, no distortion correction, just global tone adjustments), save as a 16-bit TIFF or PNG to the folder you set in RawTherapee's Preferences in the Image Processing tab > Film Simulation > HaldCLUT directory, then use that on an image.

Sample 12-level identity HaldCLUT DNG: http://filebin.net/7633s8a1tv

[michaelezra]

DrSlony, I can create a few clut-s using 3dlut-s from photoshop. I gave it a try and it works on 4096x4096 file. However, if I resize it to 1728x1728 it does not apply correctly in RT. What resizing method do you recommend?

[Beep6581]

None, you cannot resize them! Their size is determined by the number of levels and must be kept that way pixel for pixel, only adding a border to avoid having the actual image cropped.

4096x4096 is a 16-level Hald CLUT. That's fine, just add the 4 pixel border as in my code snippet.

[michaelezra]

The ident clut I have is 4096x4096.. where can I get 1728x1728?

[michaelezra]

ok, found it on rawpedia:)

[Beep6581]

The size is irrelevant as long as it stays the same after raw cropping. I edited my reply above as you posted yours, please see it.

[assaft]

Thanks, that's very impressive.

I was thinking about an alternative way:

1. Opening the original RAW file in the other RAW converter, developing it to taste, and exporting as TIFF. Opening the RAW also in RT without any profile and exporting it without any adjustment to TIFF.
2. Using a small program that I write to loop over the two TIFF files and subtract them pixel by pixel. This is similar to opening them as two layers in GIMP with the subtract layer mode. The only difference is that GIMP doesn't allow negative values while my program does. This is important for my use case.
3. Once I obtain the diff between the two TIFF files (saved in some binary format), I can add it pixel-by-pixel to the identity Hald CLUT image and create a new film simulation.

There are some problems here with the distortion correction - the proprietary RAW converter applies them automatically, so in order for the two TIFF files to be comparable I need to match the correction in RT or to disable somehow.

[Beep6581]

The TIFF once saved from the DNG is the diff, and its better than your subtraction idea because it will work with any photo of any size or bit depth.

[assaft]

I see. Thanks. My only problem is that the RAW converter I meant to use cannot work with DNG. Only ORF files. It's the Olympus Viewer 3.

[Beep6581]

I'll document this technique once someone confirms the DNG opens in LR or some other raw developer, but as for this issue there is nothing to do. Nobody will spent hours designing something as cumbersome and generally unuseful as this ;) I'll close it soon.

P.S. try imagemagick's blend modes,maybe they will help.

[michaelezra]

DrSlony, Could you add this to RT: http://filebin.net/upload/eirpvcjae6 These clut-s are based on some supplied in Photoshop (I hope there is no conflict though) and my custom derivatives

[Beep6581]

Cool, I confirm that this DNG Hald CLUT works. Houz converted the DNG to TIFF (settings used: input=linear rec709, out=sRGB) using darktable with a Pentax base-curve, and a Pentax K10D PEF opened in RawTherapee with the Neutral profile applied and this CLUT looks the same as the embedded JPEG. So as long as a program doesn't change the dimensions in any way (raw cropping, distortion correction) or do things like noise reduction or sharpening, it will work.

Michael I'll add them tomorrow.

[assaft]

Cool. Thanks for these efforts.

[Beep6581]

@michaelezra hey, I sent you an email.

[michaelezra]

DrSlony, sorry for the late response. You can rename this to CreativePack-1.

[Beep6581]

Ok.

[Beep6581]

DNG Hald CLUT documented: http://rawpedia.rawtherapee.com/Film_Simulation#Advanced_-_Identity_DNG

[renierra]

@Beep6581 hello from 2021 - I'm really having trouble generating this fake RAW HALD file as a DNG.. Do you guys still have any around because the links above have expired obviously.. Trying to profile results of a developing software here and a DNG HALD really is what I need...

[Thanatomanic]

@renierra You might have more luck if you post this request on the https://discuss.pixls.us forum. There's plenty of enthusiasts who might be able to help you.