Applying identity Hald CLUT produces cyan highlights, overall change in exposure, histogram spiking

[Beep6581]

Originally reported on Google Code with ID 2720

See:
http://rawpedia.rawtherapee.com/Film_Simulation
http://rawtherapee.com/forum/viewtopic.php?p=41250#p41250
https://code.google.com/p/rawtherapee/issues/detail?id=2411#c223

Applying Hald_CLUT_Identity_12.png to an image results in clipped highlights turning
cyan and in an overall exposure change. Reproduced in tip, 4.2 and 4.1.54.
http://i.imgur.com/c5LoaN3.jpg

Converting the PNG to TIFF eliminates the exposure change but the cyan highlights remain.

I also noticed extreme spiking in the histogram as if the image was posterized, but
even if the Hald CLUT identity image had only a few bits of precision the code is still
supposed to interpolate the unknown colors from the known ones, so posterization should
not happen. http://i.imgur.com/pi83vqc.jpg

IIRC I made the 12-level identity clut on RawPedia. I tested now with the one from
the author's website and it shows the same problem.
http://www.quelsolaar.com/technology/Hald_CLUT_Identity.png

Reported by entertheyoni on 2015-03-19 22:54:23

[Beep6581]

I made working 12 level identity CLUTs in Tiff and PNG using Imagemagick, I'll upload
them soon

Reported by sguyader on 2015-03-19 23:06:47

[Beep6581]

Here are working 12-level identity CLUTs generated with Imagemagick:

- a TIFF CLUT (huge, 17 MB, don't know how to compress it):
https://drive.google.com/file/d/0B_AvPFlUj8t5WjRqU3VNSGZOOFU/view?usp=sharing

- a PNG converted from the above TIFF with Imagemagick:
https://drive.google.com/file/d/0B_AvPFlUj8t5b3ZfaHM5U2doWGM/view?usp=sharing

Both should eliminate the cyan artifacts in clipped highlights, but the PNG still changes
the exposure in RT.

Reported by sguyader on 2015-03-19 23:17:11

[Beep6581]

The Hald_CLUT_Identity_12.png from RawPedia is broken. It has wrong color pattern (cyan
artifacts). All indentity cluts from http://www.quelsolaar.com/technology/clut.html
has wrong color patterns. 

I wrote a letter about it to the owner of www.quelsolaar.com, about year ago, and he
said that the fault was in not very correct implementation of the TGA format.

In the source code of the RawTherapee, I left a function to generate ident cluts files
with correct color pattern:

rtengine/clutstore.cc
Imagefloat* HaldCLUT::generateIdentImage( int level )

This should eliminate the cyan artifacts in clipped highlights, but still darkening
image.

Reported by dmitry.volvap on 2015-03-20 06:01:18

[Beep6581]

But why is the darkening only happening with the PNG Hald CLUT, and not with the TIF
one? Could this be a problem in libpng?

Reported by sguyader on 2015-03-20 11:08:36

[Beep6581]

I made some tests, the results:
1- None of the many Hald CLUTs I created using ImageMagick exhibited the cyan highlight
issue.
2- All of the PNG ones exhibit the small exposure decrease. None of the TIFFs do.
3- RawTherapee had no problem reading any of the compression methods for TIFF and PNG,
namely: none, zip, lzw, bzip, rle, and for PNG 00-05 and 90-95 (see this link for what
the first and second digits mean http://www.imagemagick.org/Usage/formats/#png_quality
). Compression had no influence over 1 or 2 above.
4- Setting "-gamma 1" is supposed to give the neutral effect (except the PNG is darker
for some reason, point 1 above).
5- The smallest PNG 12-level Hald CLUT was hald12_g1_16bit_92_png.png (64kB) and the
smallest TIFF was hald12_g1_16bit_zip_tif.tif (135kB) so I will use these for further
testing. Both are attached.

Reported by entertheyoni on 2015-03-20 12:18:48

---

• _Attachment: hald12_g1_16bit_92_png.png
  _
• _Attachment: hald12_g1_16bit_zip_tif.tif_

[Beep6581]

DrSlony, what options for Imagemagick did you use to obtain such small tif file?

Reported by sguyader on 2015-03-20 12:26:22

[Beep6581]

convert hald:12 -depth 16 -gamma 1 -compress zip hald12_g1_16bit_zip.tif
convert hald:12 -depth 16 -gamma 1 -quality 92 hald12_g1_16bit_92.png

Strange stuff:
When I look at the histogram of the 16-bit PNG and of the 16-bit TIFF, they differ
slightly. This is in RT and in Geeqie. Maybe this difference accounts for the change
in brightness, making it not a problem in RT. However, when I compare their histograms
in GIMP, they are identical.
Furthermore when I compare them using ImageMagick's "compare", which takes two input
images and produces a third output image where differing areas are marked in red, the
result shows no difference - all white. The same when I set layer blending mode to
"difference" in GIMP. Yet the histograms differ...

But maybe the histogram difference is only cosmetic - it can fit less than 65536 columns
on the screen, so the drawn spikes are somehow interpolated (as you'd expect when sampling
a signal at a rate below the nyquist rate), and if for any reason there is the slightest
difference in drawing the histogram between TIFF and PNG, that might account for moving
spikes in Geeqie: http://i.imgur.com/MhBmokQ.png http://i.imgur.com/mkP0Uq9.png
But the difference in RT's histogram doesn't look cosmetic: http://i.imgur.com/fCLbNWd.png
http://i.imgur.com/P7efeb6.png

I also tried converting the TIFF to PNG, and the resulting PNG matches identically
the other PNGs... I'm using libpng-1.2.52 and 1.6.16.

Reported by entertheyoni on 2015-03-20 13:31:02

[Beep6581]

I am not familiar with hald cluts nor imagemagic .. but ..

This could be a slight gamma difference i.e. pure g2.22 vs sRGB 12.92 + g2.4

As I see both png and tiff have no icc profile embedded so it should be the same for
both but maybe RT supposes a gamma 2.22 for the one and sRGB for the other ?

What is the meaning of "-gamma 1" in Imagemagic ?

If I remember correctly the hald cluts in RT should be in sRGB .. no ?. 

Reported by iliasgiarimis on 2015-03-20 14:00:16

[Beep6581]

I layered them with method 'Absolute difference'. The result was black. But after applying
an extreme brightness curve to the result, there are differences visible.

http://filebin.net/5k28l97d4y/clutdiff2.png

Reported by heckflosse@i-weyrich.de on 2015-03-20 14:07:15

[Beep6581]

Since you can make such light (in weight) TIF files, maybe it would be safer to remake
all the Hald CLUTs in TIF for RT?

Reported by sguyader on 2015-03-20 14:23:16

[Beep6581]

I did as Ingo, but no difference here, neither in GIMP nor using imagemagick's "compare".
This is weird.

When I check the generated Hald CLUT image files for detailed info, I find:
TIFF:
Gamma: 0.454545
PNG: Gamma: 0.45455
    png:gAMA: gamma=0.45454544 (See Gamma, above)

I already replaced the identity file in RawPedia with the TIFF. I'd like to wait a
while before deciding to convert the whole RawTherapee Film Simulation Collection to
TIFF.

Reported by entertheyoni on 2015-03-20 15:01:26

[Beep6581]

OK, I found 2 interesting things:

- when I convert the PNG identity CLUT (which darkens the image) to TIF, the produced
TIF doesn't darlen the image, which means that the problem is really related with with
how RT reads the PNG file

- since Ilias talked about possible gamma mismatch, I used Imagemagick with +gamma
0.5 (AFAIU it doesn't apply gamma to the image, but sets the "gamma=0.5" tag in the
PNG header), instead of -gamma 1, to generate the PNG... and bingo, the png CLUT seems
to not darken the image anymore as RT applies the gamma correction.

I don't know if it's the right way to do, but if I export images from RT treated with
either the TIF and the "+gamma 0.5" PNG and compare the resulting images, there's no
difference in their histograms (in Gimp 2.9) and the "compare -compose" in Image Magck
results in a totally white image, confirming no difference.

Reported by sguyader on 2015-03-20 17:24:46

[Beep6581]

I got it. After changing rtengine/imageio.cc line 289 ff from

    //setting gamma
    double gamma;
    if (png_get_gAMA(png,info,&gamma))
        png_set_gamma(png, 2.0, gamma);
    else
        png_set_gamma(png,2.0, 0.45455);

to

    //setting gamma
    double gamma;
    if (png_get_gAMA(png,info,&gamma))
        png_set_gamma(png, 2.2, gamma);
    else
        png_set_gamma(png,2.2, 0.45455);

there is no difference between clut png and tif in rt.

The differences in my example from #9 were caused by pwp not supporting 16 bit png
correctly.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-20 17:28:54

[Beep6581]

#13 Confirmed, I was wondering why a "gAMA=0.5" chunk in the PNG worked, as it implied
a 2.0 gamma curve instead of 2.2, but I dodn't know where to look. Thanks Ingo.

At least we won't need to reconvert the whole CLUT directory!

Reported by sguyader on 2015-03-20 18:02:18

[Beep6581]

Maybe an even better modification would be this?

    //setting gamma
    double gamma;
    if (png_get_gAMA(png,info,&gamma))
        png_set_gamma(png, 1.0/gamma, gamma);
    else
        png_set_gamma(png,2.2, 0.45455);

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-20 18:18:51

[Beep6581]

Yes Ingo, that would be a more universal solution I think.

Reported by sguyader on 2015-03-20 19:32:48

[Beep6581]

Ingo, this line means that if metadata are found use them else suppose gamma 2.2 ?.

One point is to be as exact as we can .. 1/2.2 = 0.454545 (same as with the tiff)

Second point is, how can we be sure about the gamma encoding used on the existing film
simulations ?. Do we have the prototypes ?. Do the correct metadata info exist in them
?.

Third point which can affect the decision to rebuild the film simulations ..
There is a significant difference between the existing Hald_CLUT_Identity_12.png vs
the new hald12_g1_16bit_92_png
https://drive.google.com/folderview?id=0B0NqktTgc54sfmdVS1hGZW55NF90MkhGbnFBV0FUV1AxZ0EtNXUzMHFIOG5LLXE3WVhyTWc&usp=sharing
obvious at the red histogram .. is it due to 16bit ?. A lot more clipping at the black
side but a much more refined histogram at the midtones.
PS .. I know that this is a wrong rendering as I have not changed the lines that Ingo
mentioned. But think of it as a simulation of a haldclut that changes slightly the
picture (i.e. gamma 2.0 instead of 2.2) and how much more refined the 16bit is ..

If we choose the 8bit identity.png I think that a tone curve which is more robust than
the pure 2.2 gamma is preferable. like the sRGB complex curve or rec.709 (slope 4.0
up to a point, then 1/0.45) http://en.wikipedia.org/wiki/Rec._709 
The pure exp encoding uses too many codes at the darks 

Reported by iliasgiarimis on 2015-03-20 19:42:02

[Beep6581]

I think we should ship the identity with 16-bit precision to allow users maximal wiggle-room,
but we should ship all the other CLUTs in 8-bit unless someone can specifically point
out a real-world problem with that. With 8-bit ones the histogram spikes, I don't know
why (some text on quelsolaar.com I read a year ago made me believe this should not
happen) but the images don't seem to have posterization, and they take less space,
and I don't think Pat David & co. want to regenerate 16-bit ones (I'll ask, maybe he
has them as that).

Reported by entertheyoni on 2015-03-20 21:23:30

[Beep6581]

Ilias, yes, it means use metadata if found, else suppose gamma 2.2.

Of course I can change the hard coded values, so the patched code would be this:

    //setting gamma
    double gamma;
    if (png_get_gAMA(png,info,&gamma))
        png_set_gamma(png, 1.0/gamma, gamma);   // use gamma from metadata
    else                                       
        png_set_gamma(png, 2.2, 1.0/2.2);       // no gamma in metadata, suppose gamma
2.2

About the gamma encoding in existing film simulations: I don't know!

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-20 21:52:14

[Beep6581]

Hmm the metadata are not same for all film simulations .. found two categories so far

File Name                       : Fuji Superia HG 1600.png
Image Width                     : 1728
Image Height                    : 1728
Bit Depth                       : 8
Color Type                      : RGB
Compression                     : Deflate/Inflate
Filter                          : Adaptive
Interlace                       : Noninterlaced
Gamma                           : 2.2
White Point X                   : 0.3127
White Point Y                   : 0.329
Red X                           : 0.64
Red Y                           : 0.33
Green X                         : 0.3
Green Y                         : 0.6
Blue X                          : 0.15
Blue Y                          : 0.06
Background Color                : 255 255 255

File Name                       : Fuji Superia 1600.png
Image Width                     : 1728
Image Height                    : 1728
Bit Depth                       : 8
Color Type                      : RGB
Compression                     : Deflate/Inflate
Filter                          : Adaptive
Interlace                       : Noninterlaced
SRGB Rendering                  : Perceptual
Background Color                : 255 255 255
Pixels Per Unit X               : 9449
Pixels Per Unit Y               : 9449
Pixel Units                     : meters

sRGB primaries from wiki http://en.wikipedia.org/wiki/SRGB#The_sRGB_gamut.
Chromaticity    Red     Green   Blue    White point
x   0.6400  0.3000  0.1500  0.3127
y   0.3300  0.6000  0.0600  0.3290
Y   0.2126  0.7152  0.0722  1.0000

In the first case the color primaries are those of sRGB, but a gamma 2.2 is defined
instead of the normal sRGB gamma http://upload.wikimedia.org/math/c/d/e/cde58f130d91e7621da36876ddbb199a.png

while in the second we have sRGB rendering without declaring gamma so we have to suppose
the sRGB tone curve ?. How is RT dealing with this ?.

Reported by iliasgiarimis on 2015-03-20 23:11:09

[Beep6581]

Ilias, afaik we only pass gamma and inverse gamma to libpng actually. So in first case
with modified code we pass 2.2 and 1/2.2 because we get that information from metadata.
And in second case we pass the same because we suppose it's gamma 2.2.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-03-20 23:25:16

[Beep6581]

Reported by heckflosse@i-weyrich.de on 2015-03-20 23:25:27

• Status changed: Started

[Beep6581]

#19 committed to revision f6dc4dd6bda7
Issue stays open because of #20

Reported by heckflosse@i-weyrich.de on 2015-03-23 13:33:09

• Status changed: PatchSubmitted

[Beep6581]

I know this is a bit "stretch". But I tested various RAW files with those film profiles
from G'MIC and found out that several film profiles cause image to shift insanely red.

For example as far as I could see, A99 files are working with every color profile.
While for example my S5 Pro is "no go" for many of them. I tried 1Ds MK3 files too,
pretty much same story, only bit less "red". Even quite new D810 has red spiked up
and shifted a lot.

I wondered, is there any way to fix this?

By the way, most of those film profiles seem to be in Apple RGB color profile? I guess
it doesnt matter much tho.. (except I got interesting result when converted to AdobeRGB
as "perceptual"). Not much clipping (or histogram change) much less red.

Reported by Mescalamba on 2015-04-29 23:25:03

[Beep6581]

We don't ship Hald CLUTs "from G'MIC", we ship them from Pat David and Pavlov Dmitry,
the former is where G'MIC got theirs from. Which version of RT are you using, which
version of the RawTherapee Film Simulation Collection, and which images are in Apple
RGB?

Reported by entertheyoni on 2015-04-30 07:14:16

[Beep6581]

I downloaded these..

http://rawtherapee.com/shared/HaldCLUT.zip (via http://50.87.144.65/~rt/w/index.php?title=Film_Simulation)

And I have latest RT 4.2.162 x64.

And all those Hald CLUTs are Apple RGB. At least Photoshop CC says so, when I try to
convert them to different colorspace. But for example Photoshop Bridge x64 says that
RGB is untagged (eg. no color profile).

Its possible for me to fix output of these to certain extent via color channel mixing
and additional PP work in PhotoShop, along with using LightZone as RAW developer for
my files. Tad bit.. complicated to say at least.

Interesting part is that for example A99 files work pretty much perfectly (or very
close to it, only red tint there is probably intentional one).

Reported by Mescalamba on 2015-04-30 16:05:12

[Beep6581]

Perhaps your Photoshop is set to assume an image is in Apple RGB if it has no profile
assigned. I checked all the images in the Fuji folder for example, and none of them
are Apple RGB.

Reported by entertheyoni on 2015-05-01 23:48:50

[Beep6581]

Downgrading this as the main issue is fixed. Unless #20 can be demonstrated to be a
problem I'd ignore it.

Reported by entertheyoni on 2015-05-01 23:53:44

• Status changed: FixedPendingConfirmation
• Labels added: Priority-Low
• Labels removed: Priority-High

[Beep6581]

Yep, my bad, I had somehow set working space as Apple RGB instead of my regular Adobe
RGB.

Still I wonder why it spikes red a lot for many cameras..

Reported by Mescalamba on 2015-05-01 23:54:19

[Beep6581]

Can we close this Issue?

Reported by heckflosse@i-weyrich.de on 2015-05-27 22:10:24

[Beep6581]

Let's close. If someone can demonstrate a problem, please open a new issue.

Reported by entertheyoni on 2015-05-30 15:02:53

• Status changed: Fixed