Improve highlight clip handling?

[Beep6581]

Originally reported on Google Code with ID 2857

Background:

Highlight reconstruction actually does two things. 1) reconstruct (=guess) clipped
data and 2) modify unclipped data close to clipped highlights such that smooth transition
into the clipped areas are had.

As highlight reconstruction only looks at one pixel at a time it doesn't know if a
bright highlight will transition into a clipped area or not. It will handle all things
equal. This means that in a file where there are lots of bright highlights, but actually
no clipped data, the highlight reconstruction algorithm will still modify colors. This
may seem strange and illogical to the user, and if you work with reproduction photography
it's really bad as it may distort your bright colors.

Probably due to this reason highlight reconstruction is actually disabled per default.
And really, if we don't apply any negative exposure there is no real need of highlight
reconstruction as any clipped data will be at 100% pixel value anyway.

When we do apply negative exposure, or darken otherwise (shadow/highlights), clipped
data becomes visible. If no reconstruction is made this will show pink highlights.

Long time ago RT designers thought that pink highlights is just too ugly, so even with
highlight reconstruction disabled there's pre-clipping made in rawimagesource that
makes sure no channel gets past 100%. The disadvantage of this is that also local bright
highlights that actually does not transition into clipping will get clipped. How is
this possible? Well, due to white balancing the raw max level is >100% for all channels
except one (typically green). That is we prematurely clip unclipped red and blue data.

The problems we get from the current design:

1) with highlight reconstruction disabled and negative exposure unclipped highlights
may have been clipped and cannot be recovered. With no negative exposure we're safe
though.

2) highlight reconstruction will change color also of highlights that actually don't
clip, even if nothing in the whole file doesn't clip.

Enhancement suggestion:

One enhancement/change I'd like to experiment with is as follows:

- make highlight reconstruction area-aware, that is only 
  reconstruct/desature areas which actually lead into a clipped area and 
  leave other areas untouched
- with the new area-aware reconstruction the new default should be
  highlight reconstruction enabled

The gain with this would be that highlight reconstruction can and will be enabled per
default (blend mode which is most robust) which is more user-friendly, and it won't
cause any modifications of unclipped areas.

There are a couple of problems though to be aware of -- possibly the blend mode has
bad image quality in some cases, I've forgot how well it makes clipped sun for example,
so maybe it needs some image quality improvement before it can be default. The other
issue is that I'm not 100% sure mixing rolloff into clipping with no rolloff in very
bright highlights is a good idea, or if it will show as uneven colors of highlights.
I think that needs testing.

Reported by torger@ludd.ltu.se on 2015-07-26 17:24:27

[Beep6581]

Effectively the exposure slider along with HL reconstruction/recovery are RT's means
for handling the highlights. Currently highlight whiteness can be controlled e.g. using
the film-like curve.

Could this be better automated? E.g. next to HL reconstruction add a mode - white or
coloured highlights. In some cases, e.g. sunset the burnt sun needs to retain its colour,
while daytime sun would be pictured as white.

Reported by michaelezra000 on 2015-07-27 03:57:54

[Beep6581]

Center of the sun is always clipped, ie there is no color information so it's a difficult
challenge. Color propagation can probably recreate the sun's color, but it's not robust.

There's an overlap between highlight reconstruction and a curve's highlight rolloff.

Color propagation is the only algorithm that is a pure reconstructor, ie does not touch
any surroundings, while all other will do a rolloff, just like an S-curve does. The
overlap is not harmful though as far as I can see, if the highlight reconstructor has
done desaturation towards clipping already a film-like curve will not add more.

Personally I don't use highlight compression much, I instead apply negative exposure
and compress using a tone curve, which achieves the same thing. But highlight compression
is more user-friendly so I suppose it's good to have, just as the contrast and lightness
sliders (which also can be manually replicated with a curve).

Commercial raw converters generally skip the highlight compression control though,
but instead has some auto rolloff+reconstructor (much like what the blend reconstructor
does) and let the profile curve take care of the rest of the rolloff. I'd like to have
a similar default behavior, ie good looking highlights per default without doing anything
(they can't fix a clipped sun though).

Today in RT I feel that no reconstruction is not that good for heavily clipped files,
and (blend) reconstruction may be too intrusive on unclipped files.

Reported by torger@ludd.ltu.se on 2015-07-27 08:07:11

[Beep6581]

As a photographer I would want the default method of highlight handling to not change
the hue of the transition area (see https://code.google.com/p/rawtherapee/issues/detail?id=1120#c7
for an example of horrible hue change) though changing the luminosity a little is fine.
If none of the raw channels are clipped one should be able to use all channels for
highlight handling regardless of white balance multipliers. If one or more raw channels
are clipped then it's up to the user how they should be reconstructed - whether to
just reconstruct the luminance (wedding dress), or to retain some color (flowers),
or to extrude color as color propagation does (skin).

Furthermore, I would want fewer buttons to accomplish this, or the same amount, but
certainly not more.

Reported by entertheyoni on 2015-07-27 18:41:05

[Beep6581]

I like Blend the most as a default method, but it can probably be improved concerning
those hue shifts. I would guess a rgb-hsv blending would make a better job.

Maybe I'll have a look. It depends on the weather ☺

Reported by torger@ludd.ltu.se on 2015-07-27 18:59:14

[Beep6581]

As a sidenote, many cameras and some raw converters do like this for heavily clipped
files, illustrated here with the difference between an older and newer version of Lightroom:

http://www.lumariver.com/resources/img/lr4-render.jpg

2010 is the old version (-0.5 is just baseline exposure), this is how also RT does
it, 2012 is the newer version which shows an overall brightening and desaturation of
overexposed files, to simulate the look of overexposed film.

Regardless of reconstruction algorithm (except color propagation in cases it works)
heavily clipped files will look ugly in some way, unless we turn it into an "overexposed
film frame", like Lightroom does, and indeed also my in-camera Canon JPEGs.

Doing something like that, if we even want to, would be a later thing though.

Reported by torger@ludd.ltu.se on 2015-07-27 20:28:43

[Beep6581]

#3 I've just experimented with an RGB-HSV blender that keeps hue constant, and voila
I ended up exactly with the "Luminance" result which is the same thing, it just wasn't
that obvious....

You still get that hue change in the issue 1105 example. Why? Well, to really guarantee
no hue shift we must have already blended to monochrome when the first channel clips,
because as soon as one channel is clipped we cannot be sure about what the hue actually
is.

However if we do that we will kill color deep down, for example RGB 100,10,10 would
become grayscale and it just doesn't make sense. If it is 100,10,10, some sort of red
is a much better guess than grayscale.

In practice hue is generally sort of right as long as only one channel is clipped,
but when two are clipped then the hue starts to distort.

So one wants to make a tradeoff that keeps the clipped hue as long as it's likely it's
sane. It will fail in some cases though.

Methods that can solve it look at surrounding unclipped pixels and flow the hue from
there, like color propagation does. The disadvantage of those algorithms is that they
become unstable and fail miserably in some cases.

Reported by torger@ludd.ltu.se on 2015-07-28 11:29:24

[Beep6581]

It could be noted that I don't get the result as shown in:
https://code.google.com/p/rawtherapee/issues/detail?id=1120#c14

I do get a cyan band with Luminance recovery in the newest. I think it's simply becuase
base exposure or clip levels has changed. In the 1120 example the cyan band is simply
pushed out to white.

Reported by torger@ludd.ltu.se on 2015-07-28 11:35:50

[Beep6581]

I've experimented with making a new highlight reconstruction algorithm, much similar
to Luminance, but avoiding modifying unclipped pixels. I think it's feasible in terms
of result and would be a nice default (as it will retain color higher up and would
not modify unclipped images), but unfortunately it's complicated to implement. It needs
global knowledge like the Color Propagation method as it must know the RGB levels of
the nearest unclipped pixel.

I now think trying to make such an algorithm would be the first step in improving highlight
clip handling, if one gets that to work well then there would be no reason except for
diagnostics to turn off highlight reconstruction.

Reported by torger@ludd.ltu.se on 2015-07-29 16:44:03

[Beep6581]

Sounds good!

Reported by entertheyoni on 2015-07-29 17:50:58

[heckflosse]

Some months (or more) ago I tried to solve the problems with 'Colour propagation'. Restoring more than one burned channel is really difficult. As long as there's only one burned channel 'Colour propagation' worked good in my tests.

Ingo

[heckflosse]

Let's continue this in #2929