Improved tonemapping

[Beep6581]

Originally reported on Google Code with ID 1237

This patch aims to improve "tonmapping"
1) it controls the gamut before "tonemapping" and corrects colors with an algorithm
similar to that of "vibrance" (Munsell correction)
2) it changes the luminance after "tonemapping" to restore the histogram in similar
values ​​before applying "tonemapping" (if Strength > 0)
3) corrects the colors (as in 1)) after "tonemapping"

It is enabled by a checkbox "avoid color shift"

Reported by jdesmis on 2012-02-03 11:50:38

• Blocking: #1130

---

• _Attachment: tone_map0.patch_

[Beep6581]

Hi Jacques, great to see that "avoid color shift" is propagating to other lab tools!

As I am trying this patch, I notice that the global brightness of the image is changed
when "avoid color shift" option is toggled. Is this expected?

Reported by michaelezra000 on 2012-02-03 18:51:45

[Beep6581]

Yes you are right. This corresponds to point 2)of my introduction : "2) it changes the
luminance after "tonemapping" to restore the histogram in similar values ​​before applying
"tonemapping" (if Strength > 0".

Indeed, I reviewed the images before / after tonemapping and I found significant differences
in overall luminance. For example for "Strength = 1", L (0..100) increases by about
20 on average, and "Strength = 2", this increase can reach 30 or 40. Histograms are
strongly shifted to the right. Of course it depends images ...

To verify, I downloaded "LuminanceHDR" and "PhotomatixPRO4.1." In this two cases, the
overall luminance is not changed. Of course the local luminance is changed, it is the
purpose of this type of software that make the "tonemapping"

At the end of treatment "tonemapping", I changed the overall brightness, but of course
it would probably be better (?) to rework the algorithm itself (?) :)

Reported by jdesmis on 2012-02-04 06:57:21

[Beep6581]

I thought the purpose of the tool is to lift shadows, so of course the histogram will
be shifted to the right.

Reported by ejm.60657 on 2012-02-04 07:24:58

[Beep6581]

Emil, try "PhotomatixPro" and see the result... With the change in overall luminance
I suggest (if the user does not wish, it does not check the box "avoid color shift"),
the result of RT is very similar to Photomatix ...

Reported by jdesmis on 2012-02-04 07:31:21

[Beep6581]

Hi Jacques,
Just tried the new "Avoid color shift" on a photo, but it seems to neutralize the desired
shadow lightup effect of tone mapping almost completely. Do I miss something in how
it should work?
Thanks
Olli

Reported by oduis@hotmail.com on 2012-02-04 17:49:45

• Status changed: PatchSubmitted
• Labels added: Type-Enhancement, Component-Quality
• Labels removed: Type-Defect

[Beep6581]

Here is an illustration of the problem (fully developed images, right is normal tone
mapping)

Reported by oduis@hotmail.com on 2012-02-04 18:13:59

---

• _Attachment: LeftColShift_RightNormal.jpg_

[Beep6581]

hello Oli

You're right, there is a difference, which is due to the luminance correction I've
made​​.

But if I look at the different products on the market that make "tonemapping" (Photoshop
CS, Luminance HDR and PhotomatixPro) none produced an overall increase in brightness
by default. Several even do the opposite with some options

The product made ​​by Ben_pcc is very high level, we obtain remarkable effects, which
from my point of view are at least at PhotomatixPro (reference ...)

If I introduced it to fix the overall brightness is so that RT gives essentially the
same report that the 3 products I've tried. The amendment is very simple ...

In the three products mentioned there are several functions that allow to vary the
image brightness (gamma, brightness, etc..), but by default the histogram is not significantly
shifted to right as in RT.

I'll add a "slider" which allows to vary the action of the luminance correction, between
the current setting and the one I proposed).
Nevertheless it is not simple because of this fact -if the luminance varies widely
- color correction "Munsell" should be changed to avoid  red ==> yellow

I have not been able to participate in the development of "tonemapping" because between
early November and 10 January, I had health problems with many hospital stays ...
I would be away for 4 to 5 days from Monday ...but I will try to propose something
part today.

Reported by jdesmis on 2012-02-05 07:58:31

[Beep6581]

Jacques, could you describe the additional tone curve you are applying?  I think what
has many of us confused is that a common usage of the tonemapping tool is to lift shadows,
so this will generically increase the overall brightness.  Are you intending to make
the tonemapper more of a shadow/highlight tool?  An edge-aware local contrast enhancement?
That I think would be useful and could replace the 'high quality' option for the S/H
tool, and then this would be an appropriate place to put your modification.  I have
been thinking perhaps we might rework the user interface to combine the two tools (high
quality S/H and tonemapper), both use an edge-aware blur, and differ mainly in the
manipulation performed to the lo/hi pass channels.

Reported by ejm.60657 on 2012-02-05 08:47:59

[Beep6581]

Hello Emil

Glad to talk to you again ...
When I saw the images produced by "Tone Mapping RT" I was surprised ... and I looked
at the documentation on the web about "tonemapping" and "Edge Preserving Decomposition"
...
All comments are intended to say that it is a tool "HDR" which changes the contrast
local and global.
The basic algorithms (Mantiuk) revised and improved by Ben_pcc have as its main purpose
the production of HDR type image and not a tool like D-lighting.

Certainly it can also produce these effects, but it seems that the algorithm performed
by Ben_pcc significantly increases the overall luminance (up to 40: 0..100 scale)
I made several tests measuring several photos before / after "tonemapping", the average
luminance and standard deviation. We can say that according to "Strength", the Gaussian
curve (mean standard deviation) moves in proportion to Strength.
This is not the case for PhotomatixPro or Luminance HDR

The algorithm is very simple and can certainly be improved :
I calculate the average L, before and after "tonemapping" and makes the difference
"w", then according to the difference in luminance of each pixel before / after I apply
a progressive correction, very small if the difference is small , if the maximum difference
is close to "w".

Of course we must be able to reconcile the two tools.

This is a patch that modulates changes in luminance. It is very basic and does not
correct "Munsell".

But again, the tool Ben_pcc is remarkable, at least PhotomatixPro, to create images
that are beautiful

Reported by jdesmis on 2012-02-05 09:47:17

---

• _Attachment: tone_map1.patch_

[Beep6581]

Just a clarification, when I say "does not correct Munsell" is for the variation in
luminance that the correction is not made. 
For the rest (gamut, variation in chromaticity) correction Munsell => Lab is done.

Reported by jdesmis on 2012-02-05 16:22:50

[Beep6581]

I lost ability to compile/link at the moment, so cannot test, but quick question:
I think confusion originated from the fact that in tone_map0.patch "avoid color shift"
affected global luminance as well and it is not in the checkbox description:)
In tone_map1.patch - does "avoid color shift" effect only color shift or luminance
as well?

Reported by michaelezra000 on 2012-02-05 16:40:44

[Beep6581]

With "tone_map1.patch","avoid color shift" effect color shift, and allow luminance correction
(Luminance=0 ==> maximum correction with "avoid color shift", Luminance=1 no correction;;
 luminance default=0.3)
If "avoid color shift" disabled, slider "luminance" has no effect.

Reported by jdesmis on 2012-02-05 16:48:34

[Beep6581]

It seems to me that the issue is where the neutral point of the correction should lie.
 Ben made the choice to fix the white point if I recall correctly, so that the tonal
range is compressed keeping the white point fixed.  One could of course keep another
tonal value fixed while compressing the tonal range, eg the average luminance; this
will have the effect of dimming highlights.  We could also apply a rolloff, much as
the 'highlight recovery threshold' slider does in the exposure tab, where the slider
sets the point below which the compression rolls off, and only luminance values above
that point are brought down (in the case of the tonemapping tool, one would want the
reverse -- only values below the threshold are touched).

If one damps out the change in luminance, one largely undoes the effect of the tonemapping
tool.  I'm not sure why one would want to do this.  The whole point is to change the
contrast (global, while preserving local).  As I mentioned in #8, I could see the utility
of a set of sliders along the lines in the S/H tool, that allow one to confine the
tonal compression to a certain range of L values by rolling it off in either shadows
or highlights.  If one were to set the shadow threshold to max, one would obtain the
current behavior where all tones are lifted; but if one set it to a value below midtones,
only shadows would be lifted.  And so on...

Reported by ejm.60657 on 2012-02-05 17:32:07

[Beep6581]

The rolloff would be a welcome addition since tonemapper as-is can easily cause highlight
clipping.

Reported by michaelezra000 on 2012-02-05 18:37:41

[Beep6581]

What would be nice for RT would be the kind of slider that Photoshop has for their layers
options 'blend if' interface -- two sliders, that can be split into a pair, the pair
govern the width of a transition region; one slider pair governs highlights and another
governs shadows.  Unfortunately, I doubt this is in the GTK toolkit.

Reported by ejm.60657 on 2012-02-05 19:07:13

[Beep6581]

Does GIMP have such a widget?

Reported by michaelezra000 on 2012-02-06 00:04:56

[Beep6581]

Emil's proposal looks great. Is it complicated to do?

I don't think Gtk has it (not checke though), but isn't that patented? Adobe has tons
of patent and, maybe i'm too paranoid, but it might be worth to check (if possible).
Anyway, we can do it our way :) ! I can do it a visually different but similar slider
if you really need it.

Reported by natureh.510 on 2012-02-06 00:14:41

[Beep6581]

I would love such a slider for many tools, for instance:

- Confine sharpening to midtones, so as not to sharpen skies or shadows
- Ditto for saturation
- Target NR for a range of tonal values

Surely there are others.

Reported by ejm.60657 on 2012-02-06 01:08:46

[Beep6581]

So this is essentially a simplified version of the flat curve editor we were trying
to create for NR purposes. Why not, it will even take less real estate to be used in
variety of tools. 

This may be a bit off track, but I was also thinking that with new engine would be
great for each tool to support modulation with 
 - blend options (target channel/mode), 
 - opacity and 
 - mask (based on various inputs - painted/luminosity/hue/saturation)
For that we could have a dedicated panel which would allow to change modulation options
for the tool.

Reported by michaelezra000 on 2012-02-06 02:38:23

[Beep6581]

... being used.

Reported by michaelezra000 on 2012-02-06 02:39:49

[Beep6581]

@emil

Do you need to control the shape of the ramp up&down, or does a linear ramp fulfill
your needs? Obviously, it will be a mix between the SHCSelector class and the Flat
Curve type. You can contact me privatly for the feature request of this widget, for
its variants (... pour ses variantes).

Reported by natureh.510 on 2012-02-06 18:06:26

[Beep6581]

Using tone_map1.patch, set Luminance to 1.00, now watch the histogram and enable Avoid
color shift. No change at all in the histogram, yet the preview clearly changes - not
by much, but it does change.

The visibility of the change varies with zoom level. At 25%, 33% and 50% its very visible,
at 100% it's barely noticable. Can't see the difference at all at 10%, 12%, 16%, 20%.

Reported by entertheyoni on 2012-02-07 14:30:40

[Beep6581]

I wonder if this method of cancellation of the luminance change could be used for another
advantage - to actually perform the initial tone mapping. Does it provide some kind
of an algorithmic  shortcut?

Reported by michaelezra000 on 2012-02-07 14:41:20

[Beep6581]

@Hombre: We could try a linear ramp, if the slope discontinuity at the control points
proves problematic it is easy enough to have the control points determine a smoother
function, eg f(x)=(1+tanh((x-a)/(b+eps))) where 'b' is the separation and 'a' is the
average of the pair of control points that determine the transition; 'eps' sets the
minimum width of the transition region.  This is a smoothed out version of a linear
ramp between 0 at 'a' and 1 at 'b'.

Reported by ejm.60657 on 2012-02-07 14:49:57

[Beep6581]

I can't reproduce this for any other photo, and I can't share this one, so ignore it
for now, at least until I find a shareable  photo I can reproduce it with.

Reported by entertheyoni on 2012-02-07 14:52:43

[Beep6581]

@Michael: don't know what the shortcut would be.  The present algorithm separates the
image into lopass and hipass components using an edge-aware blur; then the lopass channel
is tonemapped and the hipass added back in after some user-controlled amplification.
 Sounds like if one really wants to preserve the histogram, one should drop the mapping
of the lopass channel and just apply the edge-aware detail boost to the hipass, and
then it becomes an edge-aware version of contrast enhancement.  This is a part of what
I was suggesting by folding together the high-quality S/H tool and the tonemapping
tool.

Reported by ejm.60657 on 2012-02-07 15:03:50

[Beep6581]

Emil, yes this looks computationally more advantageous - to not to do the tone mapping
instead of restoring from it afterwards.

Topaz Detail, for example, combines tone mapping and detail boosting. Current RT tone
mapper has both functions as well - I see both the boost of local contrast and global
tone mapping. Might be interesting to allow to control these two actions separately.

The new Luminance slider for example could be used to vary the degree at which lopass
data is altered *while* it is being altered, in a single run instead of a subsequent
processing.

Reported by michaelezra000 on 2012-02-07 15:54:11

[Beep6581]

Essentially tone mapper UI can be grouped into controls of 
 - Detail enhancer
 - Luminance mapping 
 - Avoid color shift can do the magic of preventing hue shifts (is this needed for
lopass data only or high pass as well?)

Reported by michaelezra000 on 2012-02-07 16:03:18

[Beep6581]

Emil, I've tried adding the amplified detail component ("high pass") and the result
looks surprisingly similar to applying a curve to the base (what's used is in fact
a mix of the two mentalities, dominantly second). The histogram might differ though,
I don't recall examining that.

I have been using tone mapping a lot over time and to be honest don't feel the need
for what this patch is about. Yes, things don't look perfect with just tone mapping,
but that's what other adjustments are for. With before and after curves, before and
after saturation, and that vibrance thing, I feel that adding something here would
be totally redundant. I never, ever had in mind an all in one "improve the illumination"
tool and believe that's a waste of time.

My take anyway. Thank you all again for making this official, it's seeing much use
at least from me for work and for play.

Reported by nonbasketless on 2012-02-18 06:02:31

[Beep6581]

By the way, sorry for double posting, but Mantiuk did an experiment on a group of volunteers
and found that, on average (emphasis on that!), messing with just the luminance of
CIELab without any adjustments on saturation did the trick best. We can try to get
creative, but ultimately this is going to work in most cases and otherwise is a good
starting point.

http://www.cs.ubc.ca/~mantiuk/pdfs/mantiuk09cctm.pdf

Reported by nonbasketless on 2012-02-18 06:04:59

[Beep6581]

Hi Ben,  I haven't played with it yet, but do you think it unlikely that one could use
your edge-aware blur for S/H 'high quality' tool?

Reported by ejm.60657 on 2012-02-18 14:28:00

[Beep6581]

What I'm saying (in comment 29) is that you can, and that (at least after curve adjustment)
the result is very similar to what there now is.

If you want to play with it though, see ImProcFunctions::EPDToneMap. Set Compression
to zero. Then, adjusting DetailBoost (which currently works in conjunction with Compression)
will do what you're asking.

Reported by nonbasketless on 2012-02-19 22:05:04

[Beep6581]

I tested the latest patch (argh! purged /tmp and lost it, so forgot the name):
- the effects are good, it gets rid of the issue with the original EPD where the whole
image gets overexposed instead of just the dark parts. The effects as-is are so good
in fact that I would ask you to commit it as soon as possible, while you carry on working
in it,
- I always keep the "Equalizer" at 100, I've found no need to drag it down, so no need
for this slider.
- I set the "Equalizer - modulation Highlights" and "Low Lights" sliders at either
0 or 100. The effects at intermediate levels are too minor. If possible, I would like
to see the effect of having their strength doubled, so "50" now would be "25" then.
- I've tried "Reweighted iterates" many times but never found it useful, always keep
it at 0, so no need for this slider.
- Negative values of "Strength" are only usable from -1.00 at best, that slider doesn't
need to go down to -2.0. Even -1.0 is stretching it. This example shows the original
on the left, and strength -1.0 with a some curve-ninja on the right: http://i.imgur.com/WWUex.jpg
- I still cannot explain "Edge Stopping" vs "Scale" to myself, I cannot picture the
effects in my mind. Sometimes I set "Strength" to 2 - this blows the picture up, but
at least I can see the effects of these two sliders clearly.

Reported by entertheyoni on 2012-04-10 22:29:02

[Beep6581]

I found one photo where I had to use "Equalizer" to bring out the texture in a wall,
without it it looked washed out no matter what other combination of sliders and curves
I tried:
http://www.rawtherapee.com/shared/test_images/bressanone_ivy_wall.pef
http://www.rawtherapee.com/shared/test_images/bressanone_ivy_wall.pp3

Reported by entertheyoni on 2012-04-10 23:49:21

[Beep6581]

Hello

I upgraded two things:
1) change the basic settings of "tone mapping"; of course we must adjust differently

I will try with the "pef"

2) Adjusted corrections "Munsell" and "Gamut" for them to take into account variations
in luminance (important here) and chromaticity.
I amended the code of "vibrance"

With these changes (Munsell + gamut), it is possible now - quite simply - to take into
account variations in saturation and luminance elsewhere in the code of RT.

Reported by jdesmis on 2012-04-11 04:59:51

---

• _Attachment: tone2_map21.patch_

[Beep6581]

I changed the settings of "Equalizer"

* Default:  Equalizer=90
* The two sliders "modulation", have their effect increased by 30%

Reported by jdesmis on 2012-04-11 06:15:46

---

• _Attachment: tone2_map22.patch_

[Beep6581]

small improvement, with "HISTORY_MSG"

Reported by jdesmis on 2012-04-11 09:00:47

---

• _Attachment: tone2_map23.patch_

[Beep6581]

Of course I can delete the slider "Equalizer" and set the effect to match 100.
But I would like another opinion

Reported by jdesmis on 2012-04-11 11:11:22

[Beep6581]

Well considering that it did come in useful I no longer think it should be removed,
but I am worried about how complicated using this is... The two biggest issues with
adjusting it are:
1- that white areas appear in the preview which are not visible at 100% or in the saved
image, and
2- that it takes so long to update the preview, which makes me reluctant to spend time
experimenting. How did they get it to update smoothly in real time in the demonstration
video that was on the same page as the paper on EPD? Is that a variation of this algo
or have I confused the two? 

Reported by entertheyoni on 2012-04-11 12:42:47

[Beep6581]

I am not the author of EPD for RT  (it's Ben). I just tried:
1) to give a correct aspect for the images normally exposed (Equalizer)
2) to ensure colorimetry by incorporating a "Munsell" correction.

EPD uses much memory and CPU time. To use it in "debug", one must be patient, by cons
in "release" the processing time become acceptable and EPD gives beautiful images as
output.

Reported by jdesmis on 2012-04-11 13:08:25

[Beep6581]

I think when we implement tile processing, memory requirements will be much reduced.

I tried the patch a little bit, and the modifications in the shadow end are interesting;
I couldn't see much effect on highlights with the one image I looked at, I will try
to do more thorough test later.  I would also like to hear Ben's opinion, it should
carry some weight since he is the author of the tool and presumably set it up the way
he wanted it.

The effects of 'range' and 'edge preservation' are quite different in terms of how
they affect the image blur that is used to do the tonemapping; the first affects the
blur radius (very roughly, since it's not a linear filter but rather inverting a linear
operator on the image) and the second damps the linear operator in regions of high
gradient, making the blur avoid edges.  

Reported by ejm.60657 on 2012-04-11 14:43:09

[Beep6581]

Got your messages, DrSlony.

First, note that I've already commented in this thread. I stand by my belief that you
guys are making things more complicated than it needs to be. I'm not sure what's being
asked of me, but if it's opinion, that's my opinion.

Second, reweighting iterates is useful and I use it, a lot, so please don't remove
it. It produces more natural, less artifacty results in combination with edge stopping
= 1 in most (not all). On the other hand, with zero reweighting iterates edge stopping
= 1.3 - 1.4 gives usually the best results. So both parameters matter.

As for scale, it's hard to put into words but similar to "radius" for unsharp masking.

Reported by nonbasketless on 2012-04-11 20:47:14

[Beep6581]

Oh and by the way, I'd previously compared use of reweighting iterates ("1-norm iterations")
to without:

http://www.rawtherapee.com/forum/viewtopic.php?p=17884#p17884

A few artifacts are worsened, but overall the output is more natural-looking. Please
keep :->

Reported by nonbasketless on 2012-04-11 20:56:38

[Beep6581]

if no remarks, I'll commit to morrow morning (french time) :)

Reported by jdesmis on 2012-04-12 15:44:30

[Beep6581]

It seems to me that the effects of the sliders are redundant to applying a tone curve
to the output of tonemapping, so in this sense the sliders seem to me rather redundant.
 However it also sounds like there are color corrections being applied to prevent hue
shifts that occur with the large changes in luminance involved.  

I am wondering whether it would be better to keep the Munsell hue corrections, but
leave off all the sliders for adjusting the luminance distribution, whose work can
be accomplished using tone curve/L curve.  Perhaps the best thing we could do is to
develop a module that applies such a Munsell correction, either at each step along
the way, or once all luminance corrections are made, rather than having to duplicate
such code in each and every tool.  

Reported by ejm.60657 on 2012-04-12 16:20:12

[Beep6581]

Reported by natureh.510 on 2012-04-12 23:59:51

• Blocking: #130

[Beep6581]

When I tried to change "tonemapping", this is no by pleasure to do otherwise, but because
I think there is a (big) anomaly. 

When you put the cursor "strength" with high values (1.5 ... 2), the histogram shifts
to the right lot. I was inspecting the code "edgepreservingdecomposition", 
and I noticed that the values L (Lab) that are on a photo normally exposed between
0 and 1, are found between about 0.3 and 1.3.
The solution is not in my "equalizer" which is only a stopgap, but in changing the
code "edgepreservingdecomposition" .. But I do not know what to change ...

However the patch with "equalizer" - which is not exactly a "tone curve" because it
compares the image before / after tonemapping - can correct this serious deficiency.
I tried several times by acting on "Lab adjustements", "Exposure - tone curve", "Shadow
highlight" ... down results are significantly worse.
It seems abnormal to deliver a tool that brings dysfunction (significant increase in
L) without making a single paliatif a minimum.

I put on my website, a photo (_ASC4209.NEF) of flower, which can test "tonemapping"
with and without the patch.
http://jacques.desmis.perso.neuf.fr/RT/

For correction "Munsell" I think (check.., for discussion..) that it is better to incorporate
each treatment (vibrance, tonemapping, Lab adjustements, etc..), 
But it will probably remember, if treatment has already been done in the process not
to start again.
But, do we choose "UPLab" which seems to be preferred - because it is simpler - or
my algorithm "Munsell correction" which is more complex, but provides no Clip, no anomalies
or histograms color, and which properly corrects for drift (saturation and luminance).
:)

Reported by jdesmis on 2012-04-13 06:27:09

[Beep6581]

>When you put the cursor "strength" with high values (1.5 ... 2), the histogram shifts
to the right lot. I was inspecting the code "edgepreservingdecomposition", 
and I noticed that the values L (Lab) that are on a photo normally exposed between
0 and 1, are found between about 0.3 and 1.3.
The solution is not in my "equalizer" which is only a stopgap, but in changing the
code "edgepreservingdecomposition" .. But I do not know what to change ...

At what point in the EPD code do you see this change in the L range?  I am somewhat
familiar with this code, having explored it for potential use in NR before settling
on the current method in issue 1052, so I may be able to help.  I would prefer fixing
the problem at the source rather than making a slider whose sole job is to correct
a mistaken output of another slider  >:^(

Reported by ejm.60657 on 2012-04-13 07:24:38

[Beep6581]

Hello Emil

The code I looked is in "EdgePreservingDecomposition::CompressDynamicRange" :

    for(i = 0; i != n; i++){
        float ce = expf(Source[i] + u[i]*(CompressionExponent - 1.0f)) - eps;
        float ue = expf(u[i]) - eps;
        Source[i] = expf(Source[i]) - eps;
        Compressed[i] = ce + DetailBoost*(Source[i] - ue);
    }
I looked at the "min" and "max" values :
Source[i], u[i], ce, ue, Compressed[i]

For example, if Source [i] is between 0 and 1, Compressed [i], for high Strength is
between 0.3 and 1.3 (depending on the images)

Reported by jdesmis on 2012-04-13 08:15:10

[Beep6581]

@ comment 45:
I spent a few hours testing the patch on a number of photos, and I'm quite confident
to say that one can not mimic the effects of the highlights and low lights sliders
using the tone curve. I wouldn't like to see them removed, at least not at this point,
but I am happy to test patches you think are improvements in the usability of this
tool.

Reported by entertheyoni on 2012-04-13 11:20:08