Auto Levels: assymmetric?

[Beep6581]

Originally reported on Google Code with ID 940

Experimenting with Auto levels function in RT, I've observed that in majority of cases
calculated positive exposure compensation leads to unnatural looking results when it
is >0.5EV. If the calculated value of +EC is then decreased to 30-45% (E.g. from +1
to +0.3), this results in a more natural looking image. Calculated value of Black seems
ok.

Reported by michaelezra000 on 2011-08-25 14:42:29

• Blocking: #1130

[Beep6581]

This applies to even the smallest 0.0001 value of the Clip.

Reported by michaelezra000 on 2011-08-25 14:44:26

[Beep6581]

Is the output technically correct (ie is the indicated percentage of pixels clipped,
or is there an actual bug in the code)?  You should be able to check by setting camera
color, then taking the image into photoshop and running a threshold.

Reported by ejm.60657 on 2011-08-27 04:45:51

[Beep6581]

That is my observation too. Auto Levels overexposes when a Clip value is used. I don't
know the technical details but subjectively much heavier clipping occurs in the highlights
than in the blacks. So far I get the best results using a Clip value of 0.0010 and
moving Blacks to about twice the auto value.

Reported by rherink11 on 2011-08-27 21:26:25

[Beep6581]

Emil, I could try this with an artificial image filled with gradient tone from black
to white; to measure the clipping symmetry, one could measure and compare the length
(in pixels) of the dark and bright areas being clipped. Would this be an accurate test
for the clipping/auto levels in RT, or does it have to be done on the raw file from
camera necessarily?

Reported by michaelezra000 on 2011-08-28 01:42:22

[Beep6581]

This would be OK, but we might have to compensate for gamma; twice as many levels in
highlights before gamma is applied.

Reported by ejm.60657 on 2011-08-28 14:00:51

[Beep6581]

...of course the auto levels selects for a specified *percentage* of clipped pixels,
so that is independent of gamma.  

Auto levels is a bit of a crap shoot since it looks at the tails of the histogram,
whereas the perception of brightness of the image has more to do with the central mass
of the histogram.  I don't think that the issue is necessarily with the amount of clipping;
rather, the question is whether the autolevels feature should include some sort of
auto-brightness, that sets the brightness slider so that the mean of the histogram
is at eg middle grey.

Reported by ejm.60657 on 2011-08-28 14:42:31

[Beep6581]

I created a linear Gray-16 file for testing but RT cannot open it.

Are auto levels currently being applied in linear gamma? 
If not and gamma is not compensated for, then, this is probably the reason for the
asymmetric clipping. 

Reported by michaelezra000 on 2011-08-28 14:50:43

[Beep6581]

auto levels are computed from the histogram, in linear gamma.

Reported by ejm.60657 on 2011-08-28 15:12:33

[Beep6581]

Here is a couple of images from the HDR set.
When both are set to a neutral profile + auto levels with clip=0.0002 The originally
underexposed image is developed to be much brighter than the properly exposed image.

http://min.us/m8lKlUqlB

This illustrates some issue with the auto levels. 
Should this be remedied with the negative brightness for the underexposed /auto-leveled
image or by a smaller +EC than calculated by auto levels?
Adjusting the RGB brightness changes saturation, so probably a smaller +EC would be
a better adjustment.

I suppose the auto brightness function should be implemented not via RGB brightness
(slider) but via a further adjustment of the EC that is determined by the auto levels.

Reported by michaelezra000 on 2011-08-28 16:01:31

[Beep6581]

Michael, your image at the link is not showing (at least not on Firefox or Safari).

If you are doing HDR, I don't think the development should be using autolevels. Rather
I would do a conversion with the neutral profile and linear gamma on output; then combine
the images with tone mapping in some HDR software, and do final processing including
gamma in photoshop. 

The point of auto levels is to stretch the tonal levels based on not clipping too much
in shadows or highlights; its purpose is not to set the midtone brightness level. 
We can offer another feature to do that, or incorporate it in autolevels, but it is
a change in the original intent which is worth discussing more widely.  I don't think
that it is wise to use exposure to set midtone brightness, since it risks clipping
too much in highlights.  There might be an alternative using highlight recovery --
use the exposure slider to set the midtone brightness, then the recovery slider to
restore clipped highlights; if that is a more desirable method than a linear stretching
of levels based on the endpoints of the histogram.

Reported by ejm.60657 on 2011-08-28 20:34:04

[Beep6581]

Hi Emil, I will upload new files tonight.

You are correct about the HDR workflow. Here I used images from HDR set to verify that
auto-levels could make them look the same via +EC. There was a small difference between
the images in the original pair that I used for testing - a few pixels of the bright
sky. That caused a difference in auto-levels calculations for the pair when a small
value of clipping was used. When clipping % was large enough to consume that difference
auto levels produced very comparable result. I will verify this tonight by shooting
identical images at various EC on camera.

Reported by michaelezra000 on 2011-08-29 12:16:27

[Beep6581]

>There might be an alternative using highlight recovery -- use the exposure slider to
set the midtone brightness, then the recovery slider to restore clipped highlights;

I will experiment with this approach. 

I tend to keep highlight recovery at 70 with a threshold around 50 for most images.
With that in place, +EC that is calculated by current implementation of auto levels
sometimes is too aggressive for the midtones. That is why I suggested *reducing* the
calculated +EC (this cannot lead to additional clipping, vice versa).

Reported by michaelezra000 on 2011-08-29 16:10:38

[Beep6581]

Here is a set of two images, shot 3EV apart: http://min.us/mk5PsE8M3
This set does not demonstrate the problem I mentioned earlier, but there are additional
observations: 

CASE A: Clipping=0.0000 autolevels=ON HLRecovery=0 
   AutoLevels_01-3EV   EC=2.26
   AutoLevels_01       EC=0.56
   thumbs are the same. 
   Preview of AutoLevels_01-3EV.NEF is *darker* then AutoLevels_01.NEF
   (this is another issue?)

CASE B: Clipping=0.0001 autolevels=ON HLRecovery=0
   thumbs are the same, matching previews (previews are the same)

CASE C: Clipping=0.0001 autolevels=ON HLRecovery=70
   AutoLevels_01-3EV   EC=3.67
   AutoLevels_01       EC=0.74
   identical effect in thumbs vs previews, but
   in AutoLevels_01-3EV.NEF highlights are noticeably flatter;
   Is this an expected difference due to a different +EC value between the images?

Reported by michaelezra000 on 2011-08-30 03:43:55

[Beep6581]

Using Lab brightness for changing midtones works very well on a variety of images I
tested with, including skin tones (e.g. the old "eye brow" shot). Negative values of
RGB brightness seem to change color appearance differently, resulting in a not clean
color look.

One use case which may not be intuitive is when highlight recovery is set to >50 and
+EC is high (>1.5EV), image might get a flat saturated look. Playing with brightness
does not cure this at all and this is what I was trying to do initially. The correct
thing to do in such case is to adjust EC and highlight recovery to fix the flatness
and only then adjust the midtones.

Reported by michaelezra000 on 2011-09-02 01:46:49

[Beep6581]

Could we allow a negative value in the Clip?
Even 0.0000 is too tight for images that will go through post processing.
It would be helpful for auto levels to be able to stretch image to *almost* the full
range, leaving some space based on -Clip value.

Reported by michaelezra000 on 2011-09-02 15:59:53

[Beep6581]

My current workaround: an equivalent to negative Clip is to set -Lab contrast value.

Reported by michaelezra000 on 2011-09-02 16:35:20

[Beep6581]

Emil, is this correct (improcfun.cc line 658):

int maxaw = (gavg - bl) * 4 / 3 + bl; // dont let aw be such large that the histogram
average goes above 3/4

should this instead be 
int maxaw = (gavg - bl) * 3 / 4 + bl;

I tried to change this but cannot detect any difference.

Reported by michaelezra000 on 2011-09-04 17:28:33

[Beep6581]

Hi Michael,  I think it's correct as written.  awg is the gamma corrected white point;
gavg is the average (midpt) of the histogram; maxaw asks that 4/3 the average be less
than the white point, ie that the average is less than 3/4 of the white point.

Reported by ejm.60657 on 2011-09-04 20:28:38

[Beep6581]

Here is a patch to change the behavior of auto exposure.  Black is adjusted as before,
so that the specified percentage of pixels is clipped; but instead of doing the same
in highlights, instead the exposure is adjusted so that the median of the histogram
is set to middle gray.  

The next step is to adjust the highlight compression so that highlights are not clipped.
 I have worked out the math, I just need some help getting the UI to update to reflect
the new autoadjusted highlight recovery value.

Reported by ejm.60657 on 2011-10-26 02:25:44

---

• Attachment: autoexp-5-nocmakefix.patch
• _Attachment: autoexp-6noHLR_nocmkfx.patch_

[Beep6581]

Seems that callback to UI is done in improccoordinator.cc via
if (aeListener)
                 aeListener->autoExpChanged (params.toneCurve.expcomp, params.toneCurve.black);
         }

so it needs an extra parameter   params.toneCurve.hlcompr

Reported by michaelezra000 on 2011-10-26 02:47:14

[Beep6581]

I will try to make this change to autoexp-5-nocmakefix.patch and upload

Reported by michaelezra000 on 2011-10-26 03:05:51

[Beep6581]

Hi Emil, I did a quick test the hlrecovery seems very aggressive, but this patch should
let you see its values in the UI.
(until tomorrow, must run now)

Reported by michaelezra000 on 2011-10-26 03:46:44

---

• _Attachment: autoexp-5-nocmakefix_withUI_01.patch_

[Beep6581]

UI patch works well, I must have a math mistake in the computation of the HLRecovery
amount, because as you say the highlight compression is way too much.  BTW, the UI
should specify a default recovery threshold too; I am using 50 in my computations,
but I am open to suggestion as to the proper default.  A value of 50 means that only
the top two stops of data are compressed.

Reported by ejm.60657 on 2011-10-26 16:40:13

[Beep6581]

"A value of 50 means that only the top two stops of data are compressed."
That's a good explanation. If you make it a slider, could you call it that and change
50 to 2?

Does the same apply to other sliders currently in RT? If so, I also vote for using
stops instead of 0-100 (or whatever the upper limit).

Reported by entertheyoni on 2011-10-26 17:25:04

[Beep6581]

Do you really want the effect to grow exponentially with the slider position?  That
makes the control very difficult to manage when far away from small adjustments from
zero.  Is there really a problem with the smoothness of the slider as currently configured?

Reported by ejm.60657 on 2011-10-26 18:25:24

[Beep6581]

I wonder if it would make sense to set the threshold dynamically based on how much is
being recovered: threshold should be higher value for higher value of the hlcompr.

Reported by michaelezra000 on 2011-10-26 20:42:26

[Beep6581]

Emil: no, I didn't know it would. No problem with smoothness, just "2EV" sounds more
meaningful to me than "50", but now I see why it is like that.

Reported by entertheyoni on 2011-10-27 06:21:32

[Beep6581]

A patch which fixes the HLR behavior.  Some examples don't seem to be giving middle
gray as the median image value however and I need to find out why.

Reported by ejm.60657 on 2011-10-28 03:35:34

---

• Attachment: autoexp-7nocmkfix.patch

[Beep6581]

hlcompr values are much better following the +EC values now.

But I think this might need to consider an additional factor - global image contrast.
In  some cases +EC and consequently HLR lead to very flat over-saturated images.

If I were to adjust such images manually, I would use more conservative EC (even with
clip=0).

May be midpoint is better to be fine-tuned with RGB brightness, as you suggested before?

Reported by michaelezra000 on 2011-10-28 12:43:09

[Beep6581]

An advantage of the new EC calculation is that it can produce negative EC values, it
is just too aggressive for +EC.

Interestingly, when image is excessively brightened, no pixels seem to remain in the
very left area of the histogram. May be +EC should be also guided by changes made to
the very dark tones, to allow them to remain dark. This can prevent excessively high
+EC

Reported by michaelezra000 on 2011-10-28 13:15:27

[Beep6581]

I think the problem is that I'm computing the median value in linear gamma; it should
be computed in the output space.

I'm thinking that, rather than clipping percentages for the upper and lower ends, we
should do something more 'zone-based'.  For instance, the proposed change is to put
the average or median luminance at middle gray.  Then for highlights and shadows, rather
than specifying a certain percentage of pixels to be clipped at black or white, wouldn't
it be better to say we want some specified percentage of pixels in Zones 0 and X, or
similar?  If there were a more sophisticated analysis of the histogram, one might get
more toward what metering systems do.  

Reported by ejm.60657 on 2011-10-28 13:53:29

[Beep6581]

Next iteration.

Reported by ejm.60657 on 2011-10-28 21:02:24

---

• Attachment: autoexp-8nocmkfix.patch

[Beep6581]

Another update.  I discovered a bug in the RGB brightness/contrast when nonzero values
of exposure compensation or blacks are set.  This should fix that.

Reported by ejm.60657 on 2011-10-28 23:52:44

---

• Attachment: autoexp-9nocmkfix.patch

[Beep6581]

Interesting effect with this auto levels. Sometimes though it leads to bad results -
in most cases the problem is with a too low value of Highlight Recovery Amount, in
more rare cases the exposure value itself is to blame.

Try these photos:

1- http://www.rawtherapee.com/shared/test_images/fruitbowl.pef
Generally underexposed when using auto levels.

2- http://www.rawtherapee.com/shared/test_images/woodstock_people.pef
Their skin gets severely overexposed

3- http://www.rawtherapee.com/shared/test_images/renon_earth_pyramids.pef
Same as with woodstock_people, similar colors and similar overexposure.

I don't expect it to work great in every case, but I pointed those photos out above
in case you want to tweak it further.

Reported by entertheyoni on 2011-10-29 01:14:08

[Beep6581]

Slight improvements with this patch.

Slony, this patch does a reasonable job with your images, probably the worst is the
people shot.  This version of auto levels tries to set the average level to middle
gray.  So if the background is high key, as in the still life, the subject will be
rendered too dark; while if the background is low key, such as the people shot, the
subject is going to be rendered too light.  I don't see how one can do better using
just the luminance histogram for data and an auto tool, which is no substitute for
user input.

Reported by ejm.60657 on 2011-10-29 03:06:31

---

• Attachment: autoexp-10nocmkfix.patch

[Beep6581]

Just got back to a computer, but did not get a chance to test yet...
On subject of determining scene-dependent exposure, there are different metering systems
employed in cameras, spot, central (with options of radius size), evaluative (various
number of segments with weights) and probably few others. With the face recognition
technology one could use that as an additional factor as well... I suppose future improvements
in RT could benefit from some of these approaches. The central-area-based metering
could be the easiest option to introduce. One could also think of a user-selectable
area (similar to earlier Guillermo's suggesstion for white balance).

This could evolve into a dropdown control to let user pick most suitable metering method
either per image and a global default in preferences.

Reported by michaelezra000 on 2011-10-29 03:37:48

[Beep6581]

I'm trying this out and it seems to be a clear improvement so far.

Reported by nobrowser on 2011-10-31 05:35:14

[Beep6581]

Next round: Brightness adjustment to help center the histogram when using exposure compensation
will blow highlights.

Reported by ejm.60657 on 2011-10-31 12:22:16

---

• Attachment: autoexp-11nocmkfix.patch

[Beep6581]

Some feedback on version 11: 

1. In many occasions auto levels give nice results, but usually low key images get
too bright (e.g. night image with red car DSC_6504.NEF, which would look great with
auto levels and brightness=0 - see illustration in uploaded jpg, right bottom)

2. Thumbnails (existing and newly generated) sometimes become white or black with minimal
changes to EC.

3. I suppose if image has clipped highlights auto EC should not be set to positive
values (e.g. King image)? Unless it is compensated by hlcompr... Is this how it works?

4. If clipping = 0.0000 would EC be adjusted to let max just reach 65536 without clipping?

5. While testing, I found an interesting (already existing, not patch related) discrepancy
in color rendering between preview and thumbnail (I use 16 bit thumbs). If this jpg
(http://imgur.com/cU15z) is set to +3 EC, the letter A has more natural colors in thumbnail
vs preview. Same effect is shown on the car image (3).

Reported by michaelezra000 on 2011-11-01 02:55:17

[Beep6581]

I agree there is a problem with low-key images and high-key images too.  Dark or light
backgrounds then dominate the histogram, and the EC latches onto that and tries to
drive it to middle gray which then makes the subject too light or dark.  So the basic
question is what quantitatively characterizes such images via the histogram, that would
distinguish them from a normal image that is simply under- or over-exposed?

As for the thumbs, there is clearly a bug there which didn't exist before I introduced
the brightness slider modification.  Don't know what that is about.

The way the algorithm works is to compute the EC (call it gain1) that would set the
average of the histogram to middle gray; there is also another gain (call it gain2)
that puts the white clipping point (with the specified clipping percentage) at 65535.
 Then the chosen EC is sqrt(gain1*gain2) the geometric mean of the two gains.  This
provides a compromise between average brightness and clipping that for 'normal' (not
high- or low-key) images works rather well.  The highlight compression is then approximated
to the value that sets the clipping point to 65535 after the EC is set.  This doesn't
work so well if the top highlights are saturated, then one or more channels blow before
the luminance reaches 65535; I think this needs work, in particular use highlight reconstruction
rather than clipping to determine the top end of the histogram, especially if the default
is to use autoexposure in conjunction with highlight reconstruction.  Anyway, the new
brightness adjustment takes the image after EC and tries to push the average value
of the histogram the rest of the way to middle gray using the brightness slider.

As for #4, no the EC is now largely determined by the compensation that would move
the average brightness to middle gray.  The problem with using the clipping point is
that you let a handful of specular highlights determine the EC, rather than the visually
more dominant overall brightness of the image.  It is this issue that I am trying to
address with this patch -- get the average brightness more toward middle gray, then
use highlight compression to bring any stray clipped highlights back into the 0-65535
range.

Reported by ejm.60657 on 2011-11-01 03:36:16

[Beep6581]

I think the low vs high key histogram *shapes* would look like in the image below. So
low key has a higher percentage of darker pixels and high key is the opposite.

Reported by michaelezra000 on 2011-11-01 03:53:16

---

• _Attachment: Low_highkey_01.jpg_

[Beep6581]

Not sure whether it is better to have some progressive measure of low vs high key to
drive the value for the brightness slider. For example, image with 1/2 bright sky and
1/2 dark mountain. Then the same image but with slightly changed composition when percentages
deviate from 50%. I suppose the current math for the brightness slider could use a
new multiplier factor based on analysis of the shape of the histogram.

About the thumbs vs preview difference - I verified that this exists in earlier RT
versions, so it is not due to this patch. But the difference is seen on the exposure
slider (not the brightness slider). Its a pity that the bug is in the preview and the
final output as thumbnail actually looks better:)

Reported by michaelezra000 on 2011-11-01 04:03:40

[Beep6581]

P.S. about my last paragraph - I got confused (incorrectly) assuming that your 2-nd
paragraph was the answer to #5.

Reported by michaelezra000 on 2011-11-01 04:09:12

[Beep6581]

Median vs mean difference could probably provide a measure of low vs high key characteristic
of the image, as absolute difference will increase when distribution deviates from
normal distribution (approximation of a "regular" image).

"The median is better suited for skewed distributions to derive at central tendency
since it is much more robust and sensible."
(http://www.diffen.com/difference/Mean_vs_Median)

I suppose that this new factor would need to be applied in some degree to both EC and
brightness.

Reported by michaelezra000 on 2011-11-01 11:52:13

[Beep6581]

Try changing lines 717 and 738 of improcfun.cc to

        gain = (median/ave)*sqrt(gain*scale/rawmax);

        bright = (median/ave)*MAX(0,bright);

That seems to help. Still a couple of images get weird brightness values, we may need
to limit the excursion of the brightness slider.  This is just a first attempt; there
are also in the current code parameters lodev and hidev that measure the deviation
from the average on the high and low side (easily modified to deviation from median
if we desire) that will measure how skewed the histogram is.  Currently these parameters
are not used, but could be if they accurately reflect histogram shape.

Reported by ejm.60657 on 2011-11-01 14:23:19

[Beep6581]

Alternative to changing 717 is to substitute 'median' for 'ave' in line 715.

Reported by ejm.60657 on 2011-11-01 14:35:26

[Beep6581]

ok, will try in the evening

Reported by michaelezra000 on 2011-11-01 14:48:12

[Beep6581]

Possible problem with the patch:  I don't think auto exp os taking crop into account
now.  I think that's a regression.

Reported by nobrowser on 2011-11-01 19:35:48

[Beep6581]

FYI, there another issue on this in google code

Reported by michaelezra000 on 2011-11-01 19:50:52

[Beep6581]

The (median/ave) seems useful, especially for the exposure.
Exception is that large clipped areas are usually set with L around 90.

For brightness I also changed ave to median in this line (my line numbers got a bit
shifted, so cannot quote):

    float midtmp = gain*(median)/scale;

I substituted ave to median in all calculations for lodev and hidev

In most cases 

  bright = X*(median/ave)*MAX(0,bright); 

seems to work where X is between 0.25 and 1; I am still trying to find correlation
with lodev and hidev

25*(lodev/hidev) seems to be a good value for RGB contrast, I suppose we could drive
this parameter as well.

For simplicity, here is the improcfun.cc ( I also added some printf statements that
could be useful for testing.

Reported by michaelezra000 on 2011-11-02 03:41:00

---

• Attachment: improcfun.cc