Add new "perceptual" curve type

[Beep6581]

Originally reported on Google Code with ID 2837

I'm thinking of adding a new curve type "neutral" with the follwing properties:

- Constant color appearance despite contrast change
    - That is take into account human vision phenomena such as that
      higher contrast requires higher saturation to look the same
- Working space independent, that is regardless of working space the same
  result is had (clipping issues excepted)

I've done this type of work for my camera profiler DCamProf already so I'll see if
I can make a lighter weight realtime version of that. A discussion about how tone curves
affect color apperance can be found at:

http://www.ludd.ltu.se/~torger/dcamprof.html#tone_curves

Why is this needed? RT is designed to use "linear colorimetric" profiles, that is profiles
without any curve applied. This only makes sense if RT can retain the color appearance
when a curve is applied, and today there is no such curve. Weighted standard and luminance
indeed come quite close but it's possible to make an even better result.

Commerical raw converters generally include the curve in the profile itself and then
provide color rendition as a complete package with the curve and all, so the need of
a neutral tone curve is specific to RT through the use of colorimetric profiles.

Reported by torger@ludd.ltu.se on 2015-07-11 09:24:45

[Beep6581]

I've started this and come quite far.

Reported by torger@ludd.ltu.se on 2015-07-19 07:56:30

• Status changed: Started

[Beep6581]

Here's a patch.

The curve works as follows:
 - Always works in prophoto space to avoid different (bad) results when changing 
   working space. Translates back/forth if working space is different
 - Measures a contrast value from the tone curve, higher contrast = higher value
   it looks at the midtones only.
 - Uses a Luminance curve as basis and will for typical curves be very similar to
   it, but with slightly more saturation
 - Scales chroma in CIECAM02 JCh space, higher contrast value => higher chroma,
   which corresponds to human vision (more global contrast requires more saturation

   for the colors to appear the same)
 - Less chroma scaling of very saturated colors
 - More chroma scaling of very dark colors
 - Low contrast levels, less than 1 will instead lead to desaturation to keep 
   constant color appearance.

The model has been tuned after my eye. It's not a perfect model. You can probably with
really strange curves get an unsuitable contrast value detection which will lead to
a too saturated or desaturated look.

The performance is naturally considerably lower than the other curves as we need to
pass through CIECAM02.

The model can surely be further refined, but if we think it's at a "good enough" level
I'd like to introduce this curve soon, as I think RT needs this type of curve "by principle",
as discussed in #1.

Reported by torger@ludd.ltu.se on 2015-07-19 19:24:46

• Status changed: PatchSubmitted

---

• Attachment: ptc-patch1.txt

[Beep6581]

My intention is that it should work really well for typical camera "film curves", that's
where most effort of balancing the algorithm is put, and for extreme/creative curves
I'm happy if it works just decent.

The rationale is that this type of curve would be used to establish a baseline look
when starting with a curve-less colorimetric camera profile, where you most likely
want to have a neutral realistic color appearance, and then maybe use other curves
and tools to add a creative look on top.

Reported by torger@ludd.ltu.se on 2015-07-19 19:38:16

[Beep6581]

Hi Anders, thank you for working on this "holy grail" feature.
Some findings: 
1. when applying the new curve to tif image the image becomes white.
2. perhaps the effects of de-saturation an saturation are too strong. the darkening
becomes lifeless and too desaturated and brightening over-saturated. 
3. whn applying the S-shape curve, the brightening action leads to flatter appearance
of highlights, similar to the action of L-curve. The Film-like mode with the same curve
gives more delicate and detailed rendering of tones (aside of possible ugly orange
colors when over-used).

Reported by michaelezra000 on 2015-07-19 20:35:06

[Beep6581]

Here's a fix so it should work with TIFF files.

#4 I'll look and think about your comments and come back.

Reported by torger@ludd.ltu.se on 2015-07-19 21:25:51

---

• Attachment: ptc-patch2.txt

[Beep6581]

#4

(When you evaluate, if you're not already doing it concentrate on checking if the result
is accurate rather than pleasing, by comparing with the original image and thinking
away the contrast. Maybe you're already doing that but just to make sure, otherwise
we'll just be discussing taste.)

On point 3, can you specify the reason why highlights are flatter? I know of one reason
myself -- bright saturated colors are more saturated than the film-curve (as it's more
correct), but it also leads to more clipping, especially of the blue channel. This
clipping can lead to less detail in the uppermost highlight range. Is this the problem?
If so I have a solution which I already use in DCamProf, which limits saturation in
the uppermost highlight range, but not to become as desaturated as the film-curve which
I think is far too much, as it renders light blue skies almost gray.

Reported by torger@ludd.ltu.se on 2015-07-19 21:37:47

[Beep6581]

I also have an impression that it is flatter because of clipping, likely a blue channel:)

Reported by michaelezra000 on 2015-07-19 21:42:22

[Beep6581]

#4, point 2.

I'm not 100% sure what you mean with "darkening and brightening". What the algorithm
looks at is contrast, either the contrast is higher than a straight diagonal, or lower.
Except from a couple of fine adjustments it does not treat brightening or darkening
of colors separately. If global contrast increases, the global saturation increases,
and the other way around.

Say if we apply an S-curve, then saturation is increased for all colors, both light
and dark. There are two adjustments though, extremely saturated colors are limited
(regardless of lightness), and very dark colors get an extra boost in saturation.

That extra boost is meant to battle the "grayness" you with a Luminance curve when
applying a high contrast curve where you darken a lot close to black.

There may be some rebalancing required there, but I'd like to have a clearer description
of point 2 or an image example before I start experimenting.

Reported by torger@ludd.ltu.se on 2015-07-19 21:50:52

[Beep6581]

#7: I'll start working with the highlight thing first then. I can't steal the solution
straight off from DCamProf unfortunately as it's not suitable for realtime, but I'll
come up with something.

When it comes to balancing the constants for tuning the current behaviour it's quite
easy to do, it's just changing some numbers in curves.cc. I could comment the code
in more detail if someone wants to play around. The function to look at is PerceptualToneCurve::Apply
in curves.cc which contains the fine-adjustments and PerceptualToneCurve::init() which
specifies the contrast-value-to-chroma-scaling spline. A typical camera S-curve has
a contrast value of about 1.5 or so (1.0 is linear), so if you change the 1.80, 1.22
curve to say 1.80, 1.18 then there will be less chroma increase.

Reported by torger@ludd.ltu.se on 2015-07-19 21:58:47

[Beep6581]

Let's use this image for testing as it has broad range of subject matter:
http://www.michaelezra.com/Projects/RT/images_shared/RT_tonecurves_Neutral_sm.zip

I see that clipping is mostly in red channel when S curve gets applied (you can enable
clipping indicators and see individual channels in preview via shortcuts r,g,b) May
be better to see that this can get resolved and then look at calibrating saturation
vs contrast?

By darkening I mean curve with one user-added point, which is below the diagonal. Brightening
- the same point is above the diagonal.

Reported by michaelezra000 on 2015-07-19 22:10:56

[Beep6581]

BTW, I use the portrait images from #10 to evaluate the curve on skin tones.

Reported by michaelezra000 on 2015-07-19 22:13:37

[Beep6581]

Great test image!

Ok, I see. Yes I think it's better to first tune the highlight clipping issue.

The one-point curves are a bit tricky concerning contrast value, there may be an overestimation/underestimation
there. Contrast value seems to work best on s-curves.

To not end up with a very complicated curve detector, we may need to compromise performance
on some curve types. The S-shaped camera-like curve is my highest priority. Now the
curve detector is a fairly simple measurement on curve slope, but it may be the case
that to get perfect performance one need to manually tune a value for hundreds of curves
and put in a database to compare to.

Reported by torger@ludd.ltu.se on 2015-07-19 22:32:01

[Beep6581]

Hmmm... I think I need to make an adjustment of the contrast value detector after all.
But now sleep. 

Reported by torger@ludd.ltu.se on 2015-07-19 22:54:09

[Beep6581]

I'm tuning the contrast value detector now. There's one aspect that I want to exclude,
namely image analysis. Here we only look at the curve, and individual pixels isolated
from eachother.

One problem is though that a darker image may seem to have lower contrast than a brighter
image, despite exactly the same curve, that is if you pull the exposure slider the
contrast may appear to change. To solve that you would have to scan the whole image
after applying the curve and then make a contrast analysis on that. It's not impossible
but that would make it quite a lot more complicated and intrusive code.

For now I think it's overkill, and I'll stay with curve analysis only.

Reported by torger@ludd.ltu.se on 2015-07-20 08:22:01

[Beep6581]

Here's a new patch with debug features:
 - Saturation slider is disabled, instead it will set the chroma scaling factor
   of the Perceptual curve if nonzero. It's 1.0 + sat / 100.0, that is sat = 17 =>

   1.17 scale factor, -17 = 0.83
 - detected contrast value and resulting chroma scaling is printed on the console

Extreme highlights are now limited in saturation by gradual mixing in of the Film-like
curve, this leads to whiter highlights. At most 80% of the Film-like curve is mixed
in.

Minor adjustment of contrast value detector.

So far no adjustments of weights.

More comments in the code so one can see how the curve works (see curves.cc)

Reported by torger@ludd.ltu.se on 2015-07-20 12:28:01

---

• Attachment: ptc-patch-debug-3.txt

[Beep6581]

Comments on skin tone rendering:

- My focus is on keeping color appearance of original image, pleasing rendering of

  skin is somewhat secondary, but skin tones are great to look at to evaluate 
  saturation etc as the eye is extra sensitive in that range.
- I don't think the most subjectively pleasing skin tones can be combined with the

  most "accurate" curve in all cases.
- Observed problem: high key portraits with lots of clipping (in red channel) has 
  now got an improved rendering by the latest patch, however making a pleasing clip

  transition and keeping accuracy in highlights are conflicting goals. A portrait 
  photographer may need more of the pleasing clip transition with a longer range of

  desaturation, and a landscape photographer (like myself) appreciate bright sky 
  color accuracy more. I think the latest patch makes a decent tradeoff, but you may

  still prefer a pure Film-like curve for a high key portrait.
- When using a one-point darkening curve I sort of observe the same as #4 point 2,

  that is we get a certain grayness, however while more saturation may be more 
  pleasing for certain portraits it seems less accurate when comparing to the 
  original.

When comparing with the original I recommend using the exposure slider on it to adjust
midtone brightness so it matches midtone brightess of the curve image, it becomes much
easier to compare.

I'm now going to have another look on the weights.

Reported by torger@ludd.ltu.se on 2015-07-20 13:09:15

[Beep6581]

Reduced saturation increase for high contrast curves.

Reported by torger@ludd.ltu.se on 2015-07-20 13:35:00

---

• Attachment: ptc-patch-debug-4.txt

[Beep6581]

The weighted-standard still produces a more natural gradation in skintones.
http://michaelezra.com/Projects/RT/images_shared/rt-ptc-4_1.jpg

Reported by michaelezra000 on 2015-07-20 14:06:38

[Beep6581]

It might be due to over-desaturation of shadows and over-saturation of highlights when
S curve is applied.

Reported by michaelezra000 on 2015-07-20 14:15:01

[Beep6581]

BTW, use of 2-nd curve in perceptual mode leads to strong artifacts

Reported by michaelezra000 on 2015-07-20 14:16:21

[Beep6581]

Fixed second curve issue

Reported by torger@ludd.ltu.se on 2015-07-20 14:37:56

---

• Attachment: ptc-patch-debug-5.txt

[Beep6581]

#18. There is a problem of comparison. Many of the portrait images are high exposed,
so when you apply a curve that retain color, like perceptual, it will clip. The perceptual
curve keeps the color higher up in highlights than other curves, and I'm a bit reluctant
to changing that, as it will hurt landscape performance.

The first portrait image clips a lot, while the second is within range which better
shows the look of the Perceptual curve. You can experiment with some negative exposure
and see what you get.

The weighted standard curve desaturates and changes skin color a little. If we look
at the first lady, there's a little red in her skin, which weighted standard has turned
into a more golden tone. I certainly agree that it's more pleasing for that portrait,
but it's less true to the original.

In your example, apply -0.3 stop on your exposure slider and look at the first lady
again. Compare with the original.

Also note that a luminance curve will darken more than weighted standard, ie you get
more contrast with the same curve. You probably need a different curve for best result.

However I could try to experiment and use the weighted standard as base in terms of
contrast (instead of luminance curve), it may actually be a good idea. That would make
the effect of the shadow dip less dramatic.

Reported by torger@ludd.ltu.se on 2015-07-20 14:50:17

[Beep6581]

I'm going to try using Film-like as reference for the luminance level. I do see a value
in being as compatible as possible with the curves people use.

Film-like, standard and weighted standard all have similar contrast, while the remaining
three make darker shadows.

Reported by torger@ludd.ltu.se on 2015-07-20 15:07:57

[Beep6581]

Here's a new version, now with the curve luminance based on Film-like. It makes a quite
big difference in result.

Reported by torger@ludd.ltu.se on 2015-07-20 15:26:03

---

• Attachment: ptc-patch-debug-6.txt

[Beep6581]

Tested a bit and I think both highlight rendering issues and "grayness" disappeared
or at least greatly reduced now with the luminance based on film-like. What do you
think?

Reported by torger@ludd.ltu.se on 2015-07-20 15:34:10

[Beep6581]

One visual aspect that can be noted in some images that we can't do much about - if
there's a large colored flat surface such as a smooth portrait background or a clear
sky, it will look a little bit more saturated than it should.

The reason is that we increase saturation based on the curve's contrast assuming "average"
picture content. However, the eye is sensitive to local contrast, and a large flat
surface has low contrast even after we have applied an S-curve thus it will receive
more saturation increase than ideal, as the local contrast is still low. To make a
perfect result we would then need to make an image evaluation with local adjustments,
which could be future work but for now I'd like to keep it simple.

For these visual phenomena I've made the design choice rather a bit too much saturation
than too little, as most users prefer a slightly saturated look anyway.

Reported by torger@ludd.ltu.se on 2015-07-20 15:48:08

[Beep6581]

I confirm the improvement in the highlights!
The shadows, however still exhibit that grayness. If you could tweak that to behave
better than Film-like curve that, this could be Awesome. 
Shadows seem to look better with the Perceptual curve when saturation slider is around
35-40.

Reported by michaelezra000 on 2015-07-20 19:03:16

[Beep6581]

Thanks for testing, it's good to have some extra eyes. I'm not sure I see the grayness,
could you post an image with a problem area encircled?

Here's one image with ptc-6 to the left, neutral +0.2 exposure middle, and ptc-6 saturation
+35 on the right:

http://torger.dyndns.org/rt-bugs/ptc6-neutral-ptc6-sat35.jpg

(note that when displayed side by side rather than "swap-and-let-eye-adapt-for-a-few-seconds"
the difference in saturation is exaggerated.)

Reported by torger@ludd.ltu.se on 2015-07-20 21:54:35

[Beep6581]

Thanks for developing this new curve type - the results from your test pictures look
indeed very promising.

Two remarks:
a) I think it would be generally preferable to have a curve which adds as little saturation
as is necesssary to get a perceptually correct result. Such a more neutral curve would
be more universal in its application. In some of your examples skin tone saturation
is pushed too much IMO. The problem is you can't easily get rid of this extra saturation
without affecting other colors. On the other hand, if you start with less saturation
you can always add it with e.g. CIECAM chromaticity or saturation if needed but use
the skin tone protection slider to retain neutral skin tones.

b) you wrote about using the CIECAM module to do the saturation calculations. I noticed
here https://code.google.com/p/rawtherapee/issues/detail?id=2819#c47 that CIECAM is
showing some color saturation anomalies after the latest patches. I don't know if your
calculations are affected by this, just a note of caution that the results you rely
on for the tuning may be skewed by this probable bug.

Reported by stefan.ittner on 2015-07-21 08:02:18

[Beep6581]

#30

a) Yes the intention is not to add too much saturation, except for the visual phenomena
described in #27. When working and evaluating this type of curve one need to keep in
mind that there is no such thing as a perfect result as the eye/brain's visual response
is very complex. I'm not trying to make an iCAM ( https://en.wikipedia.org/wiki/ICAM_(Color_Appearance_Model)
), but it's still good to be aware of some of the spatial effects of human vision to
know which problems that can be solved and not. We're not going to have perfect result
for all images.

Comparing pictures side by side that you can do in #29 is actually a rather bad idea
(next time I'll post separate images), when I do real evaluations I do swap-and-adapt.
There's been a few cumulative patches now so there can be a need for rebalancing though.
I'll look at that again, but first I'd like to have a clarification of the grayness
issue seen by Michael.

b) I'm using LCMS's CIECAM02 implementation for now. At some point it should move to
RT's own implementation, or even a custom one to gain some speed (as it uses a subset
of functionality).

Reported by torger@ludd.ltu.se on 2015-07-21 08:43:20

[Beep6581]

Here's a new debug version where you can control the shadow saturation boost, using
the hl compression threshold slider (make sure hl compression is at 0). There was a
saturation boost there before as well, which was 1.12, replicated with hlcomprthres
slider at 12. I have extended the range so a wider shadow range is covered by this
boost.

Michael, can you play with this and see if you get a good result? I think a value between
12 and 24 look okay, 0 is definitely too gray, you don't need skintone to see that.

Reported by torger@ludd.ltu.se on 2015-07-21 09:56:04

---

• Attachment: ptc-patch-debug-7.txt

[Beep6581]

Anders, sorry for delayed response. 

A. With HL compression threshold at 20 the Perception curve gives pleasing result and
with sufficient difference from other curve types in RT, more saturated than saturation
and value blending, less than weighted. The new curve is similar to the film-like yet
without the ugly yellow-orange artifacts at higher contrast.

B. The curve action on TIF and raws is a little different. Raws seem more desaturated
with perceptual curve. So for pure darkening curve, to avoid grayness I would increase
the value of the saturation slider. And the more darkening the curve is the more saturation
would be needed upto about 25.

C. It is interesting to compare pure brightening curve in film-like and perceptual.
The film-like blends to white in highlights and that does feel more natural.

D. I noticed that perceptive curve exhibits some of the behavior illustrated here:
http://www.michaelezra.com/Projects/RT/images_shared/rt-ptc-7_1.jpg
May be I am splitting hairs here though:)

Reported by michaelezra000 on 2015-07-21 12:31:20

[Beep6581]

P.S. there is going to be a long electrical outage in my neighborhood today, so I may
not have access here.

Reported by michaelezra000 on 2015-07-21 12:32:59

[Beep6581]

In #34 D. the perceptual curve is on the left side

Reported by michaelezra000 on 2015-07-21 12:49:27

[Beep6581]

Thanks for testing, this gives me something to work with. I'll do a short mountaineering
hike this evening so there won't be new work until at least tomorrow.

B. My intention is that the curve should work the same for raws and TIFFs, but the
processing pipeline changes quite a lot when doing a TIFF so there may be some thing
I have missed. I shall look into that.

C & D. I shall test to blend in more of the film-like in extreme highlights, its desaturated
highlights gives advantages in terms of gradients, but disadvantage in terms of color
accuracy. The reason I've kept it down is to satisfy my desire for a correct hue in
light blue skies :-) (film-like turns them into dull almost gray), but I'll test and
see how much I can lower it without hurting that too much.

Reported by torger@ludd.ltu.se on 2015-07-21 14:56:43

[Beep6581]

Hmm... I'm having a massive difference between TIFF and raw curves in general. With
raw files there's a gamma applied before the curve, and not the same with TIFF. This
means that if I make a neutral TIFF file (looks exactly like the raw when no changes
are applied) and then apply exactly the same curve in the raw as in the TIFF, the raw
becomes much brighter, regardless of which curve type I use.

Is this a problem only I have, or is curves working like this in RT?

Reported by torger@ludd.ltu.se on 2015-07-22 08:48:05

[Beep6581]

Hmmm... TIFF (non-raw) curve handling in general seems broken to me. It works like this:

Raw files are converted to linear float, but get sRGB gamma assigned for the curves,
so the curves works in sRGB gamma space (~2.2), and the histogram you see inside the
curve is sRGB gamma. This is as it should be, curves are much better used with a gamma
(or else the shadow range get tiiiny at the start), and sRGB gamma is a good default.

TIFF files are loaded with their native gamma, but then in the colorspace conversion
step they are linearized to linear float, so the internal representation becomes identical
to that of the RAW. However, there is no gamma assigned for them, so the curves will
operate in linear gamma (1.0). This makes the curve behave differently.

I think this is broken. Curves should always operate with the same gamma. I need to
file a separe issue on this, and this issue is blocked until that is resolved.

Reported by torger@ludd.ltu.se on 2015-07-22 09:17:24

[Beep6581]

Reported by torger@ludd.ltu.se on 2015-07-22 09:34:37

• Blocked on: #2850

[Beep6581]

Here's a patch updated after the gamma fix in issue 2850.

It required a lot of reweighting as the curve algorithms now work on linear data (as
they should).

The only debug slider active in this patch is saturation for manually setting the chroma
scaling start value.

As a lot of reweighting has been made and my eyes are tired this evening I can have
messed up some. I shall look again tomorrow.

There's whiter extreme highlights now which should improve #34 C&D, still a tiny bit
flatter than Film-like but I don't want to go further as it then kills too much light
blue sky color.

Shadow saturation increase is a bit larger than before, but need to be reviewed now
when TIF/raw should work the same.

The TIFF curve used in testing so far need to be reworked due to the changed gamma
(issue 2850).

Reported by torger@ludd.ltu.se on 2015-07-23 22:42:42

• No longer blocked on: #2850

---

• Attachment: ptc-patch-debug-8.txt

[Beep6581]

Anders, tbanks for correcting the gamma issue so quickly:) I will get back to the workstation
with calibrated screen on Saturday and then can resume testing and share my feedback.

Reported by michaelezra000 on 2015-07-24 04:00:32

[Beep6581]

Further rebalancing this morning, overall saturation should now be more similar to what
it was in ptc-7 before the gamma fix.

Hopefully this should be close to something releasable :-)

Reported by torger@ludd.ltu.se on 2015-07-24 09:56:10

---

• Attachment: ptc-patch-debug-9.txt

[Beep6581]

Tiny change to make sure hard clip is white.

Reported by torger@ludd.ltu.se on 2015-07-24 19:37:07

---

• Attachment: ptc-patch-debug-10.txt

[Beep6581]

My own observations:
 - for the typical s-curve saturation is now as high as it can be, if I would change
   in any direction I would lower it. I'd say that now it may be a tiny bit over-
   saturated. The relatively high level is there to battle the grayness issue.
 - on over-exposed / high key portraits film-like can have a tiny bit nicer rolloff

   to white, but I've already pushed perceptual as far as I want in that regard, for
   other types of photo like landscape it's better to keep color fidelity higher up

Reported by torger@ludd.ltu.se on 2015-07-25 09:29:26

[Beep6581]

Discovered a saturated color clipping artifact that needs fixing, working on that. Otherwise
I'm personally fine with the current result and ready to commit when I've fixed this
issue.

Reported by torger@ludd.ltu.se on 2015-07-26 10:01:55

[Beep6581]

Fixed the color clipping, but I'll make some more tuning of the highlight rolloff before
publishing the next patch.

Reported by torger@ludd.ltu.se on 2015-07-26 10:49:45

[Beep6581]

I like this new curve, very nice result! (from a non-photographer point of view)

On a side note, do not forget to remove your printf (if any) before committing ;)

Reported by natureh.510 on 2015-07-26 11:01:01

[Beep6581]

Anfers, this is a great result! 

I see that one of the differences from film is that it either affects the hues, or
boost colors selectively. If not overused, it can be pleasing, but can easily lead
to ugliness. This new curve does not seem to do that and in some cases may *appear*
a bit sterile, but overall it allows broader range of adjustments and is excellent.

Thanks for adding this to RT, may be we should move this to be the default curve?

Also, may be its worth considering, changing the Contrast slider in Exposure section
to work on the basis of this curve.

Reported by michaelezra000 on 2015-07-26 13:40:27

[Beep6581]

Committed.

Fixed the #46 clipping bug, and minor reweighting in the extreme highlight rolloff,
otherwise same as last patch (minus the debug/print stuff :-) )

#49: at some point I'd certainly like this type of curve to be default, but I think
we can let it be in for a while first. We would probably want to optimize it a bit
too, like replacing the LCMS2 CIECAM02 calls with own custom code.

Reported by torger@ludd.ltu.se on 2015-07-26 18:41:58

• Status changed: FixedPendingConfirmation

[Beep6581]

It needs documentation and it needs Ingo ;)

Reported by entertheyoni on 2015-07-27 19:57:54

• Status changed: NeedsDocumentation

[Beep6581]

Ah you already documented it, great!

Reported by entertheyoni on 2015-07-27 19:59:34

• Status changed: FixedPendingConfirmation