New Highlight Reconstruction algorithm -- beta version

[Beep6581]

Originally reported on Google Code with ID 780

Now available for testing...

The new algorithm has been substituted for "Color Propagation" in the user interface.
 It attempts to propagate information in unclipped channels to infer clipped channel
values.  The recovered values are typically beyond the default output range, so one
may have to apply some negative EC to make the recovered values visible.

The method is typically faster than Color Propagation, but a good bit slower than the
rest.  The output still doesn't look natural in areas where all channels are clipped;
not sure much can be done about that -- one doesn't have much to work with!

Reported by ejm.60657 on 2011-06-18 06:29:05

---

• Attachment: HLRecon-02.patch

[Beep6581]

Bugfix...

Reported by ejm.60657 on 2011-06-18 06:43:41

---

• Attachment: HLRecon-03a.patch

[Beep6581]

Emil, this is fantastic!

it does not work yet on 100% preview or saving the image:
http://www.timelessme.com/temp/Postings/HLRecon-03a.jpg

Reported by michaelezra000 on 2011-06-18 14:20:55

[Beep6581]

Should auto levels be able to pick up the recovered highlight data?
This would be useful especially with clip=0.0000 to be able to set -EC exactly to contain
all recovered HL.

Reported by michaelezra000 on 2011-06-18 14:48:35

[Beep6581]

It appears that the highlight recovery amount=100 can reach fully recovered highlight
data even without -EC.

Reported by michaelezra000 on 2011-06-18 15:06:36

[Beep6581]

I have no idea why 100% preview is not working -- the algorithm is inserted into RawImageSource::preprocess,
so it should be operating on the raw data as it is loaded and processed.  I suspect
somewhere along the line the processing pipeline got corrupted, we saw a symptom of
that with the auto and spot WB.  I think it is also the basis of the complaint about
clipped levels being grey instead of white, with highlight reconstruction turned on.
 Also, when the tool is enabled the image brightness changes, and that is not the correct
behavior -- IMO the exposure compensation should be absolute, and if one reconstructs
highlights, then one should have to apply negative EC to reveal them.  This way the
user is aware of where the reconstructed bits are.

So, when the pipeline is operating correctly, autoLevels should interact correctly
-- in ImprocCoordinator.cc, raw preprocessing takes place before the autoexposure histogram
is calculated, which is the basis for the autoexposure processing.

Reported by ejm.60657 on 2011-06-18 15:08:12

[Beep6581]

To get correct behavior, comment out lines 419 and 420 of rawimagesource.cc

    // Color correction (only when running on full resolution)
    /*if (ri->isBayer() && pp.skip==1)
        correction_YIQ_LQ (image, raw.ccSteps);*/

This seems to restore proper functioning of 100% previews.  I have no idea yet what
is going on with this function call; it looks innocuous. 

Reported by ejm.60657 on 2011-06-18 16:15:25

[Beep6581]

Also the same is 'fixed' by simply setting "false color suppression steps"=0.
(because of line 1545)

Reported by michaelezra000 on 2011-06-18 16:43:30

[Beep6581]

What's weird is that the code seems to be finding an image to process which shouldn't
exist!  Once highlight recovery is enabled using the new reconstruction algorithm,
that algorithm changes the file rawData which is the source of demosaicing; so where
is the program finding this unreconstructed raw data, or corresponding demosaiced output?
 Even when reconstruction is checked in the profile, no less; in this case, upon opening
the image the reconstruction processing should be part of the pipeline from the beginning!

Reported by ejm.60657 on 2011-06-18 17:00:33

[Beep6581]

There may be a variable leaking somewhere

Reported by michaelezra000 on 2011-06-18 17:13:25

[Beep6581]

There is something else:

With HR = "color propagation" the 100% view is incorrect (not patching main preview),
but is the same as HR=off

With HR = "anything else"     the 100% view is incorrect (not patching main preview),
but it is changed from HR=off

Reported by michaelezra000 on 2011-06-18 17:26:23

[Beep6581]

patching = matching;)

Reported by michaelezra000 on 2011-06-18 17:27:06

[Beep6581]

Not sure what you mean, for me when I commented out the code as indicated above, 100%
preview in main window is correct.

Reported by ejm.60657 on 2011-06-18 17:42:40

[Beep6581]

Yes, comment 11 is for that code being NOT commented, just pointing to a difference
that is present in HR behavior.

Reported by michaelezra000 on 2011-06-18 17:53:00

[Beep6581]

Amazing, Emil, thank you very much! Works great for most photos!
http://i.imgur.com/LEy1H.jpg
In fact from my testing so far the only times I would not use it is if it produces
false colors, but when the image is such that it doesn't produce false colors, it has
always surpassed all three of the other highlight recovery algorithms!

Do you want any test photos where it fails, such as this one?
http://i.imgur.com/nrW37.png
Notice that HRA is at 0, yet I only get those pixels when I select color propagation.

I have to set "False color suppression steps" to 0 due to the bug in issue 741.

For me the preview looks the same as the 1:1 image.

Reported by entertheyoni on 2011-06-18 18:00:39

• Status changed: Started
• Labels added: Component-Quality

[Beep6581]

@Slony, for nrW37.png are you referring to the 'snowflake' pattern on the right side,
midway up the sky?

Reported by ejm.60657 on 2011-06-18 18:08:40

[Beep6581]

Yes Emil. I managed to reproduce it on 1 other image so far, it was a photo of clouds.

Reported by entertheyoni on 2011-06-18 18:15:19

[Beep6581]

A sample raw file would be helpful.

Reported by ejm.60657 on 2011-06-18 18:27:21

[Beep6581]

Here you go:
http://www.rawtherapee.com/shared/test_images/richmond_1.pef
http://www.rawtherapee.com/shared/test_images/richmond_2.pef

Reported by entertheyoni on 2011-06-18 19:14:52

[Beep6581]

Morning Emil, please get the latest version, I think I fixed a bug with the correction_YIQ_LQ
that lead to clipping down the line. So it probably clipped off the highlights so your
new algo could not recover them, which probably looks like the preview would not get
updated.

Reported by oduis@hotmail.com on 2011-06-19 07:28:20

[Beep6581]

Some improvements...

Reported by ejm.60657 on 2011-06-19 16:44:07

---

• Attachment: HLRecon-06.patch

[Beep6581]

Emil, thanks, this works like magic!:)

BTW, when combined with Olli's latest revision, all issues with false color suppression
>0 are gone.

Reported by michaelezra000 on 2011-06-19 17:30:55

[Beep6581]

Could this be even further enhanced to smooth transition into completely clipped areas?
Here is the illustration and link to raw/pp3:

http://www.timelessme.com/temp/Postings/HLRecon-06.jpg
http://www.timelessme.com/temp/Postings/MMFC0564.MEF.zip

Reported by michaelezra000 on 2011-06-19 18:01:52

[Beep6581]

For completely clipped areas, the best one can do is hope to match the hue; there is
no information that would allow one to recover luminance information.  I'm currently
working on getting a decent flood fill of unclipped highlight data into clipped areas;
an eventual goal is to make the transitions into completely clipped areas match hue
reasonably, and transition without abrupt boundaries.

I suspect there are a few major changes on the horizon...

Reported by ejm.60657 on 2011-06-19 18:16:09

[Beep6581]

A slow version that operates post-demosaic; generates fewer artifacts, but is quite
a bit slower.  It may be possible to speed it up some.

Reported by ejm.60657 on 2011-06-20 02:50:11

---

• Attachment: HLRecon-08.patch

[Beep6581]

Hi, Emil, I can test this only tomorrow... 
I am curious, what made you switch to post-demosaic?

There was a paper on speeding up the box blur:
http://web.archive.org/web/20060718054020/http://www.acm.uiuc.edu/siggraph/workshops/wjarosz_convolution_2001.pdf

Reported by michaelezra000 on 2011-06-20 03:30:18

[Beep6581]

Pre-demosaic was leading to quite a few artifacts stemming from sharp, clipped edges.

I've already implemented the fastest box blur; if there is any speedup to be had, it
will come from blurring all channels at once, rather than each one sequentially.

Reported by ejm.60657 on 2011-06-20 03:58:17

[Beep6581]

The HL recovery behavior changed significantly, specifically, when HL reconstruction
is off - this can lead blown out areas to turn to gray. This is easy to see on MEF
file from comment 23. 

With HL recovery amount >0, once any HL reconstruction method is enabled the L value
of blown areas gets increased. It appears that with HL reconstruction ON HL recovery
behaves desirably.

On a side note, the progress bar does not reflect that HL recovery is still taking
place, it reports "Ready". This issue is pre-existing, not caused by this patch.

Reported by michaelezra000 on 2011-06-20 11:37:55

[Beep6581]

another observation- the HL recovery threshold does not work the same as before.

Reported by michaelezra000 on 2011-06-20 11:46:48

[Beep6581]

Re: #26: Is this with all versions, or just the latest patch?  With HL reconstruction
off, highlight recovery should be the same as before, since the new algorithm is not
in the imaging pipeline.  With HL reconstruction on, L values of blown areas should
go up -- the clipped channel(s) are being pushed to an approximation of their unclipped
values, which are necessarily larger than the clipping point of the corresponding channel.

Progress bar and other prettification not yet implemented.  Actually, I think there
is a lot of work to be done installing progress bar notifications along much of the
pipeline.

Reported by ejm.60657 on 2011-06-20 13:34:28

[Beep6581]

"blurring all channels at once, rather than each one sequentially"
Sounds good :]

Reported by entertheyoni on 2011-06-20 21:10:01

[Beep6581]

Emil, after more careful comparison, I conclude that HL recovery and threshold functionality
have not changed at all. Sorry for the false alarm.

Reported by michaelezra000 on 2011-06-21 01:05:54

[Beep6581]

I just tested HLRecon-08.patch with my images from Issue 715, seems like all works fine.

Branch: default
Version: Dev-3.1m6.21
Changeset: 3f39f6299197
Compiler: GCC 4.5.0
Processor: undefined
System: Windows
Bit depth: 32 bits
Gtkmm: V2.22.0
Build type: Release
Build flags:   -DNDEBUG
Link flags:   
OpenMP support: ON
MMAP support: ON
Rawzor support: OFF

Reported by sghpunk on 2011-06-21 16:03:33

[Beep6581]

Works fine for me in the latest default, fixes the pixel issue in comment 15.

Reported by entertheyoni on 2011-06-22 21:03:49

[Beep6581]

Another update...  needs some desaturation near the new white point, have to think how
best to do that; at that point I think it will be ready to commit.

Reported by ejm.60657 on 2011-06-23 02:51:10

---

• Attachment: HLRecon-12hiliteblend.patch

[Beep6581]

Looks nice, but you are right, the burnt areas have distinct coloration.

Reported by michaelezra000 on 2011-06-23 04:07:35

[Beep6581]

Here is a a comparison of patch 12 vs 8 in a problem area
http://www.timelessme.com/temp/Postings/hl12vshl8_01.jpg

Reported by michaelezra000 on 2011-06-23 04:16:39

[Beep6581]

Yeah, there was a bug.  Try this...

Reported by ejm.60657 on 2011-06-23 06:08:27

---

• Attachment: HLRecon-13.patch

[Beep6581]

Ah, better yet this one...  suppresses the use of highlight pixels near the border of
clipped regions.

Reported by ejm.60657 on 2011-06-23 06:57:37

---

• Attachment: HLRecon-15.patch

[Beep6581]

You are not getting any sleep:)
Recovery in picture with sky/mask/feathers looks absolutely amazing. The sky between
the feathers is completely natural.

In burnt areas (check my last image with burn sky) there is strong magenta without
HL reconstruction and with high values of HL recovery.

Reported by michaelezra000 on 2011-06-23 11:25:21

[Beep6581]

What, me obsessive? :D

About the magenta w/o HL reconstruction, I neglected to re-enable clipping of highlights
when it is turned off (I do this to see how nasty the burnt areas are); to re-enable
it, remove the commenting out of code that starts at lines 320 and 344 of rawimagesource.cc

Reported by ejm.60657 on 2011-06-23 12:40:43

[Beep6581]

great, I will try in the evening

Reported by michaelezra000 on 2011-06-23 13:04:07

[Beep6581]

Good evening you two,
Just saw that this patch does not import after my last commit, so here is a adjusted
version.
New method looks great. Lab blending seems to recover more highlight detail on some
images, but this one looks better color-wise in my opinion.

Reported by oduis@hotmail.com on 2011-06-23 17:32:08

---

• Attachment: HLRecoverFix.patch

[Beep6581]

Magenta is gone after clipping is uncommented.
I confirm Olli's observation on a higher L detail recovery with "Blend" mode. Could
it be due to different clipping threshold?

http://www.timelessme.com/temp/Postings/HLRecoverFix-16.jpg

Reported by michaelezra000 on 2011-06-24 00:57:42

[Beep6581]

The detail is there, you just need to apply more negative EC to reveal it.  Compare
the hair of the masked lady near her right ear; I think you will find vastly more detail
in my version.  In my algorithm, the white point is a bit floating depending on the
properties of the scene.  Reconstructed details live beyond the original clipping point,
and the user must decide how much they wish to reveal by adjusting the editing controls.

If there is a legitimate complaint, it is that color is still not consistent; sometimes
it works, sometimes not.  I am weighing whether to put in a saturation slider to allow
the user to decide how much they want to roll off chroma in highlights. For instance
in the masked lady with fan image, one doesn't want any desaturation -- the sky looks
very natural without it.  On the other hand, in the King image, the highlights have
unnatural tinting, and unless I can find a way of improving that, it would probably
be better to just desaturate highlights.

Reported by ejm.60657 on 2011-06-24 01:47:50

[Beep6581]

I just tried -1.6EC with KING image - it is remarkable how much detail is there!
Although, w/o experience in using this new capability it does not feel intuitive that
EC must be different for different methods of HL reconstruction to get the same L detail.
May be a label with suggestion text/tooltip may be used next to HL reconstruction tool
to indicate that additional & different level of -EC may be necessary.

Shouldn't auto-levels (at clip=0) be able to set EC to the correct value to help to
reveal all recovered detail?

Saturation slider sounds appetizing!

Here is a better comparison. But there is a small artifact here:
http://www.timelessme.com/temp/Postings/HLRecoverFix-16a.jpg

Reported by michaelezra000 on 2011-06-24 02:11:33

[Beep6581]

Yes I noticed that artifact and you will find it fixed in the next patch.

Reported by ejm.60657 on 2011-06-24 02:28:49

[Beep6581]

if there was anyone I wanted to get as a Santa! :)

Reported by michaelezra000 on 2011-06-24 02:39:06

[Beep6581]

Re: EC, my attitude has been that one is inferring highlights in EV zones that the camera
strictly speaking didn't capture, so one could argue that the rendering should remain
unchanged until the user consciously decides they want to pull the image and reveal
them.  In particular, if one does this automatically, then unclipped tones start jumping
around as the HLR tool is switched on and off.  If instead one does nothing, the unclipped
tones remain as they were before.  I suppose one could apply some sort of highlight
compressing tone curve, but that is what the highlight recovery slider is for, after
all.  One could have the HLR tool reset the HL recovery slider to some value, but I
think people would find that behavior annoying (I think I would).

Reported by ejm.60657 on 2011-06-24 03:35:11

[Beep6581]

Should HL recovery at 100% be able to pull in all reconstructed detail?

Reported by michaelezra000 on 2011-06-24 03:55:14

[Beep6581]

Either way would work, just as long as is clear how to use the tool.

Do you think it would be valuable to have a (new) option to auto-find setting(s) at
which all recovered detail is revealed, to simplify user's hunting for it. Perhaps
a checkbox or a button within HLR section?

Reported by michaelezra000 on 2011-06-24 04:04:04