New Highlight Reconstruction algorithm -- beta version

[Beep6581]

Originally reported on Google Code with ID 780

Now available for testing...

The new algorithm has been substituted for "Color Propagation" in the user interface.
 It attempts to propagate information in unclipped channels to infer clipped channel
values.  The recovered values are typically beyond the default output range, so one
may have to apply some negative EC to make the recovered values visible.

The method is typically faster than Color Propagation, but a good bit slower than the
rest.  The output still doesn't look natural in areas where all channels are clipped;
not sure much can be done about that -- one doesn't have much to work with!

Reported by ejm.60657 on 2011-06-18 06:29:05

---

• Attachment: HLRecon-02.patch

[Beep6581]

HL recovery is not guaranteed to bring back details, though we presumably could alter
its strength or increase the range of the slider.

Checkbox might be helpful, or simply a tooltip that one should apply autolevels after
HL reconstruction.  Unfortunately, current behavior of autolevels is broken in conjunction
with this tool -- I suspect it is calculated before, or autolevels histogram is not
recalculated, I'll have to look.  I suspect raw white point adjustment probably also
will not play well with it, and will need some code added.

Reported by ejm.60657 on 2011-06-24 12:39:37

[Beep6581]

"Do you think it would be valuable to have a (new) option to auto-find setting(s) at
which all recovered detail is revealed"
I don't, because highlight recovery is most often an aesthetic undertaking, where you
want the recovered areas to be a certain color or to trail off into white in a certain
way. An algorithm won't know what looks nice.

I found quite a few test images (all very bright; most +2EV bracketed) where this new
color propagation causes large chunks of white pixels to turn black in the processed
image, not visible in the preview, but I also upgraded from lcms-2.0 to 2.2, so I need
to test more to verify what's causing the issue. More on that tomorrow.

Reported by entertheyoni on 2011-06-24 21:00:08

[Beep6581]

Another update.  Not finished fixing some issues with totally burnt areas.  Fixed interplay
of HLRecon with HLRecovery.

Reported by ejm.60657 on 2011-06-25 19:32:11

---

• Attachment: HLRecon-16.patch

[Beep6581]

Should I un-comment the clipping (comment 41)?

Reported by michaelezra000 on 2011-06-25 20:18:53

[Beep6581]

Tried w/o uncommenting first (it is interesting that on KING image highlights in clipped
areas are blue now, not magenta)

When un-commented as per comment 41, burnt areas are neutral in color.

Recovery is really great,this is truly a recovery with reconstruction!!!
On my sample images HL Recovery reaches the full range of reconstructed data w/o -EC,
this is great.
Progress bar works with color propagation.
WB picker works, but not in the reconstructed areas.

What do you think - should HL recovery be moved to HL reconstruction section?

I think that HL recovery + reconstruction with color propagation is now one of the
most powerful tools in RT. This beats anything else out there!

Reported by michaelezra000 on 2011-06-25 21:01:24

[Beep6581]

I also confirm that artifact in comment 46 is fixed

Reported by michaelezra000 on 2011-06-25 21:17:58

[Beep6581]

There is a problem, which is predating this patch, but I uncovered while testing this
one.
+EC with -Black causes these areas to turn black.
This can be reproduced on my test image with the wig: set profile= neutral, EC=+5,
Black=-16384
+EC on it's own does not cause black areas, only when combined with -Black.
This may deserve another issue (?), or there actually may be one already.

Reported by michaelezra000 on 2011-06-26 00:27:25

[Beep6581]

If the bug predates this patch then let's make it a separate issue; generally it's better
to keep things separate, in case one needs to revert the new tool one isn't reverting
other needed changes.

Meanwhile, I think my issues with highlights are largely solved with this patch:

Reported by ejm.60657 on 2011-06-26 00:37:50

---

• Attachment: HLRecon-17.patch

[Beep6581]

Works great.
Here is some illustration:

This is a part of a large panorama, will finally make this image possible (Thank you!):
http://www.timelessme.com/temp/Postings/HL_reconstr_BA_01.jpg

"wig" test image
http://www.timelessme.com/temp/Postings/HL_reconstr_BA_02.jpg

Reported by michaelezra000 on 2011-06-26 01:48:56

[Beep6581]

I think it's pretty much ready for prime time.  If I don't hear of any serious bugs,
I'll commit this to default rather soon.  

There are of course some images that can't be helped, but I suspect this tool will
do a decent job with most, and for those that it doesn't help, there are always the
other reconstruction choices (for instance, I'm not super happy with the "King" image,
 half the fur reconstructed takes the blue tint from the robe rather than the tan of
the fur collar, but I suspect it would be complicated to engineer a foolproof way of
deciding which patch to draw from in painting in the missing color information).

Reported by ejm.60657 on 2011-06-26 17:23:06

• Status changed: PatchSubmitted

[Beep6581]

I agree, I see no issues so far.

Looking ahead, I am curious, how far away is this in-painting method from the content-aware
fill that was introduced in CS5?

Reported by michaelezra000 on 2011-06-26 17:35:57

[Beep6581]

It's more like a flood fill of clipped areas using unclipped highlights.  Seam carving
(aka content aware fill) might be useful, but unless a fast coding of the algorithm
were found it would be even slower than the current code, which uses a much less sophisticated
method.

Reported by ejm.60657 on 2011-06-26 19:36:23

[Beep6581]

Can we have a patch for the latest RT please?

Reported by entertheyoni on 2011-06-26 20:28:36

[Beep6581]

I still haven't figured out an efficient way to do these merges, took the better part
of an hour...

Reported by ejm.60657 on 2011-06-26 21:54:07

---

• Attachment: HLRecon-21.patch

[Beep6581]

I'm not going to creating a new issue but I have a story to tell which involves a few
quirks.

As I was testing this new color propagation, I noticed at first that it would create
huge chunks of black here the photo was supposed to be white (issue 801?). I also updated
to LCMS2.2 at the time and I couldn't be sure what caused it. Today I had time to test.
I freshly cloned RT and tested a few of the photos I had found to be problematic before.
I used the old color propagation filter, not this new one. The chunks of black were
no longer there (perhaps due to issue 801) but I found plenty of purple pixels, each
surrounded by a blue outline which got bigger the more false color suppression I used.
They were there for all the demosaicing algorithms I tried (which is most of them).
http://i.imgur.com/WzixL.jpg

I tried reverting LCMS to 2.0a and compiling RT without using OpenMP but they were
still there.

I updated back to LCMS-2.2, I updated RT to the revision this latest HLRecon-17.patch
is for and compiled it using OpenMP as I usually would, and everything looks fine.
Good. I don't know what happened in the meanwhile that caused the unpatched RT to start
putting purple pixels all over the place, but applying this patch clearly solves it.

This gallery shows the difference between versions 03a and 17 of the CP patch. The
first 4 are from 03a, the next 4 are from 17: http://min.us/mvnELlK

Reported by entertheyoni on 2011-06-26 22:06:35

[Beep6581]

"fatal error: hilite_recon.cc: No such file or directory"

hg add ? :)

Reported by michaelezra000 on 2011-06-26 22:07:13

[Beep6581]

Just found 1 issue: Custom Input ICC profiles interfere with Color propagation.
Applying custom ICC profile seems to disable the HL reconstruction effect.
The progress bar shows the HLreconst, but preview does not change.

Reported by michaelezra000 on 2011-06-26 22:47:52

[Beep6581]

http://paste2.org/p/1490776

Reported by entertheyoni on 2011-06-26 22:48:49

[Beep6581]

Sorry, forgot indeed to add...

Reported by ejm.60657 on 2011-06-26 22:50:59

---

• Attachment: HLRecon-21a.patch

[Beep6581]

#68: perhaps the custom profile does not forward values that are less than zero or greater
than 65535?  HLRecon needs the latter range...

Reported by ejm.60657 on 2011-06-26 22:54:39

[Beep6581]

This ICC profile was generated from a DNG profile created using Adobe DNG profile creator.
Do you by any chance know how to "forward values that are less than zero or greater
than 65535" in ICC profile? or how to get such a profile?

Reported by michaelezra000 on 2011-06-26 23:02:35

[Beep6581]

OK, patch applied and compiled successfully, no difference to the v17 screenshots.

Reported by entertheyoni on 2011-06-26 23:22:24

[Beep6581]

Committed to default.

Reported by ejm.60657 on 2011-06-27 12:46:34

• Status changed: PatchApplied

[Beep6581]

Emil

I've been away for more than 10 days and I find this patch.
It's great work, the quality is almost perfect.

Felicitations

Just one problem with "Mercurial"
hg pull = OK
hg update = problem with "hilite_recon.cc", but it still compiles

Reported by jdesmis on 2011-06-28 09:24:58

[Beep6581]

Patch to desaturate reconstructed highlights.  Just an idea, not a full implementation,
which would require additions to the UI.

Reported by ejm.60657 on 2011-07-04 19:29:32

---

• Attachment: HLRecon-patch3desat.patch

[Beep6581]

Desaturation works. Here is a comparison with Blend mode:
http://www.timelessme.com/temp/Postings/HLRecon-patch3desat.jpg

Reported by michaelezra000 on 2011-07-04 21:40:32

[Beep6581]

It looks like there are more details in blend mode near saturation, but that could simply
be because the white point is higher in color prop mode; does notching down the exposure
reveal any more detail?  I haven't done anything to the luminance channel, just desaturated
the chroma channels.

Reported by ejm.60657 on 2011-07-04 22:42:18

[Beep6581]

-EC or additional HL recovery don't reveal more detail than on the screenshot

Reported by michaelezra000 on 2011-07-04 23:21:21

[Beep6581]

It is strange as in KING image new Color prop method reveals not less luminance detail
than blend (illustration in comment 46)

Reported by michaelezra000 on 2011-07-05 00:05:20

[Beep6581]

Can we close this?

Reported by entertheyoni on 2011-11-06 17:50:27

[Beep6581]

Yes.

Reported by ejm.60657 on 2011-11-06 18:09:03

• Status changed: Fixed