Pinkish areas when using Highlight Reconstruction Method Colour Propagation

Beep6581 commented 9 years ago

Originally reported on Google Code with ID 2481

When using Highlight Reconstruction Method Colour Propagation you often get pinkish
areas. Sometimes they are caused by wrong white levels. But even when white levels
are correct, you can get them. The reason are some parallelized loops in hilite_recon.cc
HLRecovery_inpaint(). Here's a patch to fix that. We loose some speed by doing this,
but I'll try to get a bit of the lost speed back afterwards.

I tested with this file : http://img.photographyblog.com/reviews/nikon_d610/sample_images/nikon_d610_17.nef

Apply Default Profile + Colour Propagation and you'll see the pinkish area upper right
in the sky before patch (may depend on the number of cores your machine has).
With patch it's gone.

Processing time with that file before patch was about 1960 ms at my system
Processing time with that file after patch is about 2400 ms at my system

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-25 13:00:40

_Attachment: hlrecon_00.patch_

Beep6581 commented 9 years ago

Here's an example. This time with greenish artifacts. Left is before, right is after
patch

http://www.i-weyrich.de/rt/rt_2481_00.png

Reported by heckflosse@i-weyrich.de on 2014-08-25 18:06:30

Beep6581 commented 9 years ago

Ilias, I would be glad, if you could test this patch.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-25 20:51:21

Beep6581 commented 9 years ago

I will .. but you know it .. I am slowww :)

Reported by iliasgiarimis on 2014-08-25 22:27:23

Beep6581 commented 9 years ago

Hi Ingo, I tested with this file: http://ppmax.duckdns.org/public.php?service=files&t=2580a5573701cb3ea9b53ea234f1b7d6
(rt forum post http://rawtherapee.com/forum/viewtopic.php?f=1&t=5603)
The highlights are pink here still, but may be that is related to white level for this
camera model?

Reported by michaelezra000 on 2014-08-26 01:41:49

Beep6581 commented 9 years ago

Michael, I inspected the raw histograms of this file. By the shape of the histogram,
the WL in camconst.json is fine for this file. The histogram clips at 15230 (without
any significant distribution) while the WL defined in camconst.json is 15180.

Although we have to search for possible non linearities. What makes me thing of it
are Canon tags 
0x01fc NormalWhiteLevel 14582
0x01fd SpecularWhitelevel 15094
0x01fe LinearityUpperMargin 10000
and the related discussion with Iliah Borg
https://code.google.com/p/rawtherapee/issues/detail?id=2125#c10 #11 #12 #13 #14

I will ask ppmax for a similar sample at higher ISOs

Reported by iliasgiarimis on 2014-08-26 08:08:18

Beep6581 commented 9 years ago

Here are my results:
http://filebin.net/q98mdmu7tu

Unpatched is 16a2146921d4 + 2470hotdead0 + 2442crops1 + 2411clut20 + 2463luminancedenoise11
Patched is 2c98df196adb + 2481hl0

Reported by entertheyoni on 2014-08-26 08:08:38

Beep6581 commented 9 years ago

A good step !!. The patched output has no visible tiling and less saturation.

Reported by iliasgiarimis on 2014-08-26 08:23:02

Beep6581 commented 9 years ago

I located one problem. The wrong colors occur if two channels are clipped.
Actually, if not all channels are clipped, the clipped channels are calculated from
the non clipped channels, which works fine if only one channel is clipped, but not
if two channels are clipped.
This patch uses two channels to calculate the clipped channel if even one of the two
channels is clipped.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-26 10:53:55

_Attachment: issue2481_01.patch_

Beep6581 commented 9 years ago

Ingo :) wow :) serious progress !!

http://filebin.net/q98mdmu7tu/nikon_d610_17.nef-issue2481_01-IliasG.jpg
http://filebin.net/q98mdmu7tu/ppmax-2014-08-15_19-29-09-IliasG-issue2481_01.jpg

Still some hard (less than before) edges and desaturated islands plus some red-pepper
noise in flat "recovered" areas ... but much better in general !!.
Magenta artifacts have been replaced with almost correct colored surfaces !!.
Previously oversaturated highlights now look less saturated .. etc

Highlight recovery took 4 sec on my system (core2duo 3Ghz - win32)

Reported by iliasgiarimis on 2014-08-26 16:23:33

Beep6581 commented 9 years ago

The red pepper noise depends on demosaic method. Using 'fast' it's gone, strange...

Reported by heckflosse@i-weyrich.de on 2014-08-26 17:20:06

Beep6581 commented 9 years ago

I was ready to say that fast has no magenta points because it's soft but ahd/eahd have
negligible problem also !!. Although it seems there is a same but dumped down problem
at the transitions ..

All these pixel are at the place of Blue raw pixels ..

Reported by iliasgiarimis on 2014-08-26 18:40:19

Beep6581 commented 9 years ago

Ilias, there are also some at the place of red raw pixels. But I didn't find one at
the place of a green raw pixel

Reported by heckflosse@i-weyrich.de on 2014-08-26 18:55:43

Beep6581 commented 9 years ago

Hi, ppmax here--

Just wanted to ask if I should open an issue(s) regarding these two problems. These
are not necessarily related to using highlight compression or highlight reconstruction
per se...though I have seen the pinkish posterization that can occur when using highlight
reconstruction on this image.
1. Using the CR2 file I posted above the highlight areas appear to be rendered incorrectly
(using the default Neutral profile), possibly due to channel clipping or some channel
clamping that occurs in the render pipeline. Expected behavior is that these areas
will be rendered white or nearly white. (I also have CR2s that render similarly, but
mostly ISO 100/160 or so if anyone is interested). This issue appears to be related
to comment #8 above where clipping in two channels results in strange color renditions.

2. Using the file I posted above, using the WB eye-dropper to pick a color in one of
the orange/yellow/green cloud areas, there does not appear to be any change in the
WB (the WB value changes only incrementally and does not substantially alter the color
cast in the clouds). This may be related to comment #8 above as well...

In the RT forum iliasG requested high ISO samples that display this phenomenon. What
ISO's would be helpful? I may be able to dig up a few to contribute. 

thanks again--
ppmax

Reported by perreap on 2014-08-27 00:04:42

Beep6581 commented 9 years ago

Hi ppmax :)

I would like the samples at any ISO greater than 200 but preferably not ISO160 multiples
( i.e not 320-640- etc).

I want them to try their behaviour vs the current sample (ISO100) because a possible
reason for the color shifts is non linearity of the raw data at highlights and this
non linearity exists mostly at low ISOs (100 - 160).

If you can run RT for win32 I can upload a compiled version with the latest patch ..

Reported by iliasgiarimis on 2014-08-27 00:46:05

Beep6581 commented 9 years ago

Hi Iliasg--funny seeing you here! ;)

Should I try to find images that have two channels blown?

Reported by perreap on 2014-08-27 01:56:57

Beep6581 commented 9 years ago

Here's the next one. Output should be the same as with last patch but at the speed of
'without patch'.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-27 12:31:33

_Attachment: issue2481_02.patch_

Beep6581 commented 9 years ago

2014-08-15_19-29-09.cr2
HLRecovery_inpaint took 2090 ms
HLRecovery_inpaint took 2050 ms
HLRecovery_inpaint took 2010 ms
HLRecovery_inpaint took 2035 ms
HLRecovery_inpaint took 2082 ms
HLRecovery_inpaint took 2082 ms
HLRecovery_inpaint took 2068 ms
HLRecovery_inpaint took 2149 ms

nikon_d610_17.nef
HLRecovery_inpaint took 2347 ms
HLRecovery_inpaint took 2310 ms
HLRecovery_inpaint took 2297 ms
HLRecovery_inpaint took 2307 ms
HLRecovery_inpaint took 2296 ms
HLRecovery_inpaint took 2330 ms

The results with the patch are superior in most cases! I have two portraits I can't
show but the improvement is most visible there.
Or this:
Unpatched: http://i.imgur.com/5uY3KcB.png
Patched: http://i.imgur.com/CBbSVAs.png

Though I have noticed in a few photos a tendency for the patched version to show more
magenta instead of white:
amsterdam.pef unpatched: http://i.imgur.com/iRWNPGq.jpg
amsterdam.pef patched: http://i.imgur.com/eCE3hOw.jpg
pink_wall.pef unpatched: http://i.imgur.com/mzklJtO.jpg
pink_wall.pef patched: http://i.imgur.com/GEC3AMU.jpg
ryanair.pef unpatched: http://i.imgur.com/PWuiRu7.png
ryanair.pef patched: http://i.imgur.com/FhTn1Xs.png

Reported by entertheyoni on 2014-08-27 13:30:02

Beep6581 commented 9 years ago

Patched: http://filebin.net/q98mdmu7tu/nikon_d610_17_2481hrcp2.jpg
Patched: http://i.imgur.com/X7XkYCq.jpg Notice the many red dots.

Regarding the magenta clipping above, there is a bug related to white balance which
causes that:
amsterdam.pef patched when I open it: http://i.imgur.com/pLBDYkY.jpg
Change white balance (makes no difference whether I use the presets or just change
the number): http://i.imgur.com/cj5JShH.jpg
WB back to "camera": http://i.imgur.com/RLqzpKB.jpg

Reported by entertheyoni on 2014-08-27 13:34:36

Beep6581 commented 9 years ago

So overexposed_sky.pef after the white balance trick: http://i.imgur.com/M2JBtT6.png
Wonderful!

Reported by entertheyoni on 2014-08-27 13:35:42

Beep6581 commented 9 years ago

At my system with 2014-08-15_19-29-09.cr2:

unpatched: 1973 ms
patch 02 : 1898 ms

The red dots are also in unpatched.

Reported by heckflosse@i-weyrich.de on 2014-08-27 14:01:24

Beep6581 commented 9 years ago

DrSlony: your white balance trick is in fact a double processing of color propagation.
That's no solution :-(
Actually, color propagation is called directly after demosaic and it reads and writes
red, green and blue. If you change WB it's called again (without demosaic) and now
reads the already (from last call) changed red, green and blue and modifies them again.
That's clearly wrong! Color Propagation has to be executed only together with demosaic.

Reported by heckflosse@i-weyrich.de on 2014-08-27 15:29:14

Beep6581 commented 9 years ago

This fixes the 'double processing' of color propagation from #21

Reported by heckflosse@i-weyrich.de on 2014-08-27 16:39:47

_Attachment: issue2481_03.patch_

Beep6581 commented 9 years ago

The red dots disappear if we increase the WL correction at around 10%

This makes me to guess that the WL at camconst.json (15180, conservative truncation
of the real 15282 ..) is not respected and the absolute limit of 16383 is used .. ??
Or possibly the algotithm uses the absolute WL but should use the WL after BL (2048)
subtraction ??

Just guesses :)

Reported by iliasgiarimis on 2014-08-27 17:46:56

Beep6581 commented 9 years ago

Good find, Ilias. I'll have a look.

Reported by heckflosse@i-weyrich.de on 2014-08-27 18:12:53

Beep6581 commented 9 years ago

No difference in quality with patch 3, just the WB trick no longer works as you wrote.
Ingo can these sharp magenta borders be fixed? http://i.imgur.com/A7ks9zq.jpg

Reported by entertheyoni on 2014-08-28 07:15:24

Beep6581 commented 9 years ago

Regarding the single dots: Do they disappear if you increase "false color suppression
steps" from 0 to 1?
I've encountered these dots before in blown highlights when using LMMSE demosaic, setting
"false color suppression steps" >0 always made them disappear (the dots were not present
when using fast or Amaze demoisaic). Maybe this is caused by the same problem?

Reported by stefan.ittner on 2014-08-28 07:51:29

Beep6581 commented 9 years ago

Stefan, the bright dots are indeed 5X more with IGV and LMMSE.

The false color suppression does not make a good job there because it darkens the dot
below the surrounding area brightness and are again visible.

Reported by iliasgiarimis on 2014-08-28 09:33:55

Beep6581 commented 9 years ago

re #25: I try to fix them.

re #26: The don't disappear if I increase "false color suppression steps", but they
disappear (using amaze) if I change 

static const float maxpct = 0.95f;

to 

static const float maxpct = 0.90f;

in hilite_recon.cc

Reported by heckflosse@i-weyrich.de on 2014-08-28 10:11:23

Beep6581 commented 9 years ago

color propagation has four cases

0.) no channel clipped
1.) one channel clipped
2.) two channels clipped
3.) three channels clipped

I made a patch to visualize the cases

0.) original unchanged pixels
1.) red
2.) green
3.) blue

You can switch between modes by changing clip control value of flat field (>0 is the
colored version)

Reported by heckflosse@i-weyrich.de on 2014-08-28 11:06:21

_Attachment: issue2481_04.patch_

Beep6581 commented 9 years ago

While I searched for how to fix artifacts and sharp borders I detected some speedup
potential. Will submit a patch tomorrow (same output as last patch, but a bit faster)

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-28 22:06:51

Beep6581 commented 9 years ago

Hi again--

I have a bunch of high ISO images that display the strange yellow/green color casting
in highlight areas. Any interest in these please let me know and I'll post them. 

Images are ISO 200 - 6400

Let me know--

Thx
PP

Reported by perreap on 2014-08-30 02:57:56

Beep6581 commented 9 years ago

Hi PP :)

Please post them.

Reported by iliasgiarimis on 2014-08-30 07:06:10

Beep6581 commented 9 years ago

... A _link_ to them, of course.

DrSlony: is it possible to set the attachment size barrier from 10Mb actually to 400kb?
Or ask Google to let each project to lower down their 10Mb limit? Sorry for the off-topic.

Reported by natureh.510 on 2014-08-30 08:25:22

Beep6581 commented 9 years ago

Here are links to the files. These are intentionally blown and I tried to overexpose
to blow two channels. Please let me know if you need any additional explanation or
any additional files--I'm happy to supply them to help squash this bug.

The first file is "normal" supplied for reference:
http://ppmax.duckdns.org/public.php?service=files&t=6fdc111f2d9de663e68659d9121fa865

The rest of these are blown:
http://ppmax.duckdns.org/public.php?service=files&t=c111241f8da55102e7056d958e626ddb
http://ppmax.duckdns.org/public.php?service=files&t=8f164e1329b994818b12319d0ad6c3f1
http://ppmax.duckdns.org/public.php?service=files&t=5637276622f221461bca9dc622ee6c2e
http://ppmax.duckdns.org/public.php?service=files&t=626a26d63842586d2675d5f61a0ec592
http://ppmax.duckdns.org/public.php?service=files&t=26ab3ac4fbe185b0aac3ec155676f73a
http://ppmax.duckdns.org/public.php?service=files&t=12bafdd4b1e8a5158a8b40e9dd3d0664
http://ppmax.duckdns.org/public.php?service=files&t=8d9430d806bf8a8e8eeee2c9d67f6c13
http://ppmax.duckdns.org/public.php?service=files&t=3635be01356dd1e2d9d7d0ab95385f5a
http://ppmax.duckdns.org/public.php?service=files&t=b6b6f58523e3a83ef470d4dafd11297b
http://ppmax.duckdns.org/public.php?service=files&t=2a0ae533b483970ab304c5f8c384290b
http://ppmax.duckdns.org/public.php?service=files&t=de0692d6da5151bb5b744f3c7c234863
http://ppmax.duckdns.org/public.php?service=files&t=8877e3217fbe3d398ff202c41864f537
http://ppmax.duckdns.org/public.php?service=files&t=fd719fe0393950507c044305164e6525
http://ppmax.duckdns.org/public.php?service=files&t=c8749332ee1cc41228c5e5339144f9ec

Thanks again--
PP

Reported by perreap on 2014-08-30 14:25:49

Beep6581 commented 9 years ago

Here's a little speedup. Last patch with 2014-08-15_19-29-09.cr2 was 1898 ms.
Now it's 1334 ms. No visible differences. Small differences in amplified 'absolute
difference' of patch 04 and patch 05 output.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-30 15:00:26

_Attachment: issue2481_05.patch_

Beep6581 commented 9 years ago

After my last commit, issue2481_05.patch gets rejected. Fixed with this one.

Reported by heckflosse@i-weyrich.de on 2014-08-31 17:13:37

_Attachment: issue2481_06.patch_

Beep6581 commented 9 years ago

@ #33 Hombre unfortunately not, we even don't have the power to really delete the attachments,
[delete] just hides them to non-admins but still takes up our space.
Where we currently stand: http://i.imgur.com/Cwjk7lT.png
</offtopic>

Reported by entertheyoni on 2014-09-08 21:24:32

Beep6581 commented 9 years ago

I met the same issue with some of my raw pictures. Hope this gets fixed ASAP......

David

Reported by davidljwang on 2014-09-09 07:44:19

Beep6581 commented 9 years ago

Hello--just checking back in (not nagging!)

Has the issue related to having 2 channels blown been fixed? (I believe this is related
to post #29 above).

Thanks again
PP

Reported by perreap on 2014-09-26 23:34:21

Beep6581 commented 9 years ago

Hi pp,

#29 has never been committed. I made it only to allow visualizing the different cases.
I'll investigate soon.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-09-27 00:04:42

Beep6581 commented 9 years ago

No worries my friend. Thank you for your interest in fixing this...and please let me
know if you want any more info or frames.

thanks again
PP

Reported by perreap on 2014-09-27 04:02:35

Beep6581 commented 9 years ago

I committed a patch which removes the tiling pattern caused by race conditions mentioned
in #0.
I'll let the Issue open to continue when 4.2 is released

Reported by heckflosse@i-weyrich.de on 2014-10-20 20:44:48

Beep6581 commented 9 years ago

Any progress on the channel clipping issue? I'm on 4.2.21 (Mac) and have attached a
picture that shows clipping in the red channel in highlight areas. This is with no
exposure compensation, no highlight compression, etc.

thx!
PP

Reported by perreap on 2014-11-20 02:14:26

Attachment: [Screen Shot 2014-11-19 at 7.12.38 PM.JPG](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2481/comment-43/Screen Shot 2014-11-19 at 7.12.38 PM.JPG)

heckflosse commented 8 years ago

I close this Issue to continue in #2929

Beep6581 / RawTherapee

Pinkish areas when using Highlight Reconstruction Method Colour Propagation #2464