Better demosaicing for sensors without AA filters

[Beep6581]

Originally reported on Google Code with ID 2623

I've done some comparisons between Amaze in RT, and Lightroom, Phocus and Capture One
on my new H4D-50 which like all medium format cameras lack AA filters.

However, also high resolution DSLRs like D800E, D810, Sony A7r lack AA filters so the
group of users having these type of cameras is probably growing.

When you don't have any anti-alias filter the demosaicer will have a tendency to generate
false colors due to the aliasing. In RT we have the false color reduction slider which
helps, but kills some color details too.

I've noticed that all of the commercial converters makes a considerably better job
in many cases.

I've attached an example of renders, where Phocus makes the best result (even color
of the tree branches, and actually matching the real color well), perhaps not surprisingly
as it's the same brand as the camera. I don't think the false color reduction in RT
looks that good, it removes the colorful aliasing, but the resulting color is not right.

Lightroom makes a good result in this example, but actually Amaze is often better than
Lightroom's demosaicer when it comes to rendering tiny small black details, like text
and such.

If you wan't to experiment on the same test file, here it is:

http://torger.dyndns.org/rt-bugs/B8410574.3FR

Reported by torger@ludd.ltu.se on 2015-01-01 13:33:33

[Beep6581]

Reported by torger@ludd.ltu.se on 2015-01-01 13:37:04

• Status changed: Accepted
• Labels added: Type-Enhancement, Component-Quality
• Labels removed: Type-Defect

[Beep6581]

Anders, the wrong colors when using 'false color correction' are caused by the blueish
snow. If you spot wb on the snow, the colors with 'false color correction' are much
less wrong. Perhaps that's not what you want to do. It's just a hint, what happens
here.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-01-01 14:17:12

[Beep6581]

Thanks, I did not know that, very interesting. I can't be used as a fix in the general
case though. Here's the full Phocus rendering for reference (it was exported as an
unsharpened tiff and then sha):

http://torger.dyndns.org/rt-bugs/B_8410574.jpg (note that it's unsharpened, apply unsharp
0.50 / 250 to get same result as in crops in inital post)

The places where RT has most difficulties is transitions to the white background. I've
noted that LMMSE does a better job than Amaze there, but then LMMSE which is adapted
for noisy images have issues with detail.

Amaze renders details a bit narrower and sharper than Phocus, I guess that may be a
reason why it's less good at hiding the aliasing.

Reported by torger@ludd.ltu.se on 2015-01-01 14:51:49

[Beep6581]

Anders, perhaps it's possible to enhance false color correction. Actually it doesn't
work on raw data, but on rgb data (but called at a very early step in pipeline). If
you want to have a look: rawimagesource.cc line 2107

Ingo

Reported by heckflosse@i-weyrich.de on 2015-01-01 15:50:23

[Beep6581]

To be more specific: it works on rgb data which is then converted to YIQ. Perhaps we
can get a better result using a different conversion (lab). I recall that in xtrans
demosaic we got a better result by using cielab in an intermediate step instead of
using YPbPr. 

Reported by heckflosse@i-weyrich.de on 2015-01-01 15:58:12

[Beep6581]

Hmm, perhaps the final processing in xtrans demosaic can be adapted for other demosaic
methods. Anders, in case of interest, have a look at line 3583 ff. of demosaic_algos.cc

Reported by heckflosse@i-weyrich.de on 2015-01-01 16:14:20

[Beep6581]

I may look at it later, but not now. I destroyed my Christmas holiday with manic reverse-engineering
of the 3FR calibration data so I need some rest from programming now for a while :-).

I suspect that the Kodak CCD in my new back has more aliasing in it than my previous
back with Dalsa CCD due to smaller pixel aperture, because I did not have as much aliasing
issues with that (there was aliasing of course, but not as bad as seen here). So if
one gets it to work well with this sensor it will probably perform excellent with other
AA-filterless sensors too, ie it's a good sensor to test with.

I think it's Phocus and Capture One that's market leaders in demosaicing AA-filterless
sensors, as they have long-term experience from their medium format business, so those
are good benchmarks to compare performance with.

It would be great if it's possible to keep the detail resolution of Amaze and clean
up false colors as good as Phocus, because then I'd say we'd have market-leading demosaicing
performance for these type of sensors.

Reported by torger@ludd.ltu.se on 2015-01-02 10:28:10

[Beep6581]

Anders, I'll try the method mentioned in #6 with amaze. IIRC it was made to correct
false colours because X-trans also has no AA filter

Reported by heckflosse@i-weyrich.de on 2015-01-02 13:02:01

[Beep6581]

Hmm the artifacts here look to my eye like the artifacts we had with wrong pre-demosaic
WB on UniWB shots before the auto pre-demosaic WB patch..

And as I have no way to check the calculated R,G,B weights I just checked by selecting
auto in RT's WB menu .. where it's obvious that auto WB fails miserably :(. It's the
same algorithm that is used for pre-demosaic WB isn't it ?.

May be an algorithmic step to keep auto WB in sane area is needed for such cases.

Auto CA served me for some non-AA sensors better than "false color suppression" but
here it also fails .. may be due to wrong autoWB .. 

I had this sane autoWB in mind with some other cases where RT looks to have problems
with rendering .. like sunsets where most frame is blueish so the R weighting gets
big and as a result we take red clipping in the hilight areas.

A more extreme idea .. to make pre autoWB variable across the frame by calculating
local WB in tiles like the raw autoCA is calculated ..

Reported by iliasgiarimis on 2015-01-02 23:28:41

[Beep6581]

Worth noting about this camera is that it has a much more sensitive green channel than
red in daylight. My old Aptus 75 back is quite balanced for daylight.

This test shot has a clipped sky which perhaps together with the skewed sensitivity
causes the white balance to go haywire. 

Reported by torger@ludd.ltu.se on 2015-01-03 12:31:33

[Beep6581]

Anders, do you have any reference shots of a color checker ?.

I find the "asshotWB" coefficients (2.42/1/1.33) normal, why do you say the green is
oversensitive ?.

I converted the 3fr to DNG .. the autoWB gets "normal" and the artifacts at the branches
are much less. Although the linearity gets worse (banding at highlights which need
"green equilibtration = 2-3" to calm down)

So I think that something goes wrong after you apply the calibration data, confusing
the autoWB algorithm ... maybe a BlackLevel/WhiteLevel mismatch .. 

I remember you had left a switch to revert to asshot pre-WB but I cannot find it to
try asshot vs auto pre-WB .. 

Reported by iliasgiarimis on 2015-01-03 14:08:57

[Beep6581]

If we use the white level instead of the constant value of 64000 in auto white balance,
the result gets much better. I'll make a patch to test.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-01-03 14:18:58

[Beep6581]

Anders, I did some analysis: The sky in your image looks like it is clipped in all three
channels. But if you inspect the raw data you get a max value of 65535 for red and
blue and a max value of 65269.964981 for green channel, which means that the clipped
sky is skipped for red and blue channel in RawImage::get_colorsCoeff because 65535
> 65535 - 25, but the green channel of clipped sky is not skipped because 65269.964981
< 65535 - 25.

That could cause a wrong predemosaic white balance.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-01-03 19:51:22

[Beep6581]

Looks like the data calibration spreads the single clipping point (65535) of the original
raw data to a wide distribution .. I need to set (in camconst.json) the white level
lower than 60000 to overcome this and take a sane pre-WB, no color cast at the highlights,
autoCA working correctly.

More than this, the distribution is imbalanced so we take maze artifacts at the highlights
https://drive.google.com/file/d/0B0NqktTgc54sRHRaTEhUU1NXTFU/view?usp=sharing

Here the result with WL=58000
https://drive.google.com/file/d/0B0NqktTgc54sNThnM09LVjBFRk0/view?usp=sharing

Something similar happens when we convert to DNG although the distribution is less
wide and we are OK with a WL around 65000

Reported by iliasgiarimis on 2015-01-03 22:48:38

[Beep6581]

Hi Ilias, shall I wait for your camconst.json changes before commit of Issue 2603?

Reported by heckflosse@i-weyrich.de on 2015-01-03 23:09:48

[Beep6581]

Ingo, I think lowering the WL at less than 60000 is too much (we loose 1/6 stops of
highlight range) and as I have no way to inspect the histogram of the calibrated raw
data I prefer to wait for Anders to decide on this.

In any way it is not exactly a problem of WL but also a problem with pre-WB which we
need to be more robust and maybe for this special case of *.3fr files, optimizing the
calibration.

Regarding pre-WB ..
I find strange and ineffective to only decrease the WL by -25 for the calculations
on 16bit data .. highlight data are many times unreliable due to non linearities, channel
imbalance and maybe skewed when there is a clipping point. I would say that 1/3 or
1/2 stops at top end are unreliable.

Reported by iliasgiarimis on 2015-01-04 01:07:56

[Beep6581]

Ilias, then I'll commit Issue 2603 patch tomorrow.

Concerning pre-WB and auto WB (in gui). There is a difference between both methods.
pre-WB ignores all values which are larger than whitelevel - 25. auto WB (in gui) ignores
all values which are larger than 64000 (and so ignores white level). Perhaps we should
review auto WB too?

Ingo

Reported by heckflosse@i-weyrich.de on 2015-01-04 01:22:50

[Beep6581]

I think RGB-WB (gui) use a lot different data (demosaiced, color profiled, float arithmetic
etc).. but Jacques knows better .. and I remember he had in plan some improvements
some years ago which never happened. 
BTW this could be helpful http://web.stanford.edu/~sujason/ColorBalancing/robustawb.html

Regarding bayer WB (pre-WB) .. "many times" is not most times .. non limearities mostly
exist in base ISO. I have no clear idea how to make it robust as by just clipping lower
we skew the analogies against green which is usually the stronger channel :(. 

Reported by iliasgiarimis on 2015-01-04 02:59:23

[Beep6581]

Ilias, Auto-WB (gui) also works on raw data before demosaic

Reported by heckflosse@i-weyrich.de on 2015-01-04 11:54:41

[Beep6581]

To be more specific, Auto-WB values are determined from rawdata

Reported by heckflosse@i-weyrich.de on 2015-01-04 12:32:51

[Beep6581]

I haven't read until now, I'll try have a look later. A bug in whitelevel handling in
my calibration data patch does not seem unlikely. 

Reported by torger@ludd.ltu.se on 2015-01-04 13:22:15

[Beep6581]

I can confirm that there is a whitelevel bug in calibration data application on 3FR.
I'm working on a fix now. Here's the .fff file converted by Phocus (ie 3FR with calibration
data applied, will look about the same as an Adobe DNG conversion).

http://torger.dyndns.org/rt-bugs/B8410574.fff

With the correct clipping the aliasing color errors get less visible, I'd say it's
about the same as it was with my previous Aptus 75 back so it's not as critical to
make an improvement. However it's quite clear that Phocus does a better job still,
so it would be nice if we could make an improvement to be up there with the best.

Reported by torger@ludd.ltu.se on 2015-01-04 14:52:38

[Beep6581]

I have now committed a fix of the 3FR calibration data white level bug, so the result
becomes the same with the 3FR and the FFF file. Sorry for that.

The issue is still valid, but to a much lesser extent, Phocus is overall better but
not as much. I'm not sure an improvement to false color reduction will make it much
better, actually on the tree branches false color reduction just slightly changes the
color of the aliasing from mostly magenta to a bluer tone, it's not really reducing
it. In many areas it looks better before false color reduction than after.

Narrow things close to clipped pixels looks really bad (attached picture, from the
example 3FR here) before false color reduction though, but I think that's a separate
issue that there's no "special case handling" for demosaicing with clipped neighbors.

Reported by torger@ludd.ltu.se on 2015-01-04 18:28:38

[Beep6581]

Issue 1625 is about the false color close to clipped highlights, still valid I guess...

Reported by torger@ludd.ltu.se on 2015-01-04 18:33:19

[Beep6581]

Anders, the link in #22 gives me an empty file (0 bytes).

Reported by iliasgiarimis on 2015-01-05 01:23:25

[Beep6581]

Sorry, here's the correct link http://torger.dyndns.org/rt-bugs/B_8410574.fff

Reported by torger@ludd.ltu.se on 2015-01-05 09:40:57

[Beep6581]

now the 3FR should produce the same result though.

Reported by torger@ludd.ltu.se on 2015-01-05 09:41:39

[Beep6581]

3fr - https://drive.google.com/file/d/0B0NqktTgc54sa3FlMTUyT21aRlU/view?usp=sharing
fff - https://drive.google.com/file/d/0B0NqktTgc54sYTRGZC1fYkw3b2s/view?usp=sharing

Almost the same imbalance but a bit different brightness (3fr a bit brighter).
I needed to set the "white point correction" at 1.01 for the 3fr and 1.02 for the fff
to fight the color cast at highlights.
We have to set in camconst.json the WL at 64000 for safety I think ..

Reported by iliasgiarimis on 2015-01-05 12:45:02

[Beep6581]

I'm almost 100% sure that all clipped pixels are at 65535. I've looked at a few 3FR
and FFF files from various cameras and clipping seems to work like this, in 3FR 65334
and 65535 is clipped but with calibration data applied (FFF and nowadays 3FR in RT)
those 65534 pixels are moved to 65535 so clipping is clearly defined as 65535. This
unlike for example the Phase One IIQ format where clip level is quite unclear.

I would suspect that what you see is a demosaicing bug, a special case when clip level
is as high as 65535. Unfortunately I've forgot how white balance and clipping is handled
during demosaicing...

We could set clip level to 64000, but I'm quite certain that would be a workaround
for a limit in the demosaicing pipeline rather than that the clip level would be undefined.

When demosaicing these files note that you need to set green equiliberation to 1 for
best results.

Reported by torger@ludd.ltu.se on 2015-01-05 13:15:55

[Beep6581]

I suspect some rounding error somewhere in the pipeline, ie a value that was clipped
becomes treated as not clipped at some point, looking at code now.

Reported by torger@ludd.ltu.se on 2015-01-05 13:25:56

[Beep6581]

The conversion too DNG applies calibration equivalent to yours isn't it ?.

See the spread of G2 channel measured in the burned area between the antenna and the
branches. It could be a bit different in different sampling areas. 

Reported by iliasgiarimis on 2015-01-05 13:50:22

---

• Attachment: B8410574-Sel-1405-1721-239x181.png
  

[Beep6581]

We need to put the WL below the spread at least.

A fully clipped file could help us to be more precise  

Reported by iliasgiarimis on 2015-01-05 13:52:58

[Beep6581]

On 3fr .. With no color profile, working space Prophoto, output RT_large_g10, demosaic
none I measure 

G1(red rows)=1.2/99.2/1.2
G2 (blue rows) = 1.2/100/1.2  

a little less than 1% difference so 1.01 correction is fine ..

G1 is 1.2/98.8/1.2 on fff and dng so we need 1.02 there .. 

Reported by iliasgiarimis on 2015-01-05 14:29:02

[Beep6581]

I think the spread you see is in the transition zone from unclipped to clipped so it's
normal.

I don't think there's anything wrong with the clip level, try change the demosaicer
to mono or none and you can see in the clipped sky all values are at 100%. To make
really sure I made a quick change to dcraw and changed all 65535 pixels to 0 after
loading, and then saved in "Document mode" (no demosaicing), attached the result.

I think I know what's happening though. The demosaicer don't have any concept of clipping
but demosaics everything as it wouldn't be clipped. The input is scaled for auto white
balance and to fit the full range, so what it sees in the clipped sky is 1.00R 0.38G
0.43B. This camera has low sensitivity on the red channel, so the difference between
R and G multipliers are larger than normal.

With that small G compared to R the demosaicer makes a new pinkish color with a lower
luminance, so we get a drop below clipping on the green. The special thing with this
camera is the large difference between R and G, and that clipping is so well defined
that it's set exactly at clipping rather than a good bit below.

This is not a problem with the raw file, but a problem with the demosaicer not knowing
that it's demosaicing clipped areas, that is also what makes pixel edging to clipped
pixels look bad (like in issue 1641).

Reported by torger@ludd.ltu.se on 2015-01-05 14:55:26

[Beep6581]

Anders, amaze_demosaic_RT.cc line 649

Reported by heckflosse@i-weyrich.de on 2015-01-05 15:01:03

[Beep6581]

Thanks for the pointer! Looking into code... we did a white balance scaling change back
in issue 2145... getting some flashbacks.

Reported by torger@ludd.ltu.se on 2015-01-05 15:10:46

[Beep6581]

and also 756, 783, 1166, 1167, 1258, 1259

If you want to play around with this code, it's best to replace all __SSE2__ with __SSE2ANDERS__
in this file ;-)

Ingo

Reported by heckflosse@i-weyrich.de on 2015-01-05 15:12:26

[Beep6581]

subtract 1 from all line numbers mentioned in #37 ;-)

Reported by heckflosse@i-weyrich.de on 2015-01-05 15:14:56

[Beep6581]

#34 Anders NO, it is not transition area, this area is heavily clipped and if it was
barely clipped then the weak red channel would be the first to show a transition not
the G2 which is the strongest or even if this was possible for green it would be the
same for both G1/G2.
It's sensor non-homogenuities which persist even after DNGs calibration

But fact is that this spread (and others I didn't show) is very small (0.1%) related
to the difference we see in RT (1.2%) so it is only a small part of the problem. I
hope you will find the problem elsewhere .. :)

I measured the difference of 1.2% in #33 with demosaic none .. so it'a not a demosaicer's
fault .. :)

Your picture does not help me as it is downsampled and monochrome. And I wonder if
these data are before of after calibration ?. Did you patched Dcraw also ?.

I will try with RT to visualize this non homogeneity

Reported by iliasgiarimis on 2015-01-05 16:17:54

[Beep6581]

I found the problem, not related to highlight demosaic at all. The auto-WB calculates
two slightly different scalings for G1 and G2 and then stuff goes a bit haywire.

This sensor has need for green equiliberation that's the reason G1/G2 differs, but
I think the wb handling is broken too, the G1 and G2 should only differ if they truly
should be handled as separate colors. I'm patching rawimage.cc auto wb like this:

--- a/rtengine/rawimage.cc      Sun Jan 04 18:50:11 2015 +0100
+++ b/rtengine/rawimage.cc      Mon Jan 05 17:36:50 2015 +0100
@@ -233,6 +233,8 @@
        }
        if (pre_mul_[3] == 0)
                pre_mul_[3] = this->get_colors() < 4 ? pre_mul_[1] : 1;
+       else if (this->get_colors() < 4)
+               pre_mul_[3] = pre_mul_[1] = (pre_mul_[3] + pre_mul_[1]) / 2;

        if (colors == 1)
                for (c = 1; c < 4; c++)

Reported by torger@ludd.ltu.se on 2015-01-05 16:37:24

[Beep6581]

#39: the picture is monochrome because it's a raw data dump from original dcraw directly
after loading raw samples from the .fff file (hasselblad_load_raw()), ie after calibration
as fff file is same as 3fr but with calibration applied. Yes I made a patch to write
a .pgm directly after loading samples; you can add this block of code anywhere in dcraw.c
and compile to get a dump:

  {
      FILE *stream = fopen("test.pgm", "w");
      fprintf(stream, "P5\n%d %d\n65535\n", raw_width, raw_height);
      for (row = 0; row < raw_height; row++) {
          for (col = 0; col < raw_width; col++) {
              ushort u16 =RAW(row,col);
              fwrite(&u16, 2, 1, stream);
          }
      }
      fclose(stream);
  }

Reported by torger@ludd.ltu.se on 2015-01-05 16:50:33

[Beep6581]

#40 is committed, so if you build 4.2.60 you shouldn't have the problem any longer.

The reason we haven't seen this before is because we have not had any camera with both
a precisely defined whitelevel without margin below and bayer green split at the same
time.

Reported by torger@ludd.ltu.se on 2015-01-05 16:53:39

[Beep6581]

Anders, did you check for possible side effects on line 1332 ff of rawimagesource.cc?

Reported by heckflosse@i-weyrich.de on 2015-01-05 17:16:50

[Beep6581]

I do not have an olympus file laying around to test unfortunately, but it should not
break. The code there just calculates averages for G1/G2 and equalizes, with the normalized
auto-WB it will equalize a little bit more, but it's about tiny differences so it should
not be a problem.

However it seems to me that the Olympus code could break clipping in a similar manner
like here, but I guess there's some white level margin on that.

Reported by torger@ludd.ltu.se on 2015-01-05 17:28:04

[Beep6581]

With the format and handling fixes, I now think this issue is obsolete, the original
issue at least.

I've look at 200% maginfication all over the file and compared to what Phocus can do,
and my conclusion is that Phocus does a little better job concerning aliasing than
Amaze, but if I switch to DCB it's a draw.

The thing left is false colors around clipped highlights, but that's a separate issue,
and has nothing to do with having AA filters or not.

Reported by torger@ludd.ltu.se on 2015-01-05 17:33:19

• Status changed: Invalid

[Beep6581]

re #44, Anders, thanks for looking :-)

Reported by heckflosse@i-weyrich.de on 2015-01-05 19:00:52

[Beep6581]

Re #34: please, keep in mind that the GCode space is counted and that attachment cannot
be deleted, so only small attachments are allowed (150kb for an image).

Reported by natureh.510 on 2015-01-05 21:38:36

[Beep6581]

ok thanks for letting me know, didn't know that. Deleted the big images, no longer needed
anyway.

Reported by torger@ludd.ltu.se on 2015-01-06 08:28:55

[Beep6581]

They cannot be deleted - they are still there, just hidden to non-admins. That's the
problem :) In light of that, I 'undeleted' them.

Reported by entertheyoni on 2015-01-10 19:28:09