Wrong White Cipping Point for Canon 6D and possibly more/all Canon models .. and more

[Beep6581]

Originally reported on Google Code with ID 1695

report .. http://rawtherapee.com/forum/viewtopic.php?f=1&t=4457
raw file ... http://filebin.net/6gv1c0wh5p

The photo has overexposed areas which are rendered magenta by RT due to  Wrong raw
white point for this ISO (2500) at Dcraw code. Dcraw sets it at 15490 (hex3c82) which
is correct for other ISOs but not for 2500...

According to the RAW histogram .. the correct value after Black Point (2048) Subtraction
is 11177 ... 
http://i.imgur.com/rneTpCd.png

So there is a need for a RAW linear correction at 15490/11177 = 1.38

The strange thing is that even converted to DNG (which is the recommended solution
for such cases) with the latest Adobedngconverter sets a WP at 13000 and there is a
need to correct by the factor 1.16 or else you take pink highlights.

I tried with the latest Dcraw and it also gives magenta highlights on the CR2 file
(BP2048, WP15490) but its OK with DNG (BP 2048, WP 13000) !!!. Then i used on the CR2
-S 13000 .. again no magenta highlights !!!.

My conclusion is that there is a wrong interpretation of those White Points by RT .
While Dcraw's 15490 and DNG's 13000 refer to the full 14bit range 0-16383, RT calculates
the White Clipping Point after Black Point Point (2048) subtraction.

I did the calculation of the WP (11177) like RT (after BP subtraction) so the correction
factors I gave are valid at the moment. In the full 14bit range it would be 11177+2048=13225.

So it looks like Adobe's DNG WP (13000) IS CORRECT IF INTERPRETED RIGHT although it
clips a little bit more to be at the safe side.

Reported by iliasgiarimis on 2013-01-26 14:00:46

[Beep6581]

I wonder if the problem is related to something I recently noticed in the RawImage::get_colorsCoeff()
function (and which I reported in this thread: http://rawtherapee.com/forum/viewtopic.php?f=7&t=3093&sid=20b7d40d1e1bb48a063af1e63de60d70):
when calculating the "sat" value, the "get_black()" function is used, which if I'm
not wrong returns always 0 for DCRAW versions >=9.15. While this is not an issue for
my NIKON NEF files, which are already corrected for the black point, it could be a
source of problems for Canon files. 

In other words, for a white point of 13225 the sat variable should be equal to 13225-2048=11177,
but due to the use of "get_black()" it is set to 13225 (i.e. the white point itself).

I still need to verify my hypothesis, as I did not have a canon raw file at hand, but
I'l post again as soon as I have some concrete numbers...

Reported by aferrero1975 on 2013-01-29 19:31:38

[Beep6581]

Issue 1690 has been merged into this issue.

Reported by rinni@gmx.net on 2013-02-01 11:46:27

[Beep6581]

I can confirm this is not only a 6D problem. I can see this magenta problem with CR2
images from my EOS 650D too.

Reported by rene.t.fritz on 2013-02-05 22:23:31

[Beep6581]

Hi everybody,

I would be happy to look deeper into this issue and try to provide a patch. Would you
have an example for your 650D that you could share, as well as of the bad result you
get in RT?

Reported by aferrero1975 on 2013-02-06 20:31:14

[Beep6581]

@ rene.t.f... 

If you will upload some samples choose some at ISO 160, 320, 640 ... where the difference
is more obvious.

@ aferrero...
You can find Canon samples with clipped highlights at Imaging Resource. All LowLight
samples are OK .. but only for ISO 100, 200, ... where the problem is lesser than 160,
320 ...  

Reported by iliasgiarimis on 2013-02-06 22:29:17

[Beep6581]

@ iliasgia...

I've tried to look for a sample with one of those intermediate iso values on imaging
resource, either for the 6D or the 650D, but I couldn't find any... also 2500 iso is
not a value wich seems to be commonly used for their tests. at the moment the only
prblematic image I have at hand is the high-iso one from you, so I'll start with that.

Reported by aferrero1975 on 2013-02-07 09:04:10

[Beep6581]

Is the intermediate ISO mandatory for providing sample images ? I see magenta highlights
in clipped areas a lot with EOS 1000D images,but I don't think I have intermediate
ISO on this camera,maybe in auto mode but I don' use it.

Reported by msth67 on 2013-02-07 11:55:09

[Beep6581]

@ aferrero1975

I have a test picture which has been provided in another issue (IIRC) about strong
magenta artifact. I don't remember if it's a CR2 raw file, but i'd like to send you
the file (when i'll be back home), maybe it's related to what you've discovered?

Do you think that it will complicated to find a patch? If no, I would recommend to
block issue 1696, since it's an important patch!

Reported by natureh.510 on 2013-02-07 12:12:52

• Status changed: Accepted
• Labels added: Priority-High, Component-Quality
• Labels removed: Priority-Low

[Beep6581]

I've no idea yet how complicated is to find a solution, but I plan to spend the evening
on that and report about my findings...

Reported by aferrero1975 on 2013-02-07 12:36:15

[Beep6581]

Some more test images will surely help to check/verify the problem...

By the way, the issue is not my discovery, credits go to iliasgia... for opening it.
I'm just volunteering to help...

Reported by aferrero1975 on 2013-02-07 12:44:18

[Beep6581]

Test files are here
https://docs.google.com/folder/d/0B2cdqmOL8KTQb0hJUjdmS2I5eTA/edit?usp=sharing

0821 is a real life example. Edges on overexposed areas are magenta

0912 Test shot with magenta (upper right area)
f 4.5 1/10s iso 100

0913 Test shot without magenta
f 5.6 1/10s iso 200

I did more test shots with higher iso and can't see magenta. I'll still try.

Reported by rene.t.fritz on 2013-02-07 14:51:44

[Beep6581]

maybe these are related?
#1631
#1018
#1641
#1604

Reported by rene.t.fritz on 2013-02-07 14:56:33

[Beep6581]

I see I can easily force magenta in white areas when I set the raw-white-point slider
below 1.0

Reported by rene.t.fritz on 2013-02-07 15:16:20

[Beep6581]

@ aferrero...

May be you can track an inappropriate change here .. http://code.google.com/p/rawtherapee/issues/detail?id=1433

http://code.google.com/p/rawtherapee/issues/attachmentText?id=1433&aid=14330004000&name=dcraw2.patch&token=llH8ulLKd_asKRaSYS7mPTVSRcU%3A1360250519007

I recall Oduis wrote that something changed with Dcraw's handling of Black Offset,
not sure if it was for 9.16 or 9.15 or 9.14 .. 

Reported by iliasgiarimis on 2013-02-07 15:58:57

[Beep6581]

@ iliasgia...

Indeed there is a comment in the rawimage.h source which quotes some changes in the
black offset handling starting from dcraw 9.15:

  int get_black()  const { return black;}  // from DCRAW 9.15 =0, but reflected in
cblack
  int get_calcblack()  const { return calcBlack;}  // simulated like DCRAW before 9.15
  int get_cblack(int i) const {return cblack[i];}

As ar as I understand, get_calcblack() or get_cblack() should be used, while get_black()
always returns zero (and indeed this is the case when running the code).

@ rene.t.f...

thanks for the images, I'll check them as well... concerning the other possibly related
issues, I'm still reading though them but it seems that several are related to preview/specific
problems occurring only at certain conditions, while the problem reported here seems
to appear with any demosaicing method and at any zoom level...

----------

Here are some observations I made based on IMG_0614.CR2:

- the clipped highlights in the raw data, just after black point subtraction and taken
from rawData matrix in copyOriginalPixels(), show a peak value of 13225 in four bayer
channels. If to that we add the black point of 2048, we end up with 15273 which doesn't
look too far from the 15490 assumed by dcraw

- in the RawImage::get_colorsCoeff() function "get_black()" is used instead of "get_calcblack()"
to compute the "sat" value: the first one returns always zero, thus "sat" get overestimated.
However, correcting that does not get rid of the pink highlights...

- still in RawImage::get_colorsCoeff(), the scale_mul_[] vector is multiplied by the
pre_mul_[] vector which, in turn, originates from get_cam_mul().

That's where I get lost at the moment. My understanding is that "cam_mul[]" contains
the camera WB multipliers, which apparently get "incorporated" in the scale_mul[] vector
through the pre_mul_[] values. 
Now it turns out that for this image the pre_mul_[] coefficients strongly "suppress"
the green channel as well as (but to a lesser extent) the blue one:

               red    green1    blue   green2
pre_mul_ = 1.000000 0.436488 0.595908 0.436488

This imbalance thoward the red is not compensated by the white balance which is applied
after the demosaicing in RawImageSource::getImage(). I'm still trying to understand
the math behind the computation of the red, green and blue multipliers at the beginning
of this function, so at the moment I'm stuck...

I would also be glad if someone could explain me this code snippet in  RawImageSource::load():

    float pre_mul[4];
    ri->get_colorsCoeff( pre_mul, scale_mul, c_black);//modify  for black level

    camwb_red = ri->get_pre_mul(0) / pre_mul[0];
    camwb_green = ri->get_pre_mul(1) / pre_mul[1];
    camwb_blue = ri->get_pre_mul(2) / pre_mul[2];

Does it mean that the pre_mul[] vector itself DOES NOT represent the camera WB multipliers?
But them why is it used in the computation of scle_mul[]?

Reported by aferrero1975 on 2013-02-07 21:12:34

[Beep6581]

IMG_0614.cr2 is at ISO2500 or in other words ISO 3200 -1/3stop ..

This is in the category of intermediate ISOs (160, 320, 640, ...) which derive from
the "integer" ISOs (200, 400, 800 ...) by just multiplying by a factor (usually it's
0.80) so their clipping point is lower. The bad thing is Dcraw uses a single clipping
point for all ISOs so it is usually correct for ISO 100, 200, ... but terribly wrong
for 160, 320, 640, ... 2500. 
As you can see at the histogram the correct value should be 11177. By changing from
get_black() to get_calcblack() you perhaps corrected the interpretation of 15490 (being
at the full range 0-16394 and not after Black point subtraction) but it's still wrong
for ISOs 160, 320 ...2500 ... because now it is 13225 instead of 11177.

You can check the above by trying to correct the magenta highlights by raw linear correction.
It was 1.38 before the correction, after the correction it should be 13255/11177= 1.18.

The WB multipliers are absolutely normal (typical daylight multipliers) even if they
are in a different notation than the usual. In the usual Dcraw notation the green factor
is always 1.0 and red and blue accordingly higher so you can see with Dcraw those are
2.291016, 1.0, 1.365234, 1.0  = 1.000000 0.436488 0.595908 multiplied by 2.291016.

Reported by iliasgiarimis on 2013-02-08 01:24:42

[Beep6581]

"13225 instead of 11177" should be "15490-2048=13442 instead of 11177".
correction it should be 13442/11177= 1.20 .

Reported by iliasgiarimis on 2013-02-08 01:29:23

[Beep6581]

My fault, I forgot that the black point subtraction is done later in the scaleColors()
function, hence the white point found in copyOriginalPixels() is still the one before
subtraction... 

I can confirm that setting the white point to 13255 and using get_calcblack() instead
of get_cblack() gets rid of the pink highlights, but I could not find a way to infer
the correct white point for this raw file, except probably hacking dcraw.cc to implement
iso-dependent values...

Concerning the pre_mul[] values returned by get_colorsCoeff(), as far as I can understand
they are not related to WB (which is applied later in the rendering pipeline...), they
look more like per-channel gain corrections. In other words, I got the impression that
the red and blue channels work at a lower gain that the green one, so that they need
to be "boosted" by a factor ~2 in order to get the color balance correctly. This is
compatible with the fact that the raw white points for the three channels are the same,
while the raw histogram shows green values that are in average higher than blue and
red (red being even lower than blue, which is compatible with the relative values of
the pre_mul[] factors).

One solution would be of course to derive the white points from a "white frame", similar
to the black points from a "dark frame"...

To summarize, the only patch I can presently propose is the replacement of "get_black()"
with "get_calcblack()" in RawImage::get_colorsCoeff(), the rest requires probably some
more discussion...

Reported by aferrero1975 on 2013-02-08 13:00:27

[Beep6581]

"I can confirm that setting the white point to 13255 and using get_calcblack() instead
of get_cblack() gets rid..."

it shoud of course read "... using get_calcblack() instead of get_black() gets rid..."

Reported by aferrero1975 on 2013-02-08 13:01:45

[Beep6581]

At least on my system the attached patch fixes the pink highlights problem on "normal"
iso images (like 0912). It does not however fix the issue of "intermediate" iso values,
for which the white point is clearly wrong, nor the problem of magenta edges on overexposed
images... I'm looking into this last item just now.

The patch was created with "hg diff" in the root of the rawtherapee source tree, I
hope that's the right way to do it (it's my first contribution, so I'm still learning...).

Reported by aferrero1975 on 2013-02-08 19:07:49

---

• _Attachment: get_calcblack.patch_

[Beep6581]

I think this calc_black function gives valid results only on Canon cameras and maybe
not all Canons. There should be a condition to use it instead of get_black because
as it is it will affect all cameras giving wrong black point. 
Like "if calc_black > black then black = calc_black"

Please check with Sonys, Panasonics, Canon compacts ... 

If you can provide a win32 build I'll be able to help ...

Reported by iliasgiarimis on 2013-02-08 21:11:04

[Beep6581]

From the RawImage::loadRaw() function (see code below), I understand that calcBlack
is set to the minimum of the sum between "black" and the channel-dependent values "cblack[]".
Hence it should be valid for all types of cameras, with either one single black value
or channel-dpendent ones... but you are right, one should check.

Unfortunately I'm compiling on Linux, no experience in compiling RawTherapee on win...
maybe some other user can help on that.

      // Setting the black and cblack
      unsigned int minBlack = cblack[3];
      for (int c=0; c < 3; c++)
          if (minBlack > cblack[c]) minBlack = cblack[c];
      for (int c=0; c < 4; c++) cblack[c] -= minBlack;

      black += minBlack;
      for (int c=0; c < 4; c++) cblack[c] += black;
      calcBlack=black;  // safe for compatibility with darkframe substraction
      black=0;  // since black is already reflected in cblack now, set it to zero

Reported by aferrero1975 on 2013-02-08 21:23:38

[Beep6581]

Looks OK. Thanks

Hmmm, I believe that the correct behavior would be to keep the four independent BlackPoints
one for each channel and to subtract accordingly from each channel. It can make a difference
... less noise at the very dark areas. This is the reason RT has the 4 independent
"raw Black Levels", to micro-adjust each channel's black level.

About 650D's White Points. 
I Find for the integer ISOs by inspecting raw histograms with RawDigger ..
After Black Point Subtraction ..
ISO 100  WP=11536
ISO 200 to 6400, WP most times is 13256 but there is a small deviation at some cases
(13255, 13254, 13253)
ISO 12600 = 14321 or 14322

RT with your patch sets WP at 13645 - 2048 = 11597. It's almost OK for ISO100 .. but
you loose some highlights at ISO > = 200 .. We need a lookup table ...

Reported by iliasgiarimis on 2013-02-09 01:11:07

[Beep6581]

The black levels ARE subtracted channel-wise in RawImageSource::scaleColors(), the get_calcblack()
is only used to compute the normalization factor that brings the channels approximately
in the range [0..65535]. As I already said, the red and blue channels are in any case
quite heavily boosted with respect to the green channel, a sort of per-channel normaization
that is much larger than the effect of taking into acoount the channel-wise black levels...
but I've no idea if for other cameras the situation is different.

Reported by aferrero1975 on 2013-02-09 05:42:24

[Beep6581]

@iliasgia

Here http://dl.free.fr/m4KXAHfq3 you can download a wxp exe including patch . copy
it inside a previous build.
The exe is renamed to avoid name collision.
Branch: default
Version: 4.0.9.211
Changeset: c018aa53e645
Compiler: gcc 4.6.1
Processor: undefined
System: Windows
Bit depth: 32 bits
Gtkmm: V2.22.0
Build type: release
Build flags: -m32 -march=native   --param l1-cache-size=16 --param l1-cache-line-size=64
--param l2-cache-size=2048 -mtune=generic  -fopenmp  -mwindows -DNDEBUG -O2
Link flags: -m32 -static-libgcc -Wl,--large-address-aware   -s -O2
OpenMP support: ON
MMAP support: ON

Reported by gaaned92 on 2013-02-09 11:07:51

[Beep6581]

gaane... thanks a lot for providing the win build!

I've looked at the image 0821 with darktable (amaze demosaicing and default color matrix
camera profile), and the magenta edges on clipped regions are there as well... the
problem seems not to be closely related to the white point issue. 

Does anyone have a dcp or icc profile for the 650D? I wonder if those artifacts are
somehow enhanced by the default color matrix profile...

Reported by aferrero1975 on 2013-02-09 11:53:52

[Beep6581]

@ gaaned92
Andre, many thanks once again.

@ aferrero...
Andreas, I am talking exactly about this scaling from "raw BP"-"rawWP" to 0-65535 where
i would prefer intependent 4 channel scaling.
The scalecolors() you are talking about is White Balance and the multipliers you see
are absolutely normal. They are not computed but copied from exif metadata. 
It's the "as shot" WB which is supposed to be correct (most times but not in UniWB
cases) and as I recal Emil Martinec (ejmartin, the author of Amaze demoisaic algo)
and Guillermo Luijk http://www.guillermoluijk.com/tutorial/dcraw/index_en.htm mentioned
that it's beneficial for demosaicing to have WhiteBalanced data. So the correct would
be to use the "as metered" data instead of the "as shot" because the later could be
manipulated by the user to a wrong direction ...

Reported by iliasgiarimis on 2013-02-09 13:08:19

[Beep6581]

"I've looked at the image 0821 with darktable (amaze demosaicing and default color matrix
camera profile), and the magenta edges on clipped regions are there as well... the
problem seems not to be closely related to the white point issue."

On IMG_0821.cr2 ??. It's "purple fringing", you can use "raw CA auto correction", and
a combination of "defring" or/and "false color suppression" to eliminate it. Keep in
mind that with defring and false color suppression you can loose small color details
(jewellery) ..  

"Does anyone have a dcp or icc profile for the 650D? I wonder if those artifacts are
somehow enhanced by the default color matrix profile..."

http://rapidshare.com/files/1638914790/Canon%20EOS%20650D%20Adobe%20Standard.dcp
http://rawstudio.org/svn/rawstudio/trunk/profiles/Canon_EOS_650D_Rawstudio_Advanced_Profile.dcp

Reported by iliasgiarimis on 2013-02-09 14:43:23

[Beep6581]

http://rawtherapee.com/forum/viewtopic.php?f=1&t=4476

Reported by iliasgiarimis on 2013-02-09 16:59:16

[Beep6581]

@ iliasgia...

"The scalecolors() you are talking about is White Balance and the multipliers you see
are absolutely normal. They are not computed but copied from exif metadata."

Thanks a lot for the explanation, I'm starting to better understand the logic behind
all those multipliers... Would you also be able to explain me the difference between
the "pre_mul[]" and "cam_mul[]" vectors of dcraw? The multipliers I mentioned were
corresponding to the "cam_mul[]" ones, which indeed change as function of the in-camera
WB settings. On the other hand, the "pre_mul[]" values are always the same... in addtion,
rawtherapee calls "camwb_red", "camwb_green" and camwb_blue" the RATIO between "pre_mul[]"
and "cam_mul[]"...

"On IMG_0821.cr2 ??. It's "purple fringing"..."

Indeed you are right...

Reported by aferrero1975 on 2013-02-10 13:08:33

[Beep6581]

It would be better if developers who know the internals of Dcraw and RT (Jacques or
Emil) find the time to answer such questions.

But I can give it a try ..
As I understand it pre_mul[] are WB multipliers computed to be correct on a cc24 chart
illuminated by a standard D65 lamp.
Copy from MEGUI (GUI front end for Dcraw) help tips on the "byDefault WB" .. "uses
a fixed WB based on a color chart illuminated by a standard D65 lamp".

Reported by iliasgiarimis on 2013-02-10 15:25:08

[Beep6581]

Thanks a lot for the hint, I think I start to understand the code now...

Reported by aferrero1975 on 2013-02-13 11:32:12

[Beep6581]

I thank you too, we learn from each other ..

About the patch,
I checked some files and it works fine. Later today I can check on many more cases,
anything I can think of which could be problematic ..

But as it is now a "patch submitted" status is the appropriate I think ...

Also a review of the patch by some developers for a possible regression on some (unthought
by us) cases is useful..  before committing.  

Reported by iliasgiarimis on 2013-02-13 11:59:53

[Beep6581]

Sorry for been so late at writing my findings about aferrerro's patch ..

I checked many raw files with clipped highlights from different brands and models.
The patched RT behaves correctly and I did not detected any real regression. 
There is a perceived regression only on some Canon *.CR2 raws which is not a real regression
but the inability of Dcraw's code to adapt to the floating White Point among different
ISO (and some other settings which affect WP) with it's single value for WP. 
This named "regression" is because it happened the old wrong interpretation by coincidence
matched the real white point better than after the patch and BP subtraction.

Looking at Dcraw's setting for WP I feel it's a mess. Sometimes it's set to be correct
for ISO 100 (and wrong for ISOs >= 200) sometimes wrong for ISO 100 and correct for
higher ISOs.

For example for Canon 650D the real WP (after BP subtraction) are ISO100 11536 ISO>=200
13256.
Dcraw sets it at 13645.
With unpatched RT the WP was 13645 .. wrong for ISO 100 and needed linear correction
13645/11536 but it was "almost" correct for ISO >= 200 giving a very slight magenta
tint almost inpercievable but still there and in need of 1.03 linear correction.
With patched RT the WP is 13645-2048=11597 correct for ISO100 but gives more clipped
highlights for ISOs >= 200 which is percievable.
I think this "regression" is still preferable as it only clips 0.20 stops from the
highlights while it always suppresses the tinted highlights artifact.  

With raws converted to DNG the patch gives a more correct result as it's logical since
with DNG conversion WP is set at the correct value for each ISO - setting.

I think this patch should be included in the upcoming  4.0.10. release. 

Reported by iliasgiarimis on 2013-02-27 16:12:01

[Beep6581]

iliasgia... thanks a lot for your efforts! 

Indeed the situation with white points seems to be quite messy. I just discovered that
my D300 has different white points depending on the channel: red and blue are at 16383,
while the green channel is at 15330. In dcraw the white point for the camera is not
set explicitly and defaults to 16383 (0x3FF)... 

Reported by aferrero1975 on 2013-02-27 21:34:28

[Beep6581]

I just tested the get_calcblack.patch workaround and I am in favor of committing it
ASAP for 4.0.10. It prevented the magenta blob in the Canon raws found in this issue
and the related Forum thread, while not touching Pentax and Nikon raws I tested it
on.

Reported by entertheyoni on 2013-03-01 16:28:00

[Beep6581]

Patch from comment #20 doesn't seem to solve the purple color cast from IMG_0614.CR2.
Did i missed something?

Reported by natureh.510 on 2013-03-01 16:28:36

[Beep6581]

It is correct, it does not solve the color cast on IMG_0614.CR2, because the dcraw white
point for this camera at this iso sensitivity is wrong. This patch only solves the
bug in the computation of the "sat" value in get_colorsCoeff(). 
Fixing the iso-dependent white point issue is another story... why not to add a "white
frame" feature to get the correct channel clipping points? Then one would just need
to shoot few overexposed frames at different iso values, and let RT pick-up the most
appropriate one based on the iso value of the image being processed...

Reported by aferrero1975 on 2013-03-01 16:41:01

[Beep6581]

Hmmm... There's no reason to save a big image file just to get one value (or 3 values):
we could save a file with different lines, one per ISO values, to set the whit points
values. The user would just right click over the overexposed image (white frame) and
select "Save white point value".

Reported by natureh.510 on 2013-03-01 16:47:15

[Beep6581]

That's also a nice possibility, although I would not know how to code that in RT...

Reported by aferrero1975 on 2013-03-01 16:52:06

[Beep6581]

I'm just thinking that this a serious issue that will take the combined efforts of many
people to mitigate (fixing it is outside the scope of RT), so the best solution in
RT would be to have a method of sharing these clipping points for camera model and
ISO value. Why not have a text file with a simple format:
[make];[model];[iso];[BPr];[BPg];[BPb];[WPr];[WPg];[WPb]
Then we can easily share this info via the forum and keep adding to the text file,
the latest version of which will always be available for download on Google Code.

Someone should check whether this issue has been addresses in LibRaw.

For now, do we commit get_calcblack.patch?

Reported by entertheyoni on 2013-03-01 16:54:18

[Beep6581]

That's why we are here :), i don't know how to get the numbers on my side :). What is
the exact procedure to shot the images? How many values do we have to store (on a per
ISO basis if I've understood correctly)?

Reported by natureh.510 on 2013-03-01 16:54:57

[Beep6581]

*the idea is to have RT read from this file and automatically set the correct value,
if exists. If not then fall back on the dcraw values.

Reported by entertheyoni on 2013-03-01 16:55:18

[Beep6581]

Should this be an xml format then, with hierarchical structure?

<make value="">
 <model value="">
  <iso value="">
   <black_points red="" green="" blue=""/>
   <white_points red="" green="" blue=""/>
  </iso>
 </model>
</make>

or 

<make value="">
 <model value="">
  <BWPsettings iso="" BPr="" BPg="" BPb="" WPr="" WPg="" WPb=""/>
  </iso>
 </model>
</make>

this could probably bring in some xml implementation methods from the xmp branch:)

Another option is to create a key file, similar to pp3:
[MAKE_MODEL_ISO]
BPr=
BPg=
BPb=
WPr=
WPg=
WPb=

and look-up is via concatenated string of MAKE_MODEL_ISO
Then question is - what about intermediate ISO - if exact match is not found, how to
find the next useable? Using a key file, we may need to rely on exact match only. In
XML it can be sortable and querable.

Reported by michaelezra000 on 2013-03-01 17:27:07

[Beep6581]

#46
"i don't know how to get the numbers on my side :). What is the exact procedure to
shot the images? How many values do we have to store (on a per ISO basis if I've understood
correctly)?"

Just shoot at f/5.6 so that you have a burned area (all channels burned)and inspect
the raw histogram with rawdigger. 
Shots needed are at all ISO (even intermediates and expanded).
May be for some models there is a need for shots using all apertures wider than f/2.8
because there may be an upscaling to compensate for pixel vignetting ..http://www.dxomark.com/index.php/eng/Publications/DxOMark-Insights/F-stop-blues
... Canons do compensate in camera for this.

Canon 40D, Canon 35mm/f1.4, ISO100. f/1.4=16383 - f/2.0=14333 - f/2.8=13824 - f/4.0=13824
...
But its more complicated .. with a Zenit 55/f2.0 (no electrical connection with the
body) there is no compensation. Not sure what happens with third party lenses with
electrical communication.

An easier way is to, instead of inspecting histograms, convert to DNG and read the
exif. Adobe normally is a bit conservative i.e. instead of 13824 detected by histogram,
DNG conversion sets is for example at 13600. It's really a small difference to worry.

But keeps the same WP even for the wide apertures while the histogram shows up to 16383.

Adobe must have a data base for all modes and settings. If we could find it ... 

Reported by iliasgiarimis on 2013-03-02 00:43:56

[Beep6581]

I will commit get_calcblack.patch tonight if no one objects.

Reported by entertheyoni on 2013-03-02 22:11:12

[Beep6581]

We'll make the per Camera/ISO(/F-Stop) mechanism later (we'll need a new issue), and
in the mean time, i guess it's safe to commit this patch.

Reported by natureh.510 on 2013-03-03 20:43:37

[Beep6581]

Re #47: For now, key-files are usable in RT, not XML (and unfortunately, the XMP branch
is far from finished and usable. I'll continue working on this patch after the new
release). That's why we could go on one file per maker, with sections per model, and
one line per ISO/F-Stop combination.

Each file will be read and placed in structures anyway, so they will be querable in
both case.

On which basis does dcraw send its values now? I.e. on a per camera basis only?

If so, the fallback chain would be easy (for a given Make+Model):
 1. [Exact or most approaching ISO] and if
    possible [Exact or most approaching F-Stop] couple
 2. the actual behavior

Reported by natureh.510 on 2013-03-03 21:00:08

[Beep6581]

Aren't sensor white and black points independent on the lens? The peripheral lens-caused
issues can be avoided by using F8 and taking reading from the center of the image only.

Reported by michaelezra000 on 2013-03-03 21:02:00

[Beep6581]

I guess, another factor to consider possibly, is illuminant? However, I doubt that this
data will be gathered in practice. If we get the daylight data, that would be great
already:)

http://www.researchgate.net/publication/4133796_A_multi-spectral_image_database_and_its_application_to_image_rendering_across_illumination

Reported by michaelezra000 on 2013-03-03 21:38:06