search

Beep6581 / RawTherapee

A powerful cross-platform raw photo processing program
https://rawtherapee.com
GNU General Public License v3.0
2.75k stars 313 forks source link

Inappropriate colors in DNGs from Samsung GX-20 #966

Closed Beep6581 closed 9 years ago

Beep6581 commented 9 years ago

Originally reported on Google Code with ID 979

What is the problem?
Colors from Samsung GX-20 shown by Raw Therapee are totally changed and user is unable
to get normal results by simply editing the photo.

What steps will reproduce the problem?
1. Open any DNG file from Samsung GX-20 DSLR in Raw Therapee
2. Double click on it to go to the edit mode
3. See the problem

Branch: default
Version: 4.0.2.2
Changeset: 3475c6bd6893
Compiler: GCC 4.5.2
Processor: generic x86
System: Windows
Bit depth: 64 bits
Gtkmm: V2.22.0
Build type: RELEASE
Build flags:  -mtune=generic -O3 -DNDEBUG
Link flags:  -mtune=generic -mwindows
OpenMP support: ON
MMAP support: ON

Windows 7 64-bit

View with appropriate colors shown from embedded thumbnail:
http://i.imgur.com/9dYa0.jpg

Colors generated by Raw Therapee
http://i.imgur.com/1poRv.jpg

Reported by tstankie on 2011-09-06 20:12:37

Beep6581 commented 9 years ago 
Seems to be a general problem with the GX-20, pretty old already:
http://www.rawtherapee.com/forum/viewtopic.php?t=1489

Using a goot ICC profile will solve the issue I guess. Maybe Colin will come up with
one in one of the next RT versions.

Reported by oduis@hotmail.com on 2011-09-06 21:02:13

Beep6581 commented 9 years ago 
I was hoping that it would be solved in 4.0 - but unfortunately it's not. I've been
trying to deal with it using channel mixer but without any satyisfying result. That's
why I decided to place it here as an issue. It's propably not difficult to solve, so
I hope developers will rememember about this while preparing a new release.

Reported by tstankie on 2011-09-06 21:08:58

Beep6581 commented 9 years ago 
And it's also rather time-consuming to choose right ICC profile every time I switch
to a next photo. Or I didn't notice a possibility to permanently change the default
input ICC?

Reported by tstankie on 2011-09-06 21:21:41

Beep6581 commented 9 years ago 
Create a new default postprocessing profile.

http://rawtherapee.com/blog/documentation

Reported by entertheyoni on 2011-09-06 21:40:39

Beep6581 commented 9 years ago 
The GX20 is on my todo list of fixes for RT4.

I have had a play and come up with a new GX20 color matrix, here is an example:-

http://img687.imageshack.us/img687/1627/gx20.png

I will need to refine this and add into the dcraw code, it should be available in a
couple of weeks.

Colin Walker - Huelight

Reported by cjwalker.mobile on 2011-09-07 00:33:45

Beep6581 commented 9 years ago 
Two years on, RT4 still doesn't do anything useful with the Samsung GX20 DNG.

Dave Coffin's dcraw.c (related to RawTherapee dcraw.cc?) version 9.19 doesn't have
a adobe_coeff() table entry for the Samsung GX20, and anyway doesn't call adobe_coeff()
for the Samsung GX20, presumably because this table is mainly lifted from code for
converting other raw formats to DNG, and the Samsung output is already a DNG. However,
the DNG is the only raw format produced by the Samsung GX20, and an adobe_coeff()-like
transformation is needed.

Arranging to call adobe_coeff() for a Samsung GX20 DNG
  .
  .
  .
      if (!strcmp(model, "GX20"))
          adobe_coeff(make, model);
  .
  .
  .
and adding
  .
  .
  .
    { "Samsung GX20", 0, 0,
/* DME */    { 5550,-2100,-500,-1950,10000,2600,-800,2800,6500 } },

  .
  .
  .
produces images that approximate the JPG output of the camera with maximum resolution
and minimum in-camera processing.

The values

 5550,-2100,-500,
-1950,10000,2600,
-800,2800,6500

look like a 3x3 matrix relating the 'RGB' channels in the DNG to the RGB channels that
you would want in the converted image.

I haven't looked at the RawTherapee source yet, so I don't know how relevant any of
this might be, but in the meantime I am leaving this comment as a place-holder.

Reported by David.Melville.Edwards on 2013-08-30 16:39:02

Beep6581 commented 9 years ago 
OK, I've now looked at the sources for RT 4.11, and see that Dave Coffin's dcraw.c gets
converted automatically to RawTherapee dcraw.cc.
I arranged to call adobe_coeff() for a Samsung GX20 DNG
  .
  .
  .
      if (!strcmp(model, "GX20"))
          adobe_coeff(make, model);
  .
  .
  .
and added
  .
  .
  .
    { "SAMSUNG GX20", 0, 0,
/* DME */    { 5550,-2100,-500,-1950,10000,2600,-800,2800,6500 } },

  .
  .
(Note that for the version of dcraw.c shipped with RT 4.11 SAMSUNG must be upper case).

My built RT4 now produces usable images from the DNG files.

The values

 5550,-2100,-500,
-1950,10000,2600,
-800,2800,6500

look like a 3x3 matrix relating the 'RGB' channels in the DNG to the RGB channels that
you would want in the converted image. These values work better than those for the
Pentax K20D and the Pentax K-7, or any of the Samsung NX series, whose sensors are
most like the GX20.

However, I would be most grateful if anybody who has better values than these could
publish them; this matrix has a tendency to give people pink noses!

Reported by David.Melville.Edwards on 2013-08-31 01:18:00

Beep6581 commented 9 years ago 
 { "SAMSUNG GX20", 0, 0,
/* DME */   { 6000,-2000,-700,-2700,13500,3900,-1350,3900,9750 } },

works a little better for me, and I am now using it. So even if this comment never
gets picked up by the maintainers, I'm better off than I was. The wonders of Open Source!

Reported by David.Melville.Edwards on 2013-09-02 14:05:47

Beep6581 commented 9 years ago 
Michael, can you have a look? If not I'll do.

Ingo

Reported by heckflosse@i-weyrich.de on 2013-09-02 21:25:09

Beep6581 commented 9 years ago 
David, it would be nice to have a DNG from Samsung GX-20 for testing.

Ingo

Reported by heckflosse@i-weyrich.de on 2013-09-02 21:27:46

Beep6581 commented 9 years ago 
Hi Ingo, I can make a patch, but would also like to have the DNG for testing

Reported by michaelezra000 on 2013-09-02 22:25:48

Beep6581 commented 9 years ago 
Hi Michael, that's great :-) Now we only need the DNG...

Reported by heckflosse@i-weyrich.de on 2013-09-02 22:29:34

Beep6581 commented 9 years ago 
Hi Michael, at the bottom of this page http://www.photographyblog.com/reviews_samsung_gx20_3.php
there are 6 DNG-Files which can be used for testing :-)

Ingo

Reported by heckflosse@i-weyrich.de on 2013-09-02 22:45:51

Beep6581 commented 9 years ago 
The color matrix values for GX-20 as defined in SAMSUNG AdobeStandard.dcp (which are
exactly those that Dcraw would normaly use) are ..

0.9427 -0.2714 -0.0868 -0.7493 1.6093 0.1373 -0.2199 0.3264 0.718

so for Dcraw they should be ..
9427 -2714 -868 -7493 16093 1373 -2199 3264 718

It would be best to have a colorchecker shot to build a dedicated dcp profile ... David
is this possible ?.

Reported by iliasgiarimis on 2013-09-03 02:27:44

Beep6581 commented 9 years ago 
I mean "SAMSUNG GX20 Adobe Standard.dcp" ..and the correct last number is 7180 . :(

Alternatively one can copy the values from Pentax K20D in Dcraw.c ..
{ 9427,-2714,-868,-7493,16092,1373,-2199,3264,7180 }

Reported by iliasgiarimis on 2013-09-03 05:39:04

Beep6581 commented 9 years ago 
The K20D values produce obviously wrong colours.

I suspect that the GX20 isn't as much like the K20D as its appearance, reputation and
the Adobe DCP suggests. I recall that DXO Labs got different results from their Sensor
tests. See http://www.dxomark.com/index.php/Cameras/Compare-Camera-Sensors/Compare-cameras-side-by-side/%28appareil1%29/526|0/%28brand%29/Samsung/%28appareil2%29/213|0/%28brand2%29/Pentax.

I'm now also in correspondence with Dave Coffin, who seems to think that part of the
problem is that dcraw.c expects the DNG to contain in its metadata a valid colour matrix,
and the colour matrix is missing or bogus.

Dave has sent me a new version of dcraw.c which allows me to use the -M switch to use
a built-in matrix with a DNG, for testing. He has plugged in values that his comment
indicates would be appropriate for the Samsung GX10. The result looks better than using
the K20D values, but is not, in my opinion, good enough.

Dave would also like a photo of a colour chart. Unfortunately I do not have access
to a proper painted matt version. However, I attach a DNG made by photographing a screen
display of an sRGB Gretag-MacBeth chart sufficiently out of focus to uniformly blur
the colour squares. The DNG is too large to attach, so it is in a multi-volume 7za
archive. The first part is attached. The second part will be in my next comment.

Reported by David.Melville.Edwards on 2013-09-03 09:11:25

Beep6581 commented 9 years ago 
Now the second part. BTW, I checked the gamma of my monitor before taking the shot,
and it is correct (2.2) if I get the viewing angle just right, but apparently not otherwise.
Hopefully I replicated this angle ...

Reported by David.Melville.Edwards on 2013-09-03 09:17:09

Beep6581 commented 9 years ago 
Indeed it's very strange that the colors remain terribly wrong with Adobes matrix. Even
stranger is that the SAMSUNG GX20 Adobe standard.dcp (which includes the same matrix)
if used as custom profile gives almost correct colors. 

I found a Pentax K20D shot with colorchecker included http://www.imaging-resource.com/PRODS/K20D/K20DhMULTII0100.DNG.HTM
On this DNG I built a dcp profile with X-rite colorchecker passport utility.
https://www.dropbox.com/s/o8wriaa3y6g32np/SAMSUNG%20GX20.dcp

Additionally, try this matrix for Dcraw.cc (copy from the above dcp profile) ..

13872 -4224 -1038 -4201 12823 1489 -1273 2557 8329

Colors look better to me with the above profile but .. they give unnatural WB temp/tint
:-( ..

Reported by iliasgiarimis on 2013-09-03 15:25:28

Beep6581 commented 9 years ago 
I tried the matrix above. Blue especially looked good, red OK but green is wrong. The
values I was using were on the money for Green, but red and blue were wrong. So I averaged
the values (I assume, without looking at the code, that whatever logic handles these
arrays normalises the values, so absolute values don't matter ...) and made a further
tweak to get rid of the superabundance of pink. Processing the Gretag-Macbeth DNG with
these values gives something that looks very like the JPG out of the camera.

7000, -3112, -869, -1969, 13165, 2695, -1236, 3530, 9050

Whilst the WB values reported by RawTherapee against the DNG don't make a lot of sense
to me (a priori I would expect them to be approximately double) I see similar behaviour
with PEF files and my Pentax K-X, and if I ask RawTherapee to Autocalculate the WB
from an image, it comes up with a value similar to the value it reports for the Camera
WB.

Reported by David.Melville.Edwards on 2013-09-03 18:48:01

Beep6581 commented 9 years ago 
You did not provide any out of camera for us to compare .. so for the photo with the
flowers, I used the in DNG embedded jpeg whose colors (after assigning the missing
sRGB color space) looks almost identical to the raw conversion with my above profile.
For better visual matching I used exposure at +0.20,contrast at +25.

Anyway, these are blind steps and the only chance for a correct (and neutral) profile
is a raw shot of the real colorchecker passport (shots of Gretag-Macbeth on display
cannot help ..).

Reported by iliasgiarimis on 2013-09-03 21:25:39

Beep6581 commented 9 years ago 
I'll make an offer: I'll buy the Gretag-Macbeth colour checker and (after making my
own profiles) will send it to the first one who uses it to produce a good profile free
of charge! This person sends it to the next person (also free of charge!!!) and so
on... This way we can gather profiles from a lot of cams and the colour checker always
belongs to me :-))) If a person it's sent to doesn't send it to another person, it's
my risk and I'll blame this person.

But I don't think, that I'll have to blame anyone.

Ingo

Reported by heckflosse@i-weyrich.de on 2013-09-03 21:41:46

Beep6581 commented 9 years ago 
"You did not provide any out of camera for us to compare ..." - is this directed at
me? I thought the embedded thumbnail would be adequate, since the Gretag-MacBeth has
almost no detail. But building up a collection of colour checker snaps has to be a
good idea.

Please do not incorporate your profile in a release. I have tried it on many different
DNG's (I own a GX20 after all) apart from the picture of the screen shot, and skin
tones come way too pink and greens are horribly wrong. The one I suggest above is the
best that I have tried (and I have tried lots, in an attempt to converge on acceptable
values), both in terms of aesthetics and conformity between the out-of-camera JPG's
and an 8-bit TIFF generated by dcraw.

Reported by David.Melville.Edwards on 2013-09-03 22:42:36

Beep6581 commented 9 years ago 
David, yes I would prefer an out of camera jpeg because the embedded in DNG one has
no (or wrongly) declared color space and shows oversaturated colors on wide gamut displays.

Don't worry I have no intention to promote a profile made on a possibly different camera
(although many say that GX20 is a rebranded K20D) and on a target shot under unknown
lighting. The fact that it triggers WB to strange tint (0.5-0.6) is a serious indication
that something goes wrong. ... but despite these it does a good job on your "samsung_GX20_2.dng"
!!!.

Reported by iliasgiarimis on 2013-09-04 01:04:36

Beep6581 commented 9 years ago 
Using dcptool I get the following from "SAMSUNG GX20 Adobe Standard.dcp"

  <ColorMatrix1 Rows="3" Cols="3">
    <Element Row="2" Col="2">0.769900</Element>        
    <Element Row="2" Col="1">0.226700</Element>        
    <Element Row="2" Col="0">-0.109400</Element>       
    <Element Row="1" Col="2">0.171900</Element>        
    <Element Row="1" Col="1">1.482100</Element>        
    <Element Row="1" Col="0">-0.647100</Element>       
    <Element Row="0" Col="2">0.003500</Element>        
    <Element Row="0" Col="1">-0.454900</Element>       
    <Element Row="0" Col="0">1.128100</Element>        
  </ColorMatrix1>
  <ColorMatrix2 Rows="3" Cols="3">
    <Element Row="2" Col="2">0.718000</Element>
    <Element Row="2" Col="1">0.326400</Element>
    <Element Row="2" Col="0">-0.219900</Element>
    <Element Row="1" Col="2">0.137300</Element>
    <Element Row="1" Col="1">1.609300</Element>
    <Element Row="1" Col="0">-0.749300</Element>
    <Element Row="0" Col="2">-0.086800</Element>
    <Element Row="0" Col="1">-0.271400</Element>
    <Element Row="0" Col="0">0.942700</Element>
  </ColorMatrix2>
  <ForwardMatrix1 Rows="3" Cols="3">
    <Element Row="2" Col="2">0.985400</Element>
    <Element Row="2" Col="1">-0.207200</Element>
    <Element Row="2" Col="0">0.046900</Element>
    <Element Row="1" Col="2">-0.083200</Element>
    <Element Row="1" Col="1">0.768600</Element>
    <Element Row="1" Col="0">0.314600</Element>
    <Element Row="0" Col="2">0.039900</Element>
    <Element Row="0" Col="1">0.235100</Element>
    <Element Row="0" Col="0">0.689400</Element>
  </ForwardMatrix1>
  <ForwardMatrix2 Rows="3" Cols="3">
    <Element Row="2" Col="2">0.985400</Element>
    <Element Row="2" Col="1">-0.207200</Element>
    <Element Row="2" Col="0">0.046900</Element>
    <Element Row="1" Col="2">-0.083200</Element>
    <Element Row="1" Col="1">0.768600</Element>
    <Element Row="1" Col="0">0.314600</Element>
    <Element Row="0" Col="2">0.039900</Element>
    <Element Row="0" Col="1">0.235100</Element>
    <Element Row="0" Col="0">0.689400</Element>
  </ForwardMatrix2>

ColorMatrix1 would lead to the following (I see that ColorMatrix2 was already tried):

    { "Samsung GX20", 0, 0, //RT based on SAMSUNG GX20 Adobe Standard.dcp ColorMatrix1
    { 11281,-4549,35,-6471,14821,1719,-1094,2267,7699 } },

David, could you please clarify where did you inject 
    if (!strcmp(model, "GX20"))
       adobe_coeff(make, model);

A workaround - install adobe dng converter, reference "C:\ProgramData\Adobe\CameraRaw\CameraProfiles\Adobe
Standard\SAMSUNG GX20 Adobe Standard.dcp" from rawtherapee.

Reported by michaelezra000 on 2013-09-04 02:12:33

Beep6581 commented 9 years ago 
The original dcraw.c version is 9.17 (I downloaded RawTherapee 4.11 sources).

My mods went in a little after line 7400 in the original (the exact line number would
depend on how many lines were added to the table in adobe_coeff())
 .
 .
 .

  if (!height) height = raw_height;
  if (!width)  width  = raw_width;
  if (height == 2624 && width == 3936)  /* Pentax K10D and Samsung GX10 */
    { height  = 2616;   width  = 3896; }
  if (height == 3136 && width == 4864)  /* Pentax K20D and Samsung GX20 */
    { height  = 3124;   width  = 4688; filters = 0x16161616;
      if (!strcmp(model, "GX20"))
          adobe_coeff(make, model);
    }
  if (width == 4352 && (!strcmp(model,"K-r") || !strcmp(model,"K-x")))
/*RT*/   {          width  = 4308; filters = 0x16161616; }
  .
  .
  .
As you can see, it almost co-incides with one of your existing patches.

I will try out ColorMatrix1 shortly.

Reported by David.Melville.Edwards on 2013-09-04 07:16:53

Beep6581 commented 9 years ago 
ColorMatrix1 produces good skin tones, but:
-    Blues are greenish
-    Reds and Greens are dulled; not as saturated as they should be.
-    Reds tend towards the orange
-    Greens tend towards blue or brown

Reported by David.Melville.Edwards on 2013-09-04 07:45:09

Beep6581 commented 9 years ago 
I went and got Samsung GX20 Adobe Standard.dcp, and had a look at its dcptool decode.

"When All Else Fails, Read The Manual". Puzzled by the pair of colour matrices, I looked
at the DNG specification, and lo and behold, one is supposed to interpolate between
the pair of ColorMatrices depending on the White Balance. Note, however, the Colour
Temperature reported by RawTherapee depends on what you put in the ColorMatrices.

ColorMatrix1 should be used at or below Standard A, 2865K.

ColorMatrix2 should be used at or above D65, 6500K.

In between, the matrix element for a temperature T would be:
(1/T - 1/6500)/(1/2865 - 1/6500)*ColorMatrix1_element + (1/2865 - 1/T)/(1/2865 - 1/6500)*ColorMatrix2_element.

dcraw only has provision for a single matrix, AFAIK, so it would reasonably be the
average of ColorMatrix1 and ColorMatrix2, which corresponds to a colour temperature
of 3977K.

Using this average approach, RawTherapee reports sensible values for White Balance
temperatures. However, the resulting colours still look pretty grim compared with the
out-of-the-camera stuff.

The average matrix values would be

{10354,-3632,-411,-6982,15457,1546,-1647,2766,7440}

Reported by David.Melville.Edwards on 2013-09-07 01:58:29

Beep6581 commented 9 years ago 
Perhaps one last remark before leaving everyone in peace to deal with more urgent matters.

I remain puzzled by the disjoint between the 'correct values'
-   Computed from ColorMatrix1 and ColorMatrix2
-   Giving expected WB colour temperatures (>6000K during the day)
-   And giving images that look hideous
and values that give a more pleasing appearance
-   Images that look like the out-of-camera JPEG's (and thus, look right)
-   Giving a Camera WB temperature ~4000K reported by RT4 of 4000K or even less.

'Samsung GX20 Adobe Standard.dcp' also includes ForwardMatrix1 and ForwardMatrix2,
whose two sets of values are the same.

As a colour matrix on its own, ForwardMatrix1 isn't very good, but I can get close
to one of the 'by eye' better solutions by mixing in a proportion of the ForwardMatrix1
values to the average of ColorMatrix1 and ColorMatrix2.

Which screws up the WB colour temperatures.

FWIW, SilkyPix (the program that came with the camera) can produce JPEG's that look
exactly like those out of the camera. If anyone knows how SilkyPix does the job, and
where it finds the parameters that it uses, or could point me to relevant resources,
I would be happy to extract the relevant data from my installation, and share it.

Reported by David.Melville.Edwards on 2013-09-08 11:40:10

Beep6581 commented 9 years ago 
I believe I have finally cracked this.

The reason that the Color Matrices in the GX20 DNG don't work is that they are obfuscated
(Why, FFS?).

If you think of the matrices as three rows of three columns of 3 columns, and you subtract
1/2 from the values in the first column, add 1 to the values in the second column,
and add 0.5 to the values in the third column, you get what appears to be the right
answer. The images look right, and the reported colour temperatures are plausible.

The correct values are (averaging the two matrices, normalising and multiplying by
10000)

{ "Samsung GX20", 0, 0,
{4912,-1677,-127,-2976,6747,1215,-1033,1818,3579}},

It is quite a bit different to the K20D matrix.

Of course, you may prefer to fix up the code that reads the DNG ...

I have sent these details to Dave Coffin also.

BTW, I noticed that the matrix for the Pentax K-x in dcraw.c is different to the matrices
in the K-x DNG, if you ask the K-x to give you a DNG. I mention it because Rawtherapee
reports crazy colour temperatures for Pentax K-x PEF files, and sensible values for
DNG files. Is anyone interested?

Reported by David.Melville.Edwards on 2013-09-11 18:55:20

Beep6581 commented 9 years ago 
If anyone is interested, here is Dave Coffin's take.
____________________________________________________
     Maybe so, but DNG is a standard format, and until this
"obfuscation" is written into the standard, it's not going in
dcraw.

     Matrices in adobe_coeff() are constructed from a standard
color chart of at least 24 patches under a D65 lamp or strong
sunlight.  They are based on exact measurements, not guesswork.

                Dave Coffin  9/11/2013
_______________________________________________________________
As the owner of a GX20 who wants to use RawTherapee, I wouldn't take such an absolutist
view. However, even if you share my opinion, you may still prefer to base any patch
on Color Matrix 2 (6500K) rather than the average.

These are the obfuscated Color Matrices from a GX20 DNG
_______________________________________________________
Color Matrix 1                  : 2.564529419 -1.74029541 -0.5341949463 -0.7818756104
1.977798462 0.03483581543 0.04248046875 -0.1977539062 1.037200928
Color Matrix 2                  : 2.77166748 -1.74029541 -0.5779266357 -0.8450317383
1.977798462 0.03768920898 0.04591369629 -0.1977539062 1.122131348
_______________________________________________________
This is what I believe Color Matrix 2 ought to be.
_______________________________________________________
2.277166748 -0.74029541 -0.0779266357 -1.3450317383 2.977798462 0.53768920898 -0.4540863037
.8022460938 1.622131348

Reported by David.Melville.Edwards on 2013-09-12 06:09:03

Beep6581 commented 9 years ago 
from #20 .. "colors remain terribly wrong with Adobes matrix. Even stranger is that
the SAMSUNG GX20 Adobe standard.dcp (which includes the same matrix) if used as custom
profile gives almost correct colors."

I believe that maybe here lies the key to solve the problem. It could be the combined
use of color matrix2 and forward matrix2 when we use Adobes dcp profile ...

I don't have time to read the DNG specs and what exactly are these "forward matrices"
but .. just wondering if for example a multiplication of color matrix2 with forward
matrix2 will give a correct matrix for Dcraw.c ??.

Reported by iliasgiarimis on 2013-09-12 08:03:58

Beep6581 commented 9 years ago 
I did read the DNG Specification. The Forward Matrices are used in a later stage of
the processing.

The Specification describes how to use the two Color Matrices (notice my concession
to US spelling) iteratively. To condense (hopefully retaining correct meaning)
- Pick a colour temperature (which in effect, the average of the two does; 3900K or
so)
- Derive the matrix (as above)
- Compute the colour temperature from the matrix and the camera white point values
- Use this temperature to derive a new matrix
- Repeat until the temperature converges.

The reason that the Samsung GX20 Adobe Standard.dcp does not produce correct colours
is that the Color Matrix values are the same as for the Pentax K20D.

DXO Labs found the Samsung GX20 sensor demonstrated higher sensitivity than the K20D
when they tested it; it isn't exactly the same sensor.

Comparing the values that I advocate above with the K20D, and thus the Adobe values,
the difference appears to be in the colour filters. On the GX20, they let through more
light (hence the higher sensitivity) but they react more to the 'wrong' colours.

Reported by David.Melville.Edwards on 2013-09-14 09:13:32

Beep6581 commented 9 years ago 
David hi,

At first, where did you read this higher sesnsitivity in DxO measures ??. ISO sensitivity
is exactly the same at low ISOs and the color matrices are almost the same just a slight
difference which most possibly is due to sample variation and statistical error. I
am sure you (and me and 95% of people) can not perceive such slight differences. You
can check this by copying the Below values in RT color tab channel mixer. First row
in red second in green .. Then use the "no profile" option in color management.  

Pentax K20D D50
    R sRGB  G sRGB  B sRGB
R raw   1.82    -0.66   -0.15
G raw   -0.14   1.34    -0.2
B raw   0.02    -0.45   1.43

Samsung GX20 D50
    R sRGB  G sRGB  B sRGB
R raw   1.78    -0.64   -0.15
G raw   -0.13   1.3     -0.18
B raw   0.03    -0.45   1.42

BTW if anybody knows how to translate DxO matrix format to Dcraw's please give a hand
..

Secondly something strange (looks like an error to me), when converting Samsung v1.1
DNGs to v1.4 DNGs with Adobe DNG converter adobe uses different colormatrix2 for Samsung
GX20 on most files. 
On the six samples you provided only the first has 0.9427 -0.2714 -0.0868 -0.7493 1.6092
0.1373 -0.2199 0.3264 0.718 
and the rest have 2.3213 -1.4575 -0.484 -0.7077 1.6564 0.0316 0.0385 -0.1656 0.9398

With the second matrix colors are way off but the first one (on the converted samsung_gx20_01_1.dng)
looks OK !!!.

Reported by iliasgiarimis on 2013-09-14 16:45:02

Beep6581 commented 9 years ago 
From comment #35
> where did you read this higher sesnsitivity in DxO measures ??
>
Follow the link in comment #20.

I only provided one DNG, the one in the attachments to comment #18 and #19 above, which
is a picture of a screen displaying a Gretag-Macbeth ColorChecker chart. I know nothing
about the others.

Where did you get the matrices in comment #34 from? They are D50 matrices. The 2 matrices
from my Samsung GX20 DNG, and also in 'Samsung GX20 Adobe Standard.dcp' are Standard
Light A (Color Matrix 1) and D65 (Color Matrix 2).

Using the values from my DNG, unobfuscated, using the interpolation algorithm mandated
by the DNG 1.4 specification (which I have reproduced in comment #29 above) and scaling
to get the first element approximately equal to 1.82, I reckon the D50 matrix would
be

1.81856409  -0.6056962445   -0.0552978218
-0.7857389402    2.4363805598   0.439375517
-0.3721893499   0.6563831677    1.3107677325

This does not look at all like either of the D50 matrices in comment #35.

The matrix 2.3213 ... is obfuscated, I believe. Which means that to use it, you need
to correct it, as I indicated above.

2.3213 -1.4575 -0.484 -0.7077 1.6564 0.0316 0.0385 -0.1656 0.9398 
-.5    +1      +.5    -5      +1    +.5    -.5    +1      +.5

Which would give you:

1.8213 -0.4575 0.016  -1.2077 2.6564 0.5316 -0.4615 0.8344 1.4398

Multiplying by 2500 gives

4553 -1144 40 -3019 6635 1579 -1153 2086 3849

Compare this with

4912,-1677,-127,-2976,6747,1215,-1033,1818,3579

which are the values I suggested in comment #31, appropriate for about 3977K.

My GX20 always puts the same values in its DNG's (see comment #32). These are different
to any of the values you reference. Unobfuscated versus obfuscated suggests firmware
differences, but there remains the possibility that the values that get written to
the DNG are based on calibration tests conducted by the manufacturer on the individual
camera sensor. Which would mean there is an incentive to use the pair of values in
the DNG (unobfuscated) in preference to hard coded values in dcraw.cc, Samsung GX20
Adobe Standard.dcp, or anywhere else.

Reported by David.Melville.Edwards on 2013-09-14 18:49:05

Beep6581 commented 9 years ago 
Comment #20 is mine !!. You mean the link to Dxo at #18 .. :) well no sesnsitivity diff
there (ISO score is not sesnsitivity)

The samples are from #15, I had an unfinished-corrupted download of your sample :(

The matrices are at DxO color response tab 
http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Pentax/K20D  
http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Samsung/GX-20
They are the result of DxO measures.

I wrote not that GX20 exports DNGs with different matrices. I wrote that after running
the Samsung DNGs through AdobeDNGconverter (untick "skip source image .." they do not
have the same colormatrix2 in all files.!!.

Reported by iliasgiarimis on 2013-09-14 22:16:44

Beep6581 commented 9 years ago 
That should have read:

2.3213 -1.4575 -0.484 -0.7077 1.6564 0.0316 0.0385 -0.1656 0.9398 
-.5    +1      +.5    -.5      +1    +.5    -.5    +1      +.5

Which would give you:

1.8213 -0.4575 0.016  -1.2077 2.6564 0.5316 -0.4615 0.8344 1.4398

Reported by David.Melville.Edwards on 2013-09-14 22:19:20

Beep6581 commented 9 years ago 
Regarding the relative sensitivity of the GX20 and K20D, every graph in the DxO mark
color response tables goes higher on the GX20, DxO got .2 bits of extra portrait colour
depth, and they gave the GX20 a much more impressive overall score. Otherwise the cameras
are identical ...

Having looked at the column headings as well as D50 in the title of the DxO matrices,
and read http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements/Color-sensitivity,
I now understand the remark querying the relationship between the DxO mark Color Matrix
and dcraw's (and by implication, everything else's), and can give the answer.

The DxO matrix maps sensor space to sRGB.

The dcraw (and Adobe) color matrices map sensor space to XYZ.

Reported by David.Melville.Edwards on 2013-09-14 23:54:56

Beep6581 commented 9 years ago 
The following procedure will give you a colour matrix you could use in dcraw from the
quoted D50 Matrix at DxO mark.

Take as an example the GX20.

Samsung GX20 D50
    R sRGB  G sRGB  B sRGB
R raw   1.78    -0.64   -0.15
G raw   -0.13   1.3     -0.18
B raw   0.03    -0.45   1.42

Invert the matrix

0.5836472297  0.322841241   0.102576414
0.0592575544    0.8373105668      0.11239756
0.0064482272    0.2585243083      0.7376771898

Each row adds up to 1. These are the impacts on the sensors of monochrome primaries,
normalised.

Divide the values in the first row by the DxO Red White Balance Scale Factor (1.98)
and the values in the third row by the DxO Blue White Balance Scale Factor (1.14).

0.2993062717    0.1655596108      0.0526032892
0.0592575544      0.8373105668      0.11239756
0.0056563396     0.2267757091    0.6470852542

Invert the matrix

3.471    -0.64   -0.171
-0.2535  1.3     -0.2052
0.0585   -0.45  1.6188

Multiply this matrix by the D50 sRGB to XYZ (from http://www.brucelindbloom.com/index.html?Eqn_RGB_XYZ_Matrix.html)

0.4360747    0.3850649  0.1430804
0.2225045    0.7168786   0.0606169
0.0139322    0.0971045  0.7141733

invert the matrix result, multiply by 10000, and round, to give:

7797     -1787   -674
-6257    14829   1569
-1577    2772    9141

The same approach for the K20D gives

7037     -1614   -583
-5983    14386  1668
-1357    2541    8862

dcraw.c has for the K20D (which would be D65)

9427    -2714   -868
-7493    16092  1373
-2199    3264   7180

The same approach for the Sony Alpha A58 (a camera I chose at random) gives

5926    -1536    -390
-5039   12475    2863
-1091   1639     6447

Compare these with the values in dcraw.c

5991 -1456  -455
-4764 12135 2980
-707  1425  6701

The agreement seems too close to be a coincidence, but if anyone can think of a rationale
for the last matrix inversion, let me know. I just recognised the output of the second
to last step as the inverse of something sensible.

Reported by David.Melville.Edwards on 2013-09-16 21:12:28

Beep6581 commented 9 years ago 
David hi, thanks for investigating in depth ..

#39 "Regarding the relative sensitivity of the GX20 and K20D, every graph in the DxO
mark color response tables goes higher on the GX20, DxO got .2 bits of extra portrait
colour depth, and they gave the GX20 a much more impressive overall score. Otherwise
the cameras are identical ..."

I think 0.2 bits is a very small difference, could be sample variation.  
These scores/graphs that you mention are mostly affected by the read noise which is
a bit lower for GX20. The graph which is indicative of sensitivity (we'd better call
it Quantum efficiency = photons falled on sensor/ photons registered as electrons)
is SNR18% which is about the same for the two models with an even lower difference
(<= 0.4 db = 1/7 stops at most) better curve for GX20.

Regarding color matrices, are you sure that we have to include WB factors in the calculation
?.

In any case .. I think that 
- DNGs dcps and dcraw use the reverse order (first line and column are blue) and 
- D65 instead of D50 so I think a bradford adaptation  http://www.brucelindbloom.com/index.html?Eqn_RGB_XYZ_Matrix.html
should be a step to convert Dxo's matrix to Dcraw's.

Reported by iliasgiarimis on 2013-10-07 00:44:57

Beep6581 commented 9 years ago 
Yes, I am sure the numbers labelled by DxO as White Balance do need to go into the calculation.
It sounds wrong, I know, but they aren't really White Balance. Rather, they are normalisation
factors that have been applied to the numbers before DxO tabulate and graph them. We
need to undo this normalisation to get back to really RAW numbers.

Reported by David.Melville.Edwards on 2013-10-09 08:52:33

Beep6581 commented 9 years ago 
Fixed with revision d455f2486880

Reported by heckflosse@i-weyrich.de on 2014-02-02 14:01:03