dcraw update tracker

[Beep6581]

Originally reported on Google Code with ID 1627

Umbrella issue for dcraw update requests

Reported by entertheyoni on 2012-11-19 17:30:07

[Beep6581]

There is also a border Issue with Sony A7R ARW files.

Reported by heckflosse@i-weyrich.de on 2014-01-28 14:23:35

[Beep6581]

Just checked the new dcraw version. It only adds three cameras:
Nikon D3300, Panasonic DMC-TZ61 and Sony ILCE-5000

No bugfixes.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-28 14:44:11

[Beep6581]

OK we can have the correct frames with camconst.json, Only missing feature in current
camconst.json is the pixel order .. as it is now there is a conflict between native
raws and converted to DNG.

There is another conflict regarding the frame, sometimes the converted to DNG raw does
not include all the frame .. some parts are clipped out .. If the right/bottom borders
are clipped the solution is easy, use only absolute numbers in camconst.json (left/top
borders, width, height) and not the alternative (clip columns/lines from the end by
using negative numbers).

Ingo, Anders , any idea about where can we change the pixel order in dcraw.cc ?. 

Or alternatively one can change the order for the framing putting first the exif data
in case of DNG files (now first is the camconst.json)

Reported by iliasgiarimis on 2014-01-28 15:39:00

[Beep6581]

The pixel order is coded in the variable with name 'filters'. But I wouldn't change
it in dcraw.cc, a better place would be in rawimage.cc RawImage::loadRaw in case the
top_margin (that's the value from camconst.json) is odd. You can find a description
of the coding at the beginning of dcraw.cc

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-28 16:17:30

[Beep6581]

Addition to #51:

    static const ushort canon[][11] ....
   { 4192, 3062,  96, 17, 24,  0, 0, 16, 0, 0, 0x49 }

the 0x49 is the coding of the pixel order and gets expanded to 0x49494949 in filters.
To consider an odd border difference for rgb bayer sensor it should be sufficient to
rotate right or left the content of filters by 4 bits.

for example in line 251 of rawimage.cc just below cc->get_rawCrop(lm, tm, w, h);
insert:

if(((int)top_margin - tm) & 1) // we have an odd border difference
   filters = (filters << 4) | (filters >> 28); // left rotate filters by 4 bits

Ilias, can you try that?

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-28 17:59:24

[Beep6581]

I can after 1-2 hours (to have available my compiling machine ..), thanks. 

How do I know if it needs left or right rotation ?. 

We have at least 4 combinations just for this line change depending on the starting
pixel ..

Reported by iliasgiarimis on 2014-01-28 19:39:17

[Beep6581]

Hi Ilias, by rotating 4 bits 0x49494949 becomes 0x94949494 and 0x16161616 becomes 0x61616161
and vice versa. For odd top margin differences that should be sufficient in all cases.
It also doesn't matter if you rotate left or right in this case.
Here's a patch with the change from #56, the upgrade to latest dcraw version and I
also stuffed a memory leak which occurred only with Sony Files.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-28 20:12:35

---

• _Attachment: issue1627_02.patch_

[Beep6581]

There are two changes in dcraw.c that could cause integration problems.  cblack[] now
allows black levels to be mapped by location (up to a 64x64 pattern) as well as by
color.  If possible, scale_colors() maps the former into the latter for better performance.
 I no longer attempt to do this mapping while parsing metadata because the filter pattern
may not have been set yet.

The other pitfall is the use_camera_matrix flag, whose meaning has changed:

  if ((use_camera_matrix & (use_camera_wb || dng_version))
       && cmatrix[0][0] > 0.125) {
    memcpy (rgb_cam, cmatrix, sizeof cmatrix);  // use metadata matrix
    raw_color = 0;         // prevent subsequent call to adobe_coeff()

There are three settings for use_camera_matrix:
3 -- always use the embedded matrix if there is one.  The "+M" option.
1 -- use the embedded matrix if "-w" is set or this is a DNG file.  The default.
0 -- ignore the embedded matrix and call adobe_coeff() instead.  The "-M" option.

The matrices in adobe_coeff() contain black levels, because color rotation doesn't
work if you haven't correctly zeroed the original colors.  Problem is, DNG files might
be multiplied by e.g. four, which is incompatible with the black level in adobe_coeff().
 If you've made a habit of always doing "dcraw -M" (because this option used to have
no effect on DNG files), some DNG files will now appear foggy.

The only DNG camera I've seen that needs the "-M" option is the Samsung GX20.

Reported by dcoffin@cybercom.net on 2014-01-28 20:50:17

[Beep6581]

#56 Works fine !. :)
Can you explain what exactly the "odd border difference" is ?. Is it the dif between
dcraw and camconst borders ?.

#58 So Gaetano will no more have instabilities. When I replicated the same instabilities
I was also using Sony ARW files mostly. FINE 

Reported by iliasgiarimis on 2014-01-28 20:57:19

[Beep6581]

re #60:

It's when the difference between original border (from dcraw) and border defined in
camconst.json is 1,3,5,7,9,....

I can't guarantee that there are no more instabilities. I just fixed a small (around
100 kb per file) memory leak with Sony files. That leak was not in original dcraw but
in RT since some time (don't know when it was introduced).

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-28 21:11:06

[Beep6581]

re #59: Dave, thanks a lot for this informations. I'll check, whether I did everything
right when integrating latest dcraw in RT.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-28 21:15:03

[Beep6581]

For Sony A700 .. The BlackLevel is 126 (12bit) in Dcraw.c dcraw.cc rawimage.cc while
Sony defines it as 512 14bit (128 12bit) in ARW exif.Maker Data -SR2subIFD:0x7310

Fine if there is a reason (David ?) to keep it at 126, but as most Sonys have a BL
at 128 and generally the maker data are reliable shouldn't RT use the BL=128 (512)
also ?. A difference of two 12bit levels for BL is significant enough to ignore ..

Reported by iliasgiarimis on 2014-01-28 21:54:53

[Beep6581]

I also saw that there is a difference, but I didn't know what's correct, so I relied
on the correctness of the value in dcraw. We should define how to handle this values.
Is there any necessity for the hardcoded overrides in rawimage.cc or could we move
this overrides completely to camconst.json?

Reported by heckflosse@i-weyrich.de on 2014-01-28 22:03:49

[Beep6581]

#64 Interested in Anders's thoughts on this ..

I wouldn't say there is a necessity but helps to keep the  camconst.json file compact.
Why have we to load camconst.json (a user accessible & editable file) with hundreds
of models for which we are sure that we have correct camera constants, instead of hardcoding
in rawimage.cc. For example I would delete A700 from camconst if this BL difference
didn't exist.  

As it is now the camconst.json is easily editable by the users and this is a bit dangerous.
More than this we have two instances of it one in the installation folder (callit officia
camconst.json), the other in the user cash folder. I think the second could have a
different name (say "usercamconst.json)" name and serve for user changes experiments
etc while the first should be not easily editable .

I think that camconst.json serves very well as an easy way for fast reactions by not
programmers (like me) on some errors (which can pass in rawimage.cc if stabilized)
and for new models preliminary support. 
The other part of camconst.json (constants per ISO and per aperture) is (and will be
for long) a work in progress and in need for user input (which is missing up to now)
so it's better to keep it accessible but with editing restrictions. 

Reported by iliasgiarimis on 2014-01-28 23:21:59

[Beep6581]

Included patch from Issue 2043 into this patch because it does affect dcraw.cc and is
safe (only a #pragma omp parallel for). If there are any complaints, I can remove this
line.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-28 23:31:45

[Beep6581]

Hi Ingo, thank you for this patch and the 2043. I tried a variety of raw files and cannot
see any problems.

Reported by michaelezra000 on 2014-01-29 04:02:23

[Beep6581]

iliasgia, here's a simple technique to find your camera's exact black level:

Shoot a calibrated chart that has neutral patches at various brightness levels.  Select
a rectangle of a medium gray color patch, and pass its coordinates to dcraw e.g.:

dcraw -k 50 -A 1652 1233 86 86 testfile.raw

Now load the output file and crank up the color saturation.  The patch you selected
should be a random mottling of red, green, blue, and white.  If the other neutral patches
don't look like this, change the "-k" value.  When you get close to the correct value,
you'll see that a +-1 change has a noticeable effect.

Reported by dcoffin@cybercom.net on 2014-01-29 04:19:24

[Beep6581]

David, thank you very much for your input here.

Regarding the black level of Sony A700 do you insist that the correct value is 126
(as is hardcoded in Dcraw) and this value was a result of a proper measure ?.

Can we try the above BL measuring method with raw samples from Dpreview/Imaging recource
containing colorcheckers or step wedges ?. 
Which is the order and the physical meaning of these four coordinates ?

Ilias

Reported by iliasgiarimis on 2014-01-29 11:24:39

[Beep6581]

I removed issue1627_03.patch because it causes artifacts on Linux. Please use only issue1627_02.patch

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-29 12:07:41

[Beep6581]

Here's the corrected version. Strange, that this error doesn't occur on Windows.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-29 12:23:35

---

• _Attachment: issue1627_04.patch_

[Beep6581]

Hi Ingo, no artifacts on Windows with patch 04 as well:)

Reported by michaelezra000 on 2014-01-30 13:02:09

[Beep6581]

Hi Michael, thanks for testing :-)

Reported by heckflosse@i-weyrich.de on 2014-01-30 13:12:19

[Beep6581]

iliasgia, I set the A700 black level to 126 back in early 2010.  I just checked against
the current Adobe DNG Converter, and the correct black level is 128.  I'll change it
in the next dcraw release.

Reported by dcoffin@cybercom.net on 2014-01-31 04:03:45

[Beep6581]

Thanks David.

What about the raw crop of Canon Powershot S120.cr2s 
Read #51

Reported by iliasgiarimis on 2014-01-31 04:38:22

[Beep6581]

Ilias, could you have a look at files from Nikon D610, Sony ILCE-7R (A7R) and Sony ILCE-3000
(A 3000)? For this cameras a right border has to be defined.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-31 10:55:01

[Beep6581]

Ingo, yes I'll look in the next hours, all the models that gained support with dcraw
9.20 and some older .. and report.

But as David is reading this issue .. may be we have to wait a bit for his reaction
after the above report.

Will you put the corrections in dcraw.patch/dcraw.cc, in rawimage.cc or do you prefer
camconst.json ??. I can write the camconst.json entries if you like ..

Regarding rawimage.cc overrides that already exist .. most of them are about a different
matrix provided by Malcolm Wilson (Huelight, a well known color expert who also sells
dcp profiles). These matrices were changed from dcraw defaults to match the icc color
profiles that Huelight built for RT back then when there was no dcp support in RT.
With the current state (dcp support) there is no need to have matched matrices. In
fact when we use dcp profiles these matrices are used only for WB calculations.

Now the "problem" is that these matrices are built under a different illuminant (D50
) than the default D65 and the (on these matrices) calculated WB temp/tint are also
different. It's not a big difference but increases as we go to colder temps (shade,
underwater).
More than this the calculated WB temps/tints are used by other parts of the code (Ciecam02).

I have the impression that WB calculations suppose that the matrix is a D65 one and
if fed with a D50 data they go a bit off to the wrong side. 

If Jacques is reading .. he could give his opinion if it's better to revert to the
D65 (default) matrices .. 

Reported by iliasgiarimis on 2014-01-31 14:38:35

[Beep6581]

Hi Ilias,

thanks for looking into it. I would like to commit with the change mentioned by Dave
in #74. The other changes (crop or whatever) can be made in a separate patch afterwards.
In my opinion, it would be better to put the corrections in camconst.json until they
are corrected in dcraw.
I put Jacques into Cc:

Here's the list of files I tested yesterday or not. I only looked at the borders not
at the black levels, but colors looked good for all of them.

Canon EOS 70D             works fine
Canon EOS C500            no raw file found
Canon PowerShot G16       works fine
Canon PowerShot S120      works fine but crop needed
Fujifilm X-A1             works fine
Nikon D3300               no raw file found
Nikon Df                  works fine
Nikon D5300               works fine
Nikon D610                works fine but crop needed for right border
Nikon COOLPIX P7800       works fine
Nikon 1 AW1               works fine
Olympus E-P5              works fine
Olympus E-PL6             no raw file found
Olympus E-M1              works fine
Olympus STYLUS1           works fine
Pentax K-3                works fine
Panasonic DMC-FZ7         no raw file found
Panasonic DMC-LF1         works fine
Panasonic DMC-GM1         works fine
Panasonic DMC-GX7         works fine
Panasonic DMC-TZ61        no raw file found
Samsung EK-GN120          no raw file found
Samsung GX20              no raw file found
Sony DSC-RX100M2          works fine
Sony DSC-RX10             works fine
Sony ILCE-7R (A7R)        works fine but crop needed for right border
Sony ILCE-7 (A7)          works fine
Sony ILCE-3000 (A3000)    works fine but crop needed for right border
Sony ILCE-5000 (A5000)    no raw file found
Sony NEX-5T               works fine

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-31 15:34:47

[Beep6581]

Addition: I didn't test with DNG files from these cameras with the exception of Pentax
K-3

Reported by heckflosse@i-weyrich.de on 2014-01-31 15:36:32

[Beep6581]

FZ7 is also OK. In fact it's FZ70 http://www.imaging-resource.com/PRODS/panasonic-fz70/FZ70hSLI0100NR1.RW2.HTM

I wonder why does Dcraw have this feature (to use the best match even with less characters)!!.

It makes unexpected things sometimes as with when RX-100II was not supported dcraw
found the best match in RX-100 and used it's color matrix (which was way off).
Or now with Oly EM-10 which Dcraw opens as if it was EM-1 ..

Samsung GX-20 has still wrong colors in RT while it's OK with Dcraw 9.20 ...
http://filebin.net/64ht209z9f   (will be fully uploaded in 30 minutes)

I lost some hours at the doctor today (now I hear better :) ) so I'll have the camconst.json
ready tomorrow ..

Reported by iliasgiarimis on 2014-01-31 21:38:29

[Beep6581]

Arghh, wanted to commit the patch today. But detected that Dave made a new version and
changed a calculation. So I have to check this thing...

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-31 21:39:10

[Beep6581]

From #59 David says ..
"There are three settings for use_camera_matrix:
3 -- always use the embedded matrix if there is one.  The "+M" option.
1 -- use the embedded matrix if "-w" is set or this is a DNG file.  The default.
0 -- ignore the embedded matrix and call adobe_coeff() instead.  The "-M" option.

The only DNG camera I've seen that needs the "-M" option is the Samsung GX20."

Reported by iliasgiarimis on 2014-01-31 21:43:43

[Beep6581]

Ilias, there is already a new Version:

   $Revision: 1.461 $
   $Date: 2014/01/31 04:05:31 $

and there's a changed calculation in this version, which I want to check, whether it
has influence on RT.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-31 21:49:49

[Beep6581]

Hi Ilias, if you want to check the changed calculation of black levels in latest 1.461
dcraw, without waiting for a patch, you can change line 63 of rawimage.cc from 

cblack_[FC(c/2,c%2)] = this->get_cblack(6 + c/2 % this->get_cblack(4) * this->get_cblack(4)
+ c%2 % this->get_cblack(5));

to 

cblack_[FC(c/2,c%2)] = this->get_cblack(6 + c/2 % this->get_cblack(4) * this->get_cblack(5)
+ c%2 % this->get_cblack(5));

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-31 21:56:51

[Beep6581]

Hi Ilias, while walking with the dog, I thought about the change. It has only influence
on black-level calculation if the matrix is not a 1x1 and not a 2x2 matrix. Only in
case of 2x1 or 1x2 it's different.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-31 22:33:15

[Beep6581]

Will have a look how to handle the Samsung GX20 in RT

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-31 22:49:04

[Beep6581]

Aha that's why I saw no changes ..

I experimented with camconst.json for GX20 files. Although RT reads the matrix from
camconst.json the colors are still wrong but not on all samples, sample 3 looks OK
!

Reported by iliasgiarimis on 2014-01-31 23:09:12

[Beep6581]

Here's the patch for Samsung GX20

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-31 23:12:03

---

• _Attachment: issue1627_05.patch_

[Beep6581]

Works fine !!.

What was the problem ?.

Reported by iliasgiarimis on 2014-01-31 23:54:07

[Beep6581]

So issue 979 is fixed :) :)

http://code.google.com/p/rawtherapee/issues/detail?id=979

Reported by iliasgiarimis on 2014-01-31 23:56:53

[Beep6581]

As I saw, that dcraw has to be called with a special parameter (-M) for this files,
it was clear that we also had to treat it specially. Have a look at line 8321 of dcraw.cc

Thanks for testing, Ilias! :-)

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-31 23:59:55

[Beep6581]

re #90: Thanks for the hint. Will close it, as soon as the dcraw patch is committed
:-)

Reported by heckflosse@i-weyrich.de on 2014-02-01 00:02:31

[Beep6581]

#90

So this means that to make use of a different matrix in camconst.json than the embedded
in DNGs  this -M is needed also ?
With my test with camconst.json RT reported that the camconst matrix were used but
the result was the (wrong) !! :(

Reported by iliasgiarimis on 2014-02-01 00:08:26

[Beep6581]

Don't think so, but will have a look tomorrow to give you a correct answer. No reason
for :-( There's a solution for everything ;-)

Reported by heckflosse@i-weyrich.de on 2014-02-01 00:16:37

[Beep6581]

Panasonic TZ61 is the European version for TZ60. Here the trick using TZ6 instead of
TZ61 would help Dcraw.cc decode both versions with correct colors and BL/WL.

But the crop is not OK. I'll try with camconst.json ..

Reported by iliasgiarimis on 2014-02-01 13:15:00

[Beep6581]

Ilias, can you update the list from #78 when you completed your tests?

Ingo

Reported by heckflosse@i-weyrich.de on 2014-02-01 22:28:06

[Beep6581]

Ingo, I will update when it's finished, 

Progress is sloooow :(

Reported by iliasgiarimis on 2014-02-01 23:45:45

[Beep6581]

Changed my mind .. instead of waiting to complete fine details in camconst.json I just
update the list of new cameras now to free up Ingo and the completed camconst.json
file tomorrow in the evening (GMT+2)

Canon EOS 70D             works fine, camconst entries for per ISO and apperture are
almost ready.
Canon EOS C500            no raw file found
Canon PowerShot G16       works fine
Canon PowerShot S120      works fine, the crop needed is implemented in camconst json
Fujifilm X-A1             works fine
Nikon D3300               no raw file found
Nikon Df                  works fine
Nikon D5300               works fine
Nikon D610                works fine, crop needed for right border implemented in camconst
Nikon COOLPIX P7800       works fine
Nikon 1 AW1               works fine
Olympus E-P5              works fine
Olympus E-PL6             no raw file found
Olympus E-M1              works fine
Olympus STYLUS1           works fine
Pentax K-3                works fine
Panasonic DMC-FZ7         works fine for FZ70/FZ71?, a bit more crop needed in camconst.json
to behave better with auto-distortion correction 
Panasonic DMC-LF1         works fine
Panasonic DMC-GM1         works fine
Panasonic DMC-GX7         works fine
Panasonic DMC-TZ61        works fine for TZ61, expanded in camconst to cover TZ60,
ZS40/41 (same camera different names). Tighter crop as with FZ70. 
Samsung EK-GN120          works fine, it is more known as "Samsung Galaxy NX"
Samsung GX20              works fine
Sony DSC-RX100M2          works fine
Sony DSC-RX10             works fine
Sony ILCE-7R (A7R)        works fine, crop needed for right border is implemented in
camconst.json
Sony ILCE-7 (A7)          works fine
Sony ILCE-3000 (A3000)    works fine, crop needed for right border is implemented in
camconst.json
Sony ILCE-5000 (A5000)    no raw file found
Sony NEX-5T               works fine

Reported by iliasgiarimis on 2014-02-02 00:39:01

[Beep6581]

Generated dcraw.patch and committed to default.

Reported by heckflosse@i-weyrich.de on 2014-02-02 13:58:56

[Beep6581]

Reported by heckflosse@i-weyrich.de on 2014-02-03 19:05:21

• No longer blocking: #2032

[Beep6581]

dcraw 1.463 is out: http://www.cybercom.net/~dcoffin/dcraw/dcraw.c

Reported by heckflosse@i-weyrich.de on 2014-05-19 23:32:27