dcraw update tracker

[Beep6581]

Originally reported on Google Code with ID 1627

Umbrella issue for dcraw update requests

Reported by entertheyoni on 2012-11-19 17:30:07

[Beep6581]

Issue 1610 has been merged into this issue.

Reported by entertheyoni on 2012-11-19 17:31:24

[Beep6581]

Issue 1605 has been merged into this issue.

Reported by entertheyoni on 2012-11-19 17:33:29

[Beep6581]

dcraw 1.454 is out
http://www.cybercom.net/~dcoffin/dcraw/dcraw.c

Reported by entertheyoni on 2012-12-23 22:01:40

[Beep6581]

Here is the patch

Reported by michaelezra000 on 2012-12-27 00:32:49

• Status changed: PatchSubmitted
• Labels added: 4.1

---

• _Attachment: Issue-1627_dcraw_9.17_v1.patch_

[Beep6581]

I ran some tests, seems to work fine, but would appreciate confirmation from others,
as this is my 1-st time patching dcraw.

Reported by michaelezra000 on 2012-12-27 12:15:46

[Beep6581]

Sorry, not from my side: i never understood how to check dcraw updates. It would be
nice if Oduis could review it (one last time), since he did the previous ones (IIRC).

Reported by natureh.510 on 2012-12-27 12:52:02

[Beep6581]

There were a couple of minor formatting changes (spaces, etc) that I skipped as they
required to be reintroduced manually, just to avoid doing this manual prettification
every time.

Reported by michaelezra000 on 2012-12-27 13:11:07

[Beep6581]

committed in revision ac1d8c52ae30  

Reported by michaelezra000 on 2012-12-28 14:41:58

• Status changed: FixedPendingConfirmation

[Beep6581]

Just to make visible that it's reused for every dcraw update...

Reported by rinni@gmx.net on 2013-03-03 11:53:21

• Status changed: Accepted

[Beep6581]

Issue 1838 has been merged into this issue.

Reported by entertheyoni on 2013-04-26 22:09:40

[Beep6581]

Issue 1882 has been merged into this issue.

Reported by rinni@gmx.net on 2013-06-03 12:22:08

[Beep6581]

 $Revision: 1.456 $
   $Date: 2013/06/16 18:01:08 $
 */

#define DCRAW_VERSION "9.19"

is out

Reported by gaaned92 on 2013-06-27 13:22:45

[Beep6581]

I am looking into applying this update. However, as I execute "patch -p0 < dcraw.patch",
the rejects file is very large, so most of it has to be retrofitted manually. I wonder
if this is expected, or may be there is a step I am missing?

Olli - can you comment on the amount of of rejects you had to deal with manually?

Reported by michaelezra000 on 2013-06-28 14:11:48

[Beep6581]

Here is the patch for dcraw 9.19
I was not sure how to handle specific case on line 7497, so that is commented out.
suggestions are welcome:

    /* RT - does not compile: 
     * could not convert 'fseek(((DCraw*)this)->DCraw::ifp, -6404096, 2)' from 'void'
to 'bool'
     * 
    if (!strncmp(model,"ov",2) && !fseek (ifp, -6404096, SEEK_END) &&
    fread (head, 1, 32, ifp) && !strcmp(head,"BRCMn")) {
      strcpy (make, "OmniVision");
      data_offset = ftell(ifp) + 0x8000-32;
      width = raw_width;
      raw_width = 2611;
      load_raw = &CLASS nokia_load_raw;
      filters = 0x16161616;
    } else */ is_raw = 0;

I ran some basic testing, opening various raw formats and did not see any issues.

Reported by michaelezra000 on 2013-06-28 20:14:28

• Status changed: PatchSubmitted

---

• _Attachment: RT_dcraw9.19_01.patch_

[Beep6581]

I can't apply this patch in TIP, problem in dcraw.cc and dcraw.patch.

Reported by natureh.510 on 2013-06-29 20:19:20

[Beep6581]

oh my... I made this in a local repository which apparently was not at tip...
But just recently I updated it to the tip, after when the patch was reverted...
Is there any way to determine to which revision would this patch apply?

Reported by michaelezra000 on 2013-06-29 20:41:22

[Beep6581]

No idea. When creating the patch, you should have an option to store the changeset number
iirc, but your patch doesn't contain any.

Reported by natureh.510 on 2013-06-29 21:10:57

[Beep6581]

OK, I will fix it and upload a correct patch.

Reported by michaelezra000 on 2013-06-29 21:34:15

[Beep6581]

Please try this patch

Reported by michaelezra000 on 2013-06-29 21:37:15

---

• _Attachment: RT_dcraw9.19_02.patch_

[Beep6581]

Patch applies and build fine. I've solved the issue of comment #15 : the standard fseek
function is replaced by RT's own version, returning void. So the implementor of the
new fseek assumes that it can't fail, even if it could fail when seek past the end/begin
of the file. I've then rewrote that test so it can build.

So we have an XTrans demozaising algorithm now :). Would it be possible to use or copy
this function from dcraw?

Reported by natureh.510 on 2013-06-29 22:28:37

---

• _Attachment: RT_dcraw9.19_03.patch_

[Beep6581]

Thanks, Hombre. We should probably add a comment that this is RT enhancement, to ease
the next dcraw update.

I am curious why the patches 02 and 03 are so different in file size. Comparing the
two, there are many other differences.

Reported by michaelezra000 on 2013-06-29 22:54:26

[Beep6581]

Yes, I've seen that too, but it looks like that my MSYS' version of diff handles the
blank lines differently. Just compare patch 02 and 03 in Eclipse, they should produce
the same resulting file once applied (excepted my changes).

Reported by natureh.510 on 2013-06-30 00:04:10

[Beep6581]

I manually transferred your change to into dcraw.cc, here is version 04 which might
be final then. If no complaints I will commit in a couple days.

About the xtrans, we could probably use it in RT if no licensing restrictions.

Reported by michaelezra000 on 2013-06-30 01:12:42

---

• _Attachment: RT_dcraw9.19_04.patch_

[Beep6581]

Hi
It works again, many thanks.
Best regards

Cordialement

D. GAUDRY  Ph. D.
ASSISTANT CHEF DE TRAVAUX
LYCÈE CONDORCET MONTREUIL
31 RUE DESIRE CHEVALIER
93100 MONTREUIL SOUS BOIS
TÉL  : 01 48 57 50 63
FAX  : 01 48 57 11 06
CELL: 06 64 19 50 09

Reported by dgaudry@condorcet93.fr on 2013-06-30 04:58:23

[Beep6581]

Support for dcraw 9.19 committed in revision 6885de38a11c

Reported by michaelezra000 on 2013-07-02 15:33:38

[Beep6581]

DCRAW_VERSION "9.20" IS OUT .. http://www.cybercom.net/~dcoffin/dcraw/dcraw.c

Revision: 1.458 $
   $Date: 2014/01/15 00:24:09 $
 */

Reported by iliasgiarimis on 2014-01-16 11:17:11

[Beep6581]

Here's a first patch for dcraw 9.20. Some hunks failed. I patched them manually. Hope
I did everything right. The patch for dcraw.patch follows, when the changes are confirmed.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-18 12:13:25

---

• _Attachment: dcraw920_00.patch_

[Beep6581]

read the patch (not tested it, a bit busy these days) and it looks okay.

Reported by torger@ludd.ltu.se on 2014-01-18 16:58:17

[Beep6581]

Hi,
the patched version fails on DNG files. E.g. this one: http://www.rawtherapee.com/shared/play/playraw35/playraw35_hombre.dng

Sorry, no debugging symbols.

0x00000000009167ce in DCraw::scale_colors() ()
(gdb) bt
#0  0x00000000009167ce in DCraw::scale_colors() ()
#1  0x0000000000883708 in rtengine::Thumbnail::loadFromRaw(Glib::ustring const&, rtengine::RawMetaDataLocation&,
int&, int&, int, double, bool) ()
#2  0x00000000006f0ee0 in Thumbnail::_generateThumbnailImage() ()
#3  0x00000000006f11eb in Thumbnail::upgradeThumbImage(rtengine::procparams::ProcParams
const&, int, double&) ()
#4  0x00000000005a3250 in ThumbImageUpdater::Impl::processNextJob() ()
#5  0x00007ffff6f450b2 in ?? () from /usr/lib64/libglibmm-2.4.so.1
#6  0x00007ffff71d1b12 in ?? () from /usr/lib64/libglib-2.0.so.0
#7  0x00007ffff71d12d5 in ?? () from /usr/lib64/libglib-2.0.so.0
#8  0x00007ffff34c8f3b in start_thread (arg=0x7fffa27fc700) at pthread_create.c:308
#9  0x00007ffff320350d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Reported by Marian.Kyral on 2014-01-19 22:21:06

[Beep6581]

Ok, will have a look.

Reported by heckflosse@i-weyrich.de on 2014-01-19 22:24:04

[Beep6581]

Confirm the bug. 

Here's the bt full of my crash with the same file:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 4064.0x130c]
0x00000000008d92ed in DCraw::scale_colors (this=0x24807de0) at H:\rt_dcraw920\rtengine\dcraw.cc:3878
3878                            i/4 % iwidth % cblack[5]];
(gdb) bt full
#0  0x00000000008d92ed in DCraw::scale_colors (this=0x24807de0) at H:\rt_dcraw920\rtengine\dcraw.cc:3878
        size = 16255800
        fr = 1.33804792e-038
        fc = 6.20516485e-018
        bottom = 0
        row = 8
        col = 8
        dmax = 2.18359375
        img = 0x0
        right = 1
        x = 14127696
        c = 4
        sum = {0, 0, 0, 0, 16, 16, 16, 16}
        dmin = 1
        scale_mul = {4.06443834, 1.86135268, 2.87200904, 1.86135268}
        i = 554402
        y = 0
        val = 331
        pix = 0x112c3120
        ur = 0
        uc = 0
        dark = 0
        sat = 16124
        dsum = {7.7802651614211127e-316, 8.4799134987594853e-315, 1.4234454176879153e-315,
5.1958339535047817e-317, 1.4203096423216384e-315,
          5.1954574754826506e-317, 7.7802651614211127e-316, 2.8924213166299498e-315}
#1  0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)

Searching...

Reported by heckflosse@i-weyrich.de on 2014-01-19 23:04:51

[Beep6581]

In latest dcraw the size of array cblack[] was changed from 4 to 4102. As this array
in RT is declared in dcraw.h I didn't detect this change using the documented way to
make a patch for a new version of dcraw. After increasing the size of this array, it
doesn't crash with this file, but the colors are still wrong. We have to deeply inspect
this patch of dcraw, whether there are more of this changes and I need at least a double
check from another dev to do this.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-19 23:23:10

• Labels added: Priority-High
• Labels removed: Priority-Low

[Beep6581]

Here's a new patch which increases size of cblack[] to 4102. As a result, it doesn't
crash with dng files, but colors are still wrong with some files.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-20 16:46:43

---

• _Attachment: dcraw920_01.patch_

[Beep6581]

1) need to check which RT dcraw bug fixes that should be kept, and which should be removed.
Dcraw 9.20 fixes some bugs that was in 9.19. Most should probably be kept.

2) cblack[] handling has changed somewhat, and cblack is used in rawimage.cc etc, need
to check that. This might be some work. The "scale_colors" function is almost duplicated
in rawimage.cc, and scale_colors cblack handling has changed a little, may be a problem.

3) check changes to dcraw globals and update RT's dcraw.h to mirror that. cblack has
changed to ushort[4102], use_cam_matrix has changed value from -1 to 1, otherwise no
change.

Unfortunately no time for the moment to look into it further, but apart from the cblack
thing it seems like it's not much more to do.

Reported by torger@ludd.ltu.se on 2014-01-20 19:14:29

[Beep6581]

I've removed the bugfix for Issue 2121, because it's fixed in dcraw.

Reported by heckflosse@i-weyrich.de on 2014-01-20 20:10:40

[Beep6581]

#34 Ingo, can you report some DNG files that you checked and found correct colors ?.
Are those files from Nikon cameras (or Pentax K3 .. etc ) where the correct Black Level
is zero or near zero ?. 

With DNG files from Sony A7 RT 4.0.12.23+dcraw920_01.patch cannot recognize the black
levels (reports BL=0 while the correct is 512 ..). 

With ARW files it's OK

http://filebin.net/c8brw96x4e will be up in 10-20 minutes ..

Reported by iliasgiarimis on 2014-01-23 18:26:41

[Beep6581]

Ilias, The K-3 and K-30 dng files seem to be ok. At least the K-30 file from #9 of Issue
2207 looks same as before (I didn't make an exact before/after comparison). You can
find some in Issue 2197 and Issue 2207.

Reported by heckflosse@i-weyrich.de on 2014-01-23 18:50:45

[Beep6581]

Pentax K-5 DNG has wrong colours.

Reported by heckflosse@i-weyrich.de on 2014-01-23 18:53:36

[Beep6581]

Same with Nikon D5300 (the first and only Nikon with BL>0), NEFs work OK, but with DNGs
the recognized BL is 0 instead of 600 ..

Looks like all DNGs are affected with this bug ..

Fuji X-A1 RAFs (normal bayer CFA) again after the patch the BL (it's in exif data)
RT reports BL=0, the not patched build is OK for BL.

Olympus ORFs are recognized correctly (BL read from maker 0x0600 tag).

Reported by iliasgiarimis on 2014-01-23 19:21:18

[Beep6581]

Ingo thanks for the links :)

The files of Pentax K3 and K30 have BL = 0 (or around) and the difference between wrong
(always 0) vs correct are difficult to see .. but RT reports the BL values used, so
it's again wrong .. for example with Marian's sample the correct BL is 0 1 1 2 while
the patched RT reports 0 0 0 0 .. 

Reported by iliasgiarimis on 2014-01-23 19:34:54

[Beep6581]

Maybe this patch for Fuji BL is related with current RTs inability to read the correct
BL ..

http://code.google.com/p/rawtherapee/issues/detail?id=2049#c16 #27 #28 

Reported by iliasgiarimis on 2014-01-23 19:50:28

[Beep6581]

Ilias, thanks a lot for this hint!

Reported by heckflosse@i-weyrich.de on 2014-01-23 20:50:11

[Beep6581]

Ilias, can you try the Fuji X-A1 RAF with original dcraw.exe (you can download 9.20
exe here: http://www.fastpictureviewer.com/bin/dcraw.zip ). I had the same error with
a Canon DNG. There's an error in dcraw 9.20 in case the BL is in exif data. But I think
I know how to fix it in RT. Patch follows today.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-27 16:04:40

[Beep6581]

Ilias, no need to check for #44. Found a Fuji X-A1 RAF. It's a different case than the
Canon DNG I tested ( https://ajdaa.box.com/s/blx8xt4ehpg5v3u13ck3 ). Both are handled
well now in the upcoming patch but latter isn't handled correctly in dcraw.exe 9.20

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-27 17:42:32

[Beep6581]

This one should fix all the detected errors with BL. Please try with as many different
raw files as possible.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-27 19:32:35

---

• _Attachment: issue1627_00.patch_

[Beep6581]

Just for information:

I tested with:

Fuji X100 RAF
Pentax K-3 DNG
Canon Powershot SX160 IS DNG
Canon Powershot S120 CR2
Sony ILCE-7 DNG
Fuji X-A1 RAF
Canon Powershot SX150 IS DNG
Pentax K-30 DNG
Pentax K-5 DNG
Sony SLT-A99V ARW
Nikon D5300 DNG

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-27 22:43:33

[Beep6581]

There still seems to be an Issue with some Sony Cameras (DSLR-A700 for example), but
I think I found the reason (they've been treated specially in RT and don't need this
special treatment anymore with dcraw 9.20). Will post a new patch for this tomorrow.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-27 23:07:10

[Beep6581]

Reported by heckflosse@i-weyrich.de on 2014-01-27 23:07:44

• Blocking: #2032

[Beep6581]

Almost same as last one. Only change is, that it removes the overrides for Sony DSLR-A700
and Sony DSLR-A900 in dcraw_coeff_overrides. This also solves Issue 2206.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-01-28 11:12:49

---

• _Attachment: issue1627_01.patch_

[Beep6581]

THERE IS A NEW DCRAW VERSION RELEASED ..http://www.cybercom.net/~dcoffin/dcraw/dcraw.c
.. Revision: 1.460 Date: 2014/01/26 23:56:04 $

I tested with issue1627_00.patch

There is a problem with Canon Powershot s120.cr2 and DNG ..

Dcraw (and RT) use 96/17 pixels offset as left/top border in CR2s (Dcraw internal data)
while after converting to DNG its 120/18 (DNG exif data respected by Dcraw) and a farther
crop 

Inspecting an s120 sample http://216.18.212.226/PRODS/canon-s120/FULLRES/S120hSLI00080NR2D.CR2
there is a clear border around with lower exposure in CR2s while it's OK for the DNGs.
So we have to clip this border.
We can do it whith camconst.json ..
    {
        "make_model": "Canon PowerShot S120",
        "dcraw_matrix": [ 6961,-1685,-695,-4625,12945,1836,-1114,2152,5518 ],
        "raw_crop": [ 120, 29, 4024, 3030 ],
        "masked_areas": [ 32, 2, 3058, 80 ],
        "ranges": { "white": 4050 }
    },

but this works only for CR2s while "destroys" the dng because having a +1 top border
means the pixel order used is different. It worked fine (for both CR2 and DNG) without
Dcraw9.20.patch and 30 as top border .. the pixel order (red-green/green-blue) is the
default in this case (not supported models) and it happens the same order is defined
in DNG exif ..

I think this odd size used by Dcraw is strange at least for a bayer pattern (and it
was problematic for Amaze before the related patch ...)

In Dcraw These constants are at 
    static const ushort canon[][11] ....
   { 4192, 3062,  96, 17, 24,  0, 0, 16, 0, 0, 0x49 }, sensor tolal length, width,
left border, top border are obvious .. the rest not (for me) .. 16 could be the offset
of the black measuring area from the frame border (96-16 = 80). 

Reported by iliasgiarimis on 2014-01-28 14:01:44