DCraw Phase One IIQ decoding broken for P40+ and later backs

[Beep6581]

Originally reported on Google Code with ID 2297

I've noted that dcraw does not apply the IIQ-format built-in sensor flatfield correction
in the correct way, so it breaks for newer digital backs like the IQ260. Probably the
same problem is there for the older P65+ too, haven't checked.

I think I know what the problems are. There are two:

1) the code assumes the X/Y size to be divisible with X/Y block size, which is not
always the case.

2) it does not extend to the edge; to fix a sensor stitch line the flatfield block
is only 2 pixels wide eg, multiply the left part of the sensor with A and the right
part of the sensor with B, but dcraw just corrects two pixel columns in that case.

I'll try to fix it...

Reported by torger@ludd.ltu.se on 2014-03-17 15:38:48

[Beep6581]

Uhhh... it was not that easy. 2) is probably wrong, the problem is more likely that
these backs have multiple amplifiers which amplify slightly different and their calibration
values are hidden somewhere in the file... I just hate reverse engineering, the utmost
time-wasting experience. But as the "MF guy" of raw therapee I really want this to
work, so I'll continue trying.

Reported by torger@ludd.ltu.se on 2014-03-18 07:32:48

[Beep6581]

bad news, seems to be complex to solve. There are larg(ish) chunks of additional undocumented
sensor calibration data that needs to be parsed.

Adobe DNG converter can do it and Iridient developer too, but no open source software
as far as I know.

Reported by torger@ludd.ltu.se on 2014-03-18 12:35:06

[Beep6581]

Update: even with all calibration data disabled Adobe DNG converter does a much better
job than DCRaw. The calibration data is only fine-tuning it seems, and it's probably
not visible if it it's applied or not in most cases. It seems to be quite hard to figure
out all calibration data tags (lots of tags with a bunch of numbers in them...), so
hopefully we can do without them to start with.

The larger error seems to be occurring in basic reading of the data. I have a lead
of what the main problem may be: seems to be a black-level strip along the side of
the sensor, 16 first pixels or so (not included in the main image). Those sensors with
only splitcolumn has it only on one side (a column), those with both splitcolumn and
splitrow (DCRaw currently only cares about splitcolumn by the way) have on both long
and short edge (ie both column and row). DCRaw ignores these strips.

Maybe/probably one should adjust the black level based on measurements on these strips,
and split them on the splitcolumn/row indications.

I'll try and see what happens.

Reported by torger@ludd.ltu.se on 2014-03-19 09:42:54

[Beep6581]

Reported by torger@ludd.ltu.se on 2014-03-19 15:33:36

[Beep6581]

I've made some progress. IIQ format is hell :-).

Phase One is not too secret about their format, they've let Adobe have it and I think
you can get it through their expensive SDK, but there's probably NDAs and stuff involved,
which is not compatible with open-source. So I think it's worth making a try to reverse
engineer it so this format is opened up. Had it been a commercial effort getting the
SDK would be smoother though, reverse engineering is really painful at times, when
you sit for hours and get nowhere, and you really don't know if the time you're spending
will end with success or failure.

They change calibration data format from one generation to the next, the P40+ and P65+
have a different set of calibration data from the IQ series. I think I'm close on the
P40+/P65+, and hopefully some of the things learnt can be used to decode the calibration
data for the IQ and IQ2 series.

The black-level strips seems like it was a dead-end, got back to figuring out calibration
data tags instead.

With the P40+/P65+ crucial amplification constants are kept as floats in 0x41e (multiplier-1.0,
so 0 means 1.0), and linearization data in 0x41f. Linearization data is duplicated
in lots of tags, I'm guessing it's measurements for different ISOs and/or sensor+ mode
or something. I'll concentrate on getting it to work for base ISO as it's what these
backs are used in most of the time. Applying base ISO linearization to other ISOs will
probably be good enough anyway.

Haven't figured out the linearization format yet, but I think I'm close. If I'm lucky
that's all needed, together with a previous DCRaw flatfield patch I've made.

Reported by torger@ludd.ltu.se on 2014-03-24 08:33:31

[Beep6581]

I just did the Phase One section of the Supported Cameras article today. Based on their
presence in camconst.json I added them as supported. Please have a look at the Phase
One cameras here:
http://rawpedia.rawtherapee.com/Supported_Cameras
If something needs changing, either register an account and I'll give you edit permissions,
or tell me what to do here. Editing is easy, so would be better and quicker if you
did it yourself.

Reported by entertheyoni on 2014-03-24 08:43:27

[Beep6581]

All IQ backs and P40+ and P65+ will need a finished patch to not risk show ugly tiling.
If a back actually does show tiling depends on amplifier behavior on that particular
back, some backs have widely varying amplifiers, others not so much.

Those backs also have a special sensor+ mode (half-resolution higher iso), and I don't
intend to support that, but I think we could say that we support them even without
sensor+ support.

I've made so much progress now that I'm quite confident I'll be able to make a good
patch sooner or later, not sure it will be ready for 4.1 though and since so few are
using these high end backs with open source software I should not block anything on
it.

What will probably happen is that 4.1 release version will have tiling issue, and then
at some point hopefully not too far away I commit a patch that fixes it. So one could
say that we support the backs, but it's a bit buggy for the moment ;-)

Reported by torger@ludd.ltu.se on 2014-03-24 14:16:57

[Beep6581]

I think I've figured it out now, for all backs. I need some additional verification,
but if I'm right I should be able to provide a patch real soon, and then we should
claim support for all those Phase One backs.

Reported by torger@ludd.ltu.se on 2014-03-24 17:08:51

[Beep6581]

Ok, then I won't touch RawPedia now as it seems that what I would enter today will be
invalid soon. Please remind me to update it when you commit the patch.

Reported by entertheyoni on 2014-03-24 17:47:18

[Beep6581]

Here's a patch.

It's safe as it only affects the formats that has those tags and were broken before,
ie P40+, P65+ and the IQ series.

I'm going to apply this real soon as I consider it to be safe as long as it compiles
on the various systems :-).

Reported by torger@ludd.ltu.se on 2014-03-25 08:25:17

---

• Attachment: phaseone-dcraw-patch.txt

[Beep6581]

Reported by torger@ludd.ltu.se on 2014-03-25 08:44:09

• Blocking: #2016

[Beep6581]

Reported by torger@ludd.ltu.se on 2014-03-25 08:44:32

• Status changed: PatchSubmitted

[Beep6581]

oops, a typo. New patch here.

Reported by torger@ludd.ltu.se on 2014-03-25 08:57:28

---

• Attachment: phaseone-dcraw-patch-1.txt

[Beep6581]

I have yet a typo in it, will post a patch after lunch.

Reported by torger@ludd.ltu.se on 2014-03-25 09:58:44

[Beep6581]

I'm having a typo day... but now I hope it's right.

Reported by torger@ludd.ltu.se on 2014-03-25 10:43:56

---

• Attachment: phaseone-dcraw-patch-2.txt

[Beep6581]

Can't provide test files unfortunately, all I have are files I can't share since, got
them for personal test use only. But I've myself tested on P40+, P65+, IQ260, IQ250,
IQ180, also sanity checked some of the old formats (that should be unaffected), P25+
and P45+.

It would be nice to test with IQ160 and IQ140 files too, but I have not come across
one. There's no reason to believe that they have different tags from the IQ180 though.

Reported by torger@ludd.ltu.se on 2014-03-25 11:00:24

[Beep6581]

I'm satisfied with my own testing, committed the patch.

Reported by torger@ludd.ltu.se on 2014-03-26 06:55:54

• Status changed: Fixed

[Beep6581]

Torger,

I tried RT on an IQ260 IIQ file today and your recent Mac OSX build [with the IR camera
support] worked great. [The file is the one currently being discussed on getdpi regarding
hot pixels, and made publically available by the photographer at https://docs.google.com/file/d/0B9QoNLxWMTzETjgzWXM3a1VuNkk/edit
] 

I have a question though. In your IIQ decoding, did you come across any dark frame
data? You mention finding and applying "the IIQ-format built-in sensor flatfield correction",
but not the dark frame. Maybe dark frames are only saved for exposures longer than
a certain threshold? Anyway you can try that half-minute IQ260 IIQ file, as it definitely
has dark noise/frame information of some sort that Capture One uses.

Ray

Reported by Ray.F.Butler on 2014-07-16 14:53:49

[Beep6581]

Thanks. I've downloaded the file, I shall look at it later. It can certainly be the
way you say. There is some black level stuff applied already, but not a full dark frame

Reported by torger@ludd.ltu.se on 2014-07-17 09:15:16

[Beep6581]

Yeah, like nearly all manufacturers in my experience (Nikon, Kodak, Fuji...but not Canon),
Phase One probably sample some light-shielded border pixels to get the mean black level
(bias/offset) for subtraction. 

Do you know if they apply this subtraction before writing the RAW file, or just store
the black level info in the RAW file for later subtraction? The latter scenario is
always preferable, so as not to clip off negative readout noise on very dark pixels,
but AFAIK Canon stand alone in doing it the "right" way.

Anyway, I'd be very interested in seeing what you find in terms of the dark frame in
the IIQ file. If it turns out to be an accessible frame packed within the same file,
maybe it could be automatically routed into the dark frame field of RT? 

Or I would not mind having to first go outside RT into dcraw, to extract the dark frame
to a separate file, that could be then loaded into RT.

Reported by Ray.F.Butler on 2014-07-17 10:17:44