White level from camconst.json is not used for hdr dng files

[Beep6581]

Originally reported on Google Code with ID 2781

Create an hdr dng from an image with overexposed sky (clouds).
Apply neutral profile to the original raw file and to the dng file in rt.
dng file shows pinkish clouds, while original file is ok.

Samples uploaded to http://filebin.net/ytq6krp0c2

Ingo

Reported by heckflosse@i-weyrich.de on 2015-05-15 13:27:12

[Beep6581]

It's the same with all DNGs, RT relies on DNG exif data and does not use camconst.json
nor Dcraw constants  (except if adobe_coeff is activated for a particular model ..
see Pentax K10/K20 etc).  

Reported by iliasgiarimis on 2015-05-15 16:10:31

[Beep6581]

Ilias, should we read white level for dng files from all cams from camconst.json?
Should be no problem to do that. At least for hdr dng files created from NEF files
the white level in exif data is wrong (or not present).

Ingo

Reported by heckflosse@i-weyrich.de on 2015-05-15 16:39:52

[Beep6581]

Hmm maybe I was wrong about the WL used .. I recall jcelaya taking special care (a patch)
for the float-DNG about this but cannot find it now .. 

Ingo, It gets complicated with non standard DNGs .. the levels are non standard any
more, going up or down depending on the mixing calculations and the only reliable info
could be the DNG exif but only if the DNG builder made also correct calculation-estimation
of white level .. :( 

Reported by iliasgiarimis on 2015-05-15 17:40:28

[Beep6581]

Ilias, it's less then 20 chars change to get white level from camconst.json for hdr
dng file only (normal dng files won't be affected by this change)

Ingo

Reported by heckflosse@i-weyrich.de on 2015-05-15 18:01:55

[Beep6581]

But how could we know the white level resulted after mixing two or more shots ?. Even
with one shot converted to float we don't. 

The float DNGs (16 or 24 or 32) have upper limit a lot higher than 65535. 

It's upon to the builder of the floatDNG to define it correctly and also to make correct
calculations for it .. for a start it needs to know the correct WL of the prototype
shots. And I think this is where it failed in your sample, i.e. used the dcraw default
instead of adapting to variable WL with a database like camconst.json 

Reported by iliasgiarimis on 2015-05-15 18:50:39

[Beep6581]

Ilias, if Issue 2396 #0 is still valid, the upper limit for a hdr dng file should be
the same as the upper limit of the original shots. Maybe I miss something?

Ingo

Reported by heckflosse@i-weyrich.de on 2015-05-15 19:42:53

[Beep6581]

Issue 2431 is related too

Reported by heckflosse@i-weyrich.de on 2015-05-15 19:53:49

[Beep6581]

I feel that it's me who misses something :)

I don't understand the reason to use 16bit (or more) float to represent 14stops of
dynamic range !!. As I understand it RT expects linear representation and 16383 is
at most 14 stops. 
.. float16 can go up to 32 stops .. multiple Dslr shots can cover easily more than
16 stops ..

Any way, maybe Javier's HDR-DNGs are as described but some other may use the full range

Let's ask Anders Torger about http://www.lumariver.com/Products/LumariverHDR/, or Alex
(Magic Lantern) about his implementation .. 

Reported by iliasgiarimis on 2015-05-15 20:32:35

[Beep6581]

#7 I agree with issue2431 exif should be 1st preference.

But I had the impression that camconst.json is second preference after exif but before
Dcraw internals.
Ingo can you declare ?. Are some of camconst.json data (WL, BL) at 1st priority ?.

Reported by iliasgiarimis on 2015-05-15 20:41:24

[Beep6581]

Ilias, for non dng raw files, camconst.json is always first preference. It wouldn't
be possible to override exif or dcraw internals with camconst otherwise. Second preference
is rt internals (dcraw_coeff_overrides in rawimage.cc). Third preference is dcraw internals
(adobe_coeff in dcraw.cc). Last preference is exif.

For dng files it's different. Have to look first.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-05-15 20:59:47

[Beep6581]

re #8, Ilias:

Take your example of 14 bit raw files (14 stops, max value 16383): in integer space
the first stop will only have two possible values (0 or 1), second stop has 2 possible
values (2 or 3), third stop has 4 possible values (4,5,6,7) and so on. Using floating
point it's different (depending on float representation). Using floating point at the
same bit depth as in integer space, the granularity for all stops will be almost equal

Ingo

Reported by heckflosse@i-weyrich.de on 2015-05-15 21:16:13

[Beep6581]

Ilias, to be more precise: dng 16 bit float has a mantissa granularity of 10 bits

Reported by heckflosse@i-weyrich.de on 2015-05-15 21:40:09

[Beep6581]

I understand the granularity thing. What I don't understand is the white level of floatDNG
at a fixed value same as with the 14bit linear raw.

Let say we want to fully capture a scene with 16 stops dynamic range. We have a camera
which gives raws with 14bits and useful* dynamic range 11 stops. *By useful we mean
at acceptable level of noise (say SNR>4) for the dark parts. 
We can do it (capture 16 stops DR) by using multiple shots with 5 stops exposure difference
(darkest to brightest).
Then we have our -5EV shot (say shot 1) which at say 16383 14bit level, represents
the top level of the DR we want to capture and the rest shots are exposed more so they
clip the highlights in order to have cleaner shadows.

To represent the DR of 16 stops we combine the multiple shots so that the level 16383
of shot 1 is 16stops (65536X) higher than the LSB. We can do this by either multiplying
the underexposed shots (or dividing the overexposed) by the proper factor, then combine.

If we use float16DNG .. at which level in [0,1] range will this 14bit 16383 value need
to be,  so that it linearly represents the 16stops distance from the LSB and have good
granularity at the darks ??. At which integer value will it be represented to be compared
with the camconst.json "white" ?. 

Reported by iliasgiarimis on 2015-05-16 08:52:40

[Beep6581]

Maybe this will help
https://github.com/jcelaya/hdrmerge/blob/master/Image.cpp
or just ask Javier on github.

Reported by entertheyoni on 2015-05-16 17:48:56

[Beep6581]

Maybe the question should be: How to handle dng files with wrong white levels? D700
NEF files converted to dng using Adobe DNG Converter have a white level of 15892 in
metadata, but the one in camconst.json (15750) is a bit more on the safe side.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-05-16 18:55:52

[Beep6581]

Ingo, is the DNG_exif_WL stable at 15892 for all ISOs ?. 

About D700:
I set the WL lower to cover the non linearities due to LENR (it is in fact a black
frame subtraction) and also the non-uniformities of the sensor at the seam and at the
corners. I had not reliable data (Russell's D700 was strogly non uniform and different
from other D700s) so I went in the safe side. I could differentiate by ISO because
the non uniformity and the clipping_point_spread with LENR_ON increase with ISO (the
stronger the noise of the black frame the stronger the spread gets) so it could be
15900 at ISO200 and decrease to 15700 at ISO6400 .. 

About handling possibly wrong metadata:
- Ask the authors of HDR-DNGs to use the available info (our camconst.json data maybe).

Anders is the author of the camconst.json utility I would be surprised if he didn't
make any use of it. Javier is also cooperative with RT .. 
Alex @ Magic Lantern knows Canon Whitelevels' deviations better than all of us and
I think he uses correct values.

the problem is we cannot be sure if a WL value is wrong or not after all the blending
manipulations during the building of HDR-DNGs 

- raw autodetection can exist if there is a hard clipping at the raw green channels..

- rgb detection .. check highlights to find if we have pixels where Red and Blue components
are clipped but Greens not .. hmm may be a fast pre-demosaic would needed for this
just 2X2 tiles averaging (Dcraw -h) or maybe larger 3X3 tiles with 2 pixel pitch (more
robust color) or 4X4 tiles ..  

Reported by iliasgiarimis on 2015-05-16 21:39:17

[Beep6581]

Ilias, I just made ISO 200,400,800,1600,3200,6400,12800 and 25600 shots with the D700
and converted them to DNG using Adobe DNG Converter. White level reported by RT is
15892 for all these DNG files.

Ingo

Reported by heckflosse@i-weyrich.de on 2015-05-16 22:15:43

[Beep6581]

@heckflosse @iliasg status?

[heckflosse]

@Beep6581 As a workaround I added possibility to specify custom white level in hdrmerge some time ago. That fixed the problem.

[iliasg]

Hmm .. what if .. at the raw tab, we had a GUI puldown list to alter the WL priorities between DNG_exif and camconst.json ?

[Beep6581]

Would it not be best if the user ran exiv2/exiftool or any of the available GUIs to fix the white levels in their files?

Which files other than HDR DNG does this issue address?

[iliasg]

@Beep6581 your proposal is not best for the normal users (little to zero knowledge about white level, exiftool syntax)

I think all third party DNGs can possibly be affected (HDRmerge, magic Lantern, custom made) even Adobe's at early support stages (I remember a case with Panasonic LX100 @heckflosse ;) )

Alternative option is to use a paragraph in RTpedia for exactly this issue (problems from non correct white level data in DNG exif) and suggest playing with raw.white point correction or exitool corrections taking data from camconst.json (but what if there are no data in camconst.json ??).

[heckflosse]

@maybe an easy solution for the case the dng is produced by hdrmerge would be to use camconst for floating point dng files...

[iliasg]

@heckflosse Alex @ Magic Lantern also exports float_DNG with a program like HDRmerge .. CeroNoise https://www.magiclantern.fm/forum/index.php?topic=9581.0 We could have a conflict if he uses a different approach for WL reporting in his float DNGs which is correct for his file and different from camconst.json ..

[heckflosse]

@iliasg Ok, then this is not an option.

[iliasg]

@heckflosse yes .. not an option .. maybe .. but ..

• CeroNoise is stalled and has very limited user base .. although it or any equivalent (Lumariver ?) can come up again :(

• If there is a signature of HDRMerge in the exif then use this info and act accordingly ;)

[Beep6581]

your proposal is not best for the normal users (little to zero knowledge about white level, exiftool syntax)

@iliasg I agree with that. These same users with little to zero knowledge would come asking us for help, and we'd have to analyze their files and find the correct white levels to put into camconst.json so that the suggested combobox could be used. Or am I wrong? If I'm not wrong, then we are faced with two options - both require effort, but one offers a permanent fix (edit bad Exif) while the other is temporary (work-around in RT). Maybe coding a combobox and adding support for handling broken DNG files from multiple sources would be more work than writing a RawPedia article explaining how to find correct white levels with a link to a simple to use GUI Exif editor from which they could fix it, with CLI examples as well. Just a thought.

[iliasg]

@Beep6581 The best fix (quality wise) I see is a patched HDRmerge (or equivalent) to make it use camconst.json data .. this will give reliable results from the start. The current state of HDRmerge after Ingo's HDRMerge patch (ability to use custom WL) is also at the right path .. Correcting merged data is not the same as doing the merge correctly. Think of the case where the merged files are not at same ISO .. For other mergedDNG exporters .. we can ask the same .. may be we will find helping hands and brains this way ..

For RT I think a combobox or even a check box (use camconst data as reference for DNGs) to alter priority order will give freedom to find the best settings and as you say (I like your thought) push the users to supply samples .. ;)

I am many times in the case of needing a "priority" tag in camconst (instead of these ungly adobe_coeff workarounds in Dcraw.cc) .. not only for floatDNGs .. I have met Adobe DNGs with problems (LX100), Pentax in_camera DNGs with problematic WL (it is reported after BL subtraction ..) .. Leica DNGs with non standard color matrix (d50) .. many smartphones with strange color data and structure .. maybe others I haven't checked yet .. And there are and some other features missing from camconst (like multiple frame sizes ..) .. We have to review and update the camconst structure in the near future I think ..

[Beep6581]

@iliasg @heckflosse could you please update the title of this issue to better match the desired outcome, whatever that is? Seems like it's not a bug but an enhancement for a white level priority combobox to Raw > Raw White Levels?