White levels, D3s

[mks9900]

I finally got around to measure the white levels of the D3s (thanks Ingo!). You can see the data that needs to be in camconst.json by following this link: http://pastebin.com/HyridN3w

Don't know how to do the scaling factor depending on aperture, but my test shots were taken at f/5.6. I have a series taken at f/2.8 as well, but didn't figure out the scaling... Also didn't know what quality A to C meant, so I left it with C :-)

BR, Johan Thor

[Beep6581]

Hey

Could you upload at least one of the raws so we can verify?

[iliasg]

Hi Mks9900, thanks for the data

It would be nice to have these raw samples (they could be small enough after compressing all them together as solid rar or 7z).

We have to investigate a bit the histograms in order to detect non linearities in more detail .. It would also be nice if we have overexposed samples with long exposures and LENR on because this is the worst case scenario.

[mks9900]

Hi! Gladly; I'm not sure on the process itself and certainly not what Ilias means by histogram investigation, so please: https://drive.google.com/folderview?id=0B7pUaGeFwBEoV2VqN3VEaF8yS0k&usp=sharing

If I should retake the series, please give feedback on how to improve them :-)

[mks9900]

Ilias, forgot to answer: LENR is set to off on my camera. Should I redo them with this on?

[iliasg]

Hi Johan, we can detect white levels with 95% confidense with the current samples .. if we will have the absolutelly proper samples the confidense will rise to 98% ;) so do as you like ..

The samples we need are very overexposed shots where all the frame should be clipped i.e shoot a flat homogeneous area at > +5EV. With the above settings it's perfect to have normal set and one more with LENR ON. But, as the worst case is usually the shots with LENR (because after black frame subtrraction all values increase or decrease accordingly so the clipping point follows the distribution (noise) of the black frame , this set is enaugh. i.e. a single clipping peak at say 15800 after a subtraction of a black frame with gaussian distribution between -100 to +100 will also have a gaussian distribution between 15700-15900 and this can give color cast at places or sporadically at pixels ..

The detection is easier and more precise by inspecting the raw histograms with rawdigger. Although we can come very close with RT and raw.white_point_correction. With RT the easiest detection is by using the "camera standard" color profile and increasing the black point of the standard curve so that we take increased contrast at the highlights https://drive.google.com/file/d/0B0NqktTgc54sajV3SG1aWkcxTUk/view?usp=sharing You can easily see the seam (all FF sensors have one or more seams) and non homogeneous white (seen as bands in your ISO100 sample). To counteract on this we have to increase raw correction at 1.06. We can also guess a bit more for the case of LENR in use, so our safe value is 16383/1.06=15455 and rounded down to 15400 for LENR safety .. (safety margin increases by increasing ISO because the noise is more so the distribution of the subtracted values is greater ..)
I can will post all this in RT forums with pictures .. if I find time ;)

[Beep6581]

ilias: in RawPedia please, not the forum.

[iliasg]

After inspecting the samples I came to the following camconst.json entry for D3s

{ // quality B, samples by Johan Thor at RT.Issues, measures at long exposures with LENR are missing 
  //            but a safety margin is included - aperture scaling makes no significant difference  
     "make_model": "Nikon D3S",
     "dcraw_matrix": [ 8828,-2406,-694,-4874,12603,2541,-660,1509,7587 ], // dcp d65
     "ranges": {
         "white": [
           { "iso": [ 100, 125, 160, 200, 250, 320, 400, 500, 640, 800, 1000, 1250, 1600, 2000, 2500 ], "levels": 15500 }, // typical G1,G2 15520-15800, R,B 16383
           { "iso": [ 3200, 4000, 5000, 6400, 8000, 10000, 12800 ], "levels": 15350 }, // typical G1/G2 15400-15800, R,B 16383
           { "iso": [ 16000, 20000, 25600, 51200, 102400 ], "levels": 16200 } // typical G1,G1 16270-16350, R,B 16383
                 ],
     "white_max": 16383
              }
},

[heckflosse]

Ilias, it seems that white levels can be different for R,G1,G2 and B Why don't you specify them different in camconst.json then? I'm just curious. For reference: https://github.com/Beep6581/RawTherapee/blob/master/rtengine/camconst.json#L94

Ingo

[mks9900]

Ilias, a confidence level 95 % seems like a good step forward to me :-) And after your inspection above I can only say "thank you"! I will make some shots at f/5.6, highly overexposed, with LENR enabled and upload them for inspection. Is that ok?

Thanks again! Johan

[mks9900]

Ok, I think I clicked a button I wasn't supposed to click... Sorry. Issue reopened.

[iliasg]

Ingo,

• It was a bit quick and dirty job, thats why I used a single per channel WL.
• Add to this http://rawtherapee.com/forum/viewtopic.php?f=1&t=5537&p=38136&hilit=d700#p38136 where Russell found that "G1 is always the lowest value; and as with the D800E, the pink highlights only disappear when all the color values are lowered to this value."
• I am not sure how RT deals with with variable WLs, is there a chance to drive WB calculations insane ?

As the G1/G2 difference is small (around 50 levels) and we are not sure that this diff is constant (still waiting for samples under different illumination and WB settings from your D700 ;) ) we could change to the following and do some tests .. I suppose we are mostly interested in how will "color propagation" respond to this .. { // quality B, samples by Johan Thor at RT.Issues, measures at long exposures with LENR are missing // but a safety margin is included - aperture scaling makes no significant difference
"make_model": "Nikon D3S", "dcraw_matrix": [ 8828,-2406,-694,-4874,12603,2541,-660,1509,7587 ], // dcp d65 "ranges": { "white": [ { "iso": [ 100, 125, 160, 200, 250, 320, 400, 500, 640, 800, 1000, 1250, 1600, 2000, 2500 ], "levels": [ 16300, 15500, 16300] }, // typical G1,G2 15520-15800, R,B 16383 { "iso": [ 3200, 4000, 5000, 6400, 8000, 10000, 12800 ], "levels": [ 16300, 15350, 16300] }, // typical G1/G2 15400-15800, R,B 16383 { "iso": [ 16000, 20000, 25600, 51200, 102400 ], "levels": [ 16300, 16200, 16300 ] } // typical G1,G1 16270-16350, R,B 16383 ], "white_max": 16383 } },

[iliasg]

Johan, yes this will be fine. And if you can upload 2 sets with different illumination and WB settings (daylight/tungsten) it will be even better :)

[mks9900]

Ilias, I will try! Do you want a whole set (ISO =100... 102400) with LENR=off and another set with LENR=on, both with WB={daylight, tungsten}?

[iliasg]

Johan, thanks I think that for the tungsten set three shots are fine at ISOs 200, 6400, 25600 , just to detect if a different scaling takes place.

[mks9900]

Ilias, here comes a series with wb=tungsten and (I think) overexposed by approx 5EV. Will do wb=daylight tomorrow. Do these qualify, you think?

LENR=off, wb=tungsten: https://drive.google.com/a/birkagatan.com/folderview?id=0B7pUaGeFwBEoTW1EMHktY2VzbFk&usp=sharing

LENR=on, wb=tungsten: https://drive.google.com/a/birkagatan.com/folderview?id=0B7pUaGeFwBEoQjJfakphMUZZSVU&usp=sharing

Perhaps a bit more than you asked for, but I was in a testing mood ;-)

[iliasg]

Johan, I took a view of the samples .. you have to use really long exposures (>1sec I think ..let's say >3sec is 100% safe ) for all ISOs or else LENR ON is not really active ..

[mks9900]

Ah... Ok, I think only the first in the respective series qualify then. Does it matter if the shots gets even more overexposed due to this?

[iliasg]

Johan, all your test shots are useful .. .. but for this kind of measures, the more overexposed the shots are, the easiest gets the detection of WLevel. We need ALL PIXELS clipped. In fact your shots are not overexposed a lot (the blue channel is not totally clipped ..)

2015-10-31 19:41 GMT+02:00 mks9900 notifications@github.com:

Ah... Ok, I think only the first in the respective series qualify then. Does it matter if the shots gets even more overexposed due to this?

— Reply to this email directly or view it on GitHub https://github.com/Beep6581/RawTherapee/issues/2947#issuecomment-152754991 .

[mks9900]

I sent Ilias a new set... :-)

[iliasg]

Hi Johan, is it possible for the full overexposed set (daylight LENR) to be available in the next days ?.

[mks9900]

Of course! Sorry for the delay. I hope you can download a sample set from here: https://drive.google.com/folderview?id=0B7pUaGeFwBEoQTV5TEdkLUQwSXc&usp=sharing

The settings are: WB=Tungsten LENR=on f=5.6 t >= 6 secs.

Thanks a lot Ilias! :-)

[iliasg]

Thanks Johan, LENR makes a significant difference in levels !!.

I changed the camconst entry accordingly but I am facing a problem .. RT (https://github.com/Beep6581/RawTherapee/commit/ba217b695a891e86cb0c33cbd7bf8c2cd33854c5 ) does not read the correct value for ISOs >= 12800 .. instead it uses the value from low ISOs (100-400).

Ingo help ..

find the camconst.json and the raw files at https://drive.google.com/folderview?id=0B0NqktTgc54sNGZxS2Q0dG8xR3M&usp=sharing

[heckflosse]

Ilias, I'll take a look at the ISO >= 12800 issue

[heckflosse]

Ilias, I will push a fix in a few hours.

[mks9900]

Hey, great work you two! Thanks a lot for this! :-)

[heckflosse]

I pushed a fix for the Nikon ISO > 12800 bug along with Ilias' changes to camconst.json and the dcraw.patch file to master

Ingo

[iliasg]

Ingo thanks, Although the camconst.json as posted is in beta stage I will provide an update in the next few days. For the moment and the current beta camconst.json, in order to make use of the camconst entries for Leica Q and SL DNG files (color matrix copied from adobe dcp instead of the one Leica provides in exif) we need to also update dcraw.cc & dcraw.patch

insert around line 9250 if(!strncmp(make, "Leica", 5) && !strncmp(model, "Q",1)) adobe_coeff (make, model); if(!strncmp(make, "Leica", 5) && !strncmp(model, "SL",2)) adobe_coeff (make, model);

[Beep6581]

@heckflosse good aftermorning. I presume the issue affects all cameras which can capture at ISO higher than 65535 due to the short integer used in the standard Exif ISO field. Please see the Sony A7S II for example, it has an ISO range of 50-409600. Shots at ISO51200 show the correct ISO in RT, but ISO80000 shows 65535. Raw file: http://www.imaging-resource.com/PRODS/sony-a7s-ii/YDSC00142.ARW.HTM From: http://www.imaging-resource.com/PRODS/sony-a7s-ii/sony-a7s-iiGALLERY.HTM

exiftool shows:

-EXIF:ISO=65535
-EXIF:RecommendedExposureIndex=80000
-MakerNotes:SonyISO=76852
-MakerNotes:BaseISO=100
-MakerNotes:StopsAboveBaseISO=9.6

exiv2 shows:

Exif.Photo.ISOSpeedRatings                   Short       1  65535
Exif.Photo.RecommendedExposureIndex          Long        1  80000

I don't know whether there is a standard place for storing these high ISOs, I'll ask the Exiv2 devs.

[iliasg]

@Beep6581 This is a different issue, as you can see the problems for Nikon D3s start from 25600

[mks9900]

Ingo, I think there's a bug which causes RT to crash: I've recompiled RT in debug mode and opened this file: https://drive.google.com/file/d/0B7pUaGeFwBEoZGFfYXlMeEFTTlk/view?usp=sharing

RT crashes and here's the backtrace: http://pastebin.com/d9zm1mcw

Using RT 4.2.468, master branch.

BR, Johan

[heckflosse]

Johan, that looks like a bug in wavelet code. Can you open a new issue please?

Ingo

[heckflosse]

@Beep6581 I pushed a fix for the Sony ISO bug. Also works for Canon. @iliasg I included the Leica stuff you mentioned https://github.com/Beep6581/RawTherapee/issues/2947#issuecomment-155257068

[heckflosse]

@Beep6581 @iliasg Pentax 645Z files are also infected by high ISO bug (ISO > 65535). I try to solve this tomorrow. Additionally I detected that for Pentax 645Z files focal length is also wrong. I try to solve this when the high ISO bug is solved.

Ingo

[heckflosse]

Status report: I solved 2/3 of the Pentax 645Z issues (the focal length issue and the high ISO issue for Info dialog and Metadata tab). 1/3 is still unsolved (the high ISO issue for dcraw/camconst), but I'm working on it.

Ingo