Closed Beep6581 closed 9 years ago
Reported by entertheyoni
on 2014-04-23 19:45:33
If it loads, then just commit? We can't get wide testing anyway until people start using
it widely.
Reported by torger@ludd.ltu.se
on 2014-04-23 20:53:51
I will update camconst.json with some more corrections and models supported in the next
10 hours.
Meanwhile I'd like to discuss things about
- How much conservative should the WL values be, as I found very significant deviations
when using in camera black frame subtraction at hi ISOs on Canon files. In this case
we have a (single or double) gaussian (bell) distribution at highlights and the bells
width gets larger by going up in ISO ..
40D ISO 1600 nominal 16224, but 15100 in BF_subtraction denoised shot
How should we deal with this ??.
a) Use WLs which are over-conservative for normal shots to cover some seldom abnormalities
?. We would need samples with long exposure and BF subtraction and this will increase
the time consumed and attention payed so it will make more difficult to find samples.
b) A new "baseISO long exposure WL offset" on which RT will auto-scale-up for calculating
higher ISOs would be nice .. and if RT could auto recognize these files (using exif
data)..it would be perfect.
i.e. Measure the offset on a shot at base ISO, then RT will scale the offset by ISO/baseISO
so for example an offset of 60 at ISO100 will scale to 16*60=960 for ISO1600
c) The easiest and dirtiest solution is to just document this in wiki and push the
users to use raw.linear correction for these files ..
- Possibility of a different name for the "official" camconst.json (in RT installation
dir) and the "user" camconst.json in data directory and report from RT about the exact
camconst.json used and maybe ask the user during installation about merging or updating
the related data ..
I think that as it is now there is danger of a conflict between the two. Think of a
"user camconst.json" with preliminary support of some models, then a new official camconst.json
comes with a new RT build including better support of these models .. as it is now
the updated official will have no effect on these models if the user forgets to update-change
his user camconst.json ..
- Use the exif data to calculate the BL for Panasonic files as described in #0
As it is now (fixed BL) we can find large errors up to 20 12bit levels which is too
much.
- HELP FOR FINDING SAMPLES .. ideas ?.
Reported by iliasgiarimis
on 2014-04-24 12:58:35
I would rather lose a small percentage of the ability to recover highlights than to
have them turn pink as soon as i apply a flat field or similar. I don't know in practice
how much of a difference to highlight recovery 15100 vs 16224 makes when not using
DF/FF.
I would keep camconst.json as that and make a new camconst-user.json, even if its empty.
The reason is that for example now I'm not sure where to put this user camconst.json
- I use -DBUILD_BUNDLE="ON" so I have no "data" directory and I must assume that the
file just goes into the same dir as the AboutThisBuild.txt file, or maybe into ~/.config/RawTherapee4/
? but if there was a blank camconst-user.json there already, all my doubts would be
gone :)
I can't get the D5300 raw file from http://rawtherapee.com/forum/viewtopic.php?p=36937#p36937
to display correctly even though I see it's in camconst.json. RawTherapee doesn't inform
the user that it found and is using camconst.json. Would be nice if it did! Preferably
in the UI (Raw tab), but even a printf in the terminal would be nice, something like:
Using /home/drslony/rt_default_Release_2327cl1_2301segfaultInPipette06/camconst.json
Using /home/drslony/.config/RawTherapee4/camconst-user.json
HELP FOR FINDING SAMPLES - we can make a new "Raw samples" forum section, but you would
need to explain in simple terms what photos are needed (how many, at what ISOs and
apertures, should there be clipping in them, etc. The camconst.json file explains this,
but I wouldn't expect many people to look in there), where to upload them, write that
by submitting them there is an implied agreement that the raw photos are made perpetually
available to the RawTherapee project (or maybe "to any open-source project", that would
be nicer) for improvement purposes and may be featured in screenshots and such, and
will be publicly accessible.
Reported by entertheyoni
on 2014-04-25 07:39:13
I'm silent because currently don't have much time to contribute with feedback, so please
don't wait for me if you do :-). Anyway, I prefer if we just get something deployed
first and then we can fine-tune perhaps after 4.1 release.
I'd like to cleanup the whitepoint raw tab stuff after 4.1 release etc, and then we
can do further camconst improvements.
Reported by torger@ludd.ltu.se
on 2014-04-25 11:00:23
I lost my time with detecting the correct black levels with some overburned shots of
Canon 40D yesterday so the updates are postponed for later today ..
The problem is that when there is strong light at the left side of the frame some light
leaks in the optically black area affecting up to three columns there so Dcraw/Rt detect
wrong BlackLevels. Removing these columns from the "masked areas" in camconst.json
the problem disappears ;).
This is another grey area which needs to be clarified in the documentation. I had the
impression that Dcraw/RT automatically excludes some columns from the borders of the
"masked areas" but in the case of Canon 40D it does not ..
Or excludes less columns than needed ? .. I cannot read the code so please help on
this ..
Rawtherapee does inform the user that it is using camconst.json in the console window
(if you set verbose=true in the options file).
But not details about which exactly camconst.json is used ..
For the problematic D5300 NEF .. it is in the optional 12bit format so the black levels
( 600 14bit) should be adapted to 150 12bit. The same problem exists for Dcraw 9.20..
Although the White level is detected correctly (4095) dcraw/RT does not adapt the BL
accordingly.
But here rises another problem with camconst.json and the models with double options
for raw bitdepth .. white levels will also be wrong ..
For the moment I will comment out D5300 from camconst.json as there is no need for
this anymore. I left it in because the white level after conversion to DNG was lower
(15892) but it looks like DNG converter and Dcraw do not decode the NEFs in the same
way .. so the auto detected 16383 white level is correct for Dcraw/RT
Reported by iliasgiarimis
on 2014-04-25 11:29:46
#5 Anders nice to read your message ;) .. indeed I was waiting for your comments ..
no harry .. job first :)
#4 Drslony, 15100 vs 16224 is 0.11 stops .. not very significant .. you can interchange
the values in camconst.json and see the visual difference .. to my eyes it is not negligible
but not significant also .. I have to test this with highlight reconstruction, maybe
things change differently there ..
Reported by iliasgiarimis
on 2014-04-25 11:59:27
Could you commit so it gets a few days testing before 4.1?
Reported by entertheyoni
on 2014-04-29 05:44:07
I have no commit rights ..
but I could upload the new camconst.json in the forums for anyone who wants to test
as he only needs to replace the old one with the updated ..
A "patched" win32 build is also possible ..
As I just received some samples I want some hours to "look" at them and then I can
update the patch.
Reported by iliasgiarimis
on 2014-04-29 16:28:38
Ilias, when you uploaded the updated patch, I, or another member with commit rights,
can commit it.
Reported by heckflosse@i-weyrich.de
on 2014-04-29 16:42:08
Ingo, I hope it will be "ready" tonight :)
Just started working on this ..
Reported by iliasgiarimis
on 2014-04-29 19:47:32
I have to apologize .. As I had to be away from home for some days I made very little
progress with camconst.json :( so some (many) things are midway ..
@DrSlony,
As 4.1 is almost here can you give a timeframe in which things should come in a stable
and "correct" stage ?.
I can find some hours this weekend to "finish" with some changes and new items ..
TODO
- Per ISO white levels for some recent Panasonic bodys as it is lower at the lower
ISO and maybe the first intermediate ISO .. I am searching for samples but I cannot
find all the samples I need (intermediate ISOs missing).
But this will be only half the corrections needed. The strongly different black levels
cannot be corrected in camconst.json as they change per shot.
The only way visible is to use the exif info ( see #0) and calculate from there. I
wish I could code this but I cannot so if a developer can spend some time on this we
could cooperate and RT will gain much better support for Panasonics ..
- Per ISO and per apperture WL for Canon models.
Here I have to crosscheck the behavior when Long Exposure Noise Reduction is applied
(in fact it is black frame subtraction) and the corresponding minus offset needed.
The samples I have from 40D look to be from an old destabilized body and are not indicative
of the general behavior of Canons.
I am waiting for samples from a friend but I don't know when I'll have them available
:(
Reported by iliasgiarimis
on 2014-05-03 10:24:07
Well I'm almost done with the splash screen so I'm almost ready to tag. Anders is away
for the weekend, so I was thinking of tagging tomorrow. 4.1 should have these camconst
changes, so when can you be ready?
Reported by entertheyoni
on 2014-05-03 11:08:40
I can do the win64 build. But we need to resolve the issue of rt going into infinite
loop before 4.1
Reported by michaelezra000
on 2014-05-03 11:14:41
#13,
Hmm .. ready ? .. after a month maybe :)
Now seriously .. I have limited time today, so I feel that I can post a good patch
tomorrow at midnight (GMT+2). I am progressing very slowly at the moment because I
still discover things and I have no proper samples at hand.
Then we have to update the model's support list and there we have to mention the state
of support. For some models we will have preliminary support as with
- Nikon D4s where the new Nikon format "sRAW" is not covered,
- Canon G1X II where only the default 3/2 aspect is supported and the user have to
comment-uncomment lines in camconst.json for the optional 4/3 or the full frame ..
etc
I will report again tonight about the state of progression.
Reported by iliasgiarimis
on 2014-05-03 14:54:32
re #14: Michael, do you mean Issue 2347 or this one from forum http://rawtherapee.com/forum/viewtopic.php?f=3&t=5402&start=15#p36977
?
Ingo
Reported by heckflosse@i-weyrich.de
on 2014-05-03 14:59:02
The model support list http://rawpedia.rawtherapee.com/Supported_Cameras needs to be
filled in, but this is not something 4.1 should wait for. It would be a big help if
all RT devs just update it as they go along - everyone has access to a different bunch
of photos.
Reported by entertheyoni
on 2014-05-03 15:07:28
Hi Ingo, it is the issue 2347
Reported by michaelezra000
on 2014-05-03 19:38:23
Downloaded many raw samples of Panasonic and Canon models.
The bad thing is it's difficult to find raw samples at intermediate ISOs, the good
thing is that for recent Canons even jpegs give us info about WL (exif.canon. White
level tags) and looks like usually WLevels are same for many models, so we can use
the exif tags and when we find a match with another model use the same measurement.
With a bit more conservatism at WL values we can be safe I think .. :)
Detailed samples of EOS-M with shots using Long exposure Noise Reduction will come
during next week ..
Reported by iliasgiarimis
on 2014-05-04 00:31:00
What's the ETA on committing the camconst.json patch? Don't want to wait for EOS-M,
just for the patch from the first post with corrections.
Reported by entertheyoni
on 2014-05-04 09:27:03
ETA is today at midnight (GMT+2) .. :)
Reported by iliasgiarimis
on 2014-05-04 16:42:59
Could you take a look here? http://rawtherapee.com/forum/viewtopic.php?f=1&t=5420&p=37052
Reported by entertheyoni
on 2014-05-04 21:46:06
Half an hour for a missing coma :(
Correction to Oly E-M10 raw crop
Corrections to Canon 40D
-white levels after receiving better samples
-Less columns in masked areas because light leaking results in wrong black level calculation
Canon 1200D/T5 support Not included in Dcraw9.20, not full set of samples but the missing
values are safely guessed.
Canon EOS-M .. not full set of samples but the missing values are safely guessed.
Panasonic GH3, GM1, GX7 per ISO levels
Commended out Nikon D5300
Changed Canon White levels to more conservative (-100 14bit levels from nominal to
cover non-linearities with Long Exposure NR)
Reported by iliasgiarimis
on 2014-05-04 21:58:24
Committed the patch above, with one fix for 5DII which I assumed was a typo:
{ "aperture": 1.4, "scale_factor": 1.100 }, // guessed by relative 5DIII data
{ "aperture": 1.4, "scale_factor": 1.077 },
changed to
{ "aperture": 1.2, "scale_factor": 1.100 }, // guessed by relative 5DIII data
{ "aperture": 1.4, "scale_factor": 1.077 },
Ie, I assumed the 1.1 scaling factor was for 1.2.
Reported by torger@ludd.ltu.se
on 2014-05-05 09:50:39
Fixed
Yes it was for f/1.2 , thanks Anders :)
Reported by iliasgiarimis
on 2014-05-05 11:10:46
Update 02
Added preliminary support for Nikon 1 V3
Added Pentax k-3,
Corrections to Canon EOS 1200D and addition of 600D in the same item (the two models
share the same WL, colormatrix and aperture scaling)
Some corrections and filling of missing values here and there.
Reported by iliasgiarimis
on 2014-05-15 02:41:34
I will commit this before tagging 4.1. If there are objections, please shout ASAP.
Reported by entertheyoni
on 2014-05-19 10:48:58
Will it be a good strategy to post the updated camconst.json files in the RT forums
?.
Prompting the users to put it in the user's cash directory ..
We could gather some more testing from users with no ability to compile this way ..
Reported by iliasgiarimis
on 2014-05-19 16:31:08
Reported by iliasgiarimis
on 2014-05-19 17:04:34
PatchSubmitted
camconst02.patch committed.
Yes, it certainly won't hurt :)
Reported by entertheyoni
on 2014-05-19 18:39:54
Reported by entertheyoni
on 2014-05-21 07:31:20
Fixed
Reported by iliasgiarimis
on 2014-06-04 07:47:28
WorkAround
Not been able to build a patch (away from my PC) I insert here the item for Nikon D610
to solve
http://rawtherapee.com/forum/viewtopic.php?f=3&t=5485
https://code.google.com/p/rawtherapee/issues/detail?id=2101
Request to anyone who can .. please build the patch and commit ..
{ // quality B, lacks WL measures at intermediate ISOs (160-250-320 ..) and measures
at long exposures with LongExposureNoiseReduction
// aperture scaling known to exist, but little to gain as the levels
are so close to white_max
"make_model": "Nikon D610",
"dcraw_matrix": [ 8178,-2245,-609,-4857,12394,2776,-1207,2086,7298 ], // dcp
d65
"raw_crop": [ 0, 0, 6034, 4028 ], // Dcraw has no raw crop for D610
"ranges": {
"white": [
{ "iso": [ 50, 100 ], "levels": [ 15800, 15700, 15800 ] }, // typical
G1/G2 15778, R/B 15879 lowered to 15700, 158800 for possible WL distribution under
LENR
{ "iso": [ 200, 400, 800 ], "levels": [ 16300, 15700, 16300 ] }, //
15878, 16383
{ "iso": 1600, "levels": [ 16300, 16100, 16300 ] }, // 16145-165, 16383
{ "iso": [ 3200, 6400, 12800, 25600 ], "levels": [ 16300, 16300, 16300
] } // 16383
],
"white_max": 16383
}
},
Reported by iliasgiarimis
on 2014-06-04 07:59:35
Ilias, I'll do that today :-)
Reported by heckflosse@i-weyrich.de
on 2014-06-04 08:47:45
Committed to revision c91e64a681d4
Reported by heckflosse@i-weyrich.de
on 2014-06-04 21:03:29
Updated camconst.json based on dng_v8.5 data
Corrections for color matrix for fuji finepix s1 and Panasonic GH4, based on DNG_V8.5
New entries for Nikon 1 J4, S2, corrected matrix for V3
New item for X-T1 & X-E2 to support the upcoming x-trans patches
New items for new Sony ILCA-77II (A77II), ILCE-7s (A7s), RX-100M3
Most new models are already supported with the upcoming dcraw_v9.22 patch
Reported by iliasgiarimis
on 2014-07-04 13:46:22
Add Samsung NX-3000 in the change list .. new model added
Reported by iliasgiarimis
on 2014-07-04 13:50:07
camconst_dng8.5_b.patch is included in xtrans_21.patch in Issue 2415
Reported by heckflosse@i-weyrich.de
on 2014-07-08 21:57:57
Ilias, can you have a look at the changes and remove the ones, which are not needed
anymore for dcraw 9.22 1.467?
Ingo
Reported by heckflosse@i-weyrich.de
on 2014-07-08 22:47:33
Since this patch will be bundled with the X-Trans patch which will be committed imminently,
I hope you don't mind if I close this issue now, to make a clean blocker list for 4.2
and to "wipe the slate" for future camconst.json updates.
As always, thank you for the work Ilias!
Reported by entertheyoni
on 2014-07-09 11:31:13
Fixed
Updates for camconst.json
Data from RT forums ..
Sony NEX-C3 : imported in camconst.json to revert the color matrix from Colin Walker's
D50 to DNG d65 because of problems with the former on underwater shots.
Nikon D800E : I lowered 70 levels the WL from Russell Cottrell's measures at RT forums,
Nikon D700 : with nominal WL instead of Russell's conservative WL due to non linearities
of his sensor
Canon 7D : with data from CharlyW at RT forums
Nikon D810 : WL and BL from measures provided by Donpark at RT forums. Color data from
dng 8.7rc1 instead of the wrong from Dcraw9.22
There is a problem with Nikon models which export multiple bitdepths (14 & 12), where
the Black Level is set to the 14bit value and needs correction in RT's raw black points.
If RT could detect the bitdepth from exif data it would be perfect. Dcraw has also
the same problem for some of them
Data from DNG 8.6
Panasonic LUMIX FZ1000
Data from DNG 8.7rc1
Fuji X30
Nikon D750
Sony ILCE 5100
Preliminary Support not included in current Dcraw & DNG
Canon 7D Mark II, color matrix copied from 70D
Canon G7 X, color matrix copied from G16
Panasonic DMC LX100, color data copied from GX7 .. measures fine with an average DE2000=2.4
Panasonic DMC-GM5, color data copied from GM1
Canon Powershot SX60 HS, color data copied from SX50
Some cleanups ..
Reported by iliasgiarimis
on 2014-10-23 18:02:20
PatchSubmitted
"There is a problem with Nikon models which export multiple bitdepths (14 & 12), where
the Black Level is set to the 14bit value and needs correction in RT's raw black points"
Until RawTherapee allows for such situations to be resolved behind the scenes, it would
be good if you could include information on the recommended raw black/white point slider
values in http://rawpedia.rawtherapee.com/Supported_Cameras
Reported by entertheyoni
on 2014-10-23 18:35:39
I cannot login in Rawpedia :(
Reported by iliasgiarimis
on 2014-10-23 19:25:26
Ilias, I committed your changes to revision bd4695a31fbb. Thanks a lot :-)
Reported by heckflosse@i-weyrich.de
on 2014-10-23 19:58:03
Hi, a little question on that :
I have a Pana GX7 and I pasted the dcp profile from the adobe stuff from the start
of the year. Is it better to remove it to let your settings? Is it completly the same?
Which one is applyied in priority?
Reported by cyrilaporte
on 2014-10-24 09:28:33
Hello Cyril,
The color matrix in camconst.json is a substitute of the internal matrix when this
is absent (for new not yet supported by Dcraw models) or a correcting profile in case
the internal is wrong/problematic (NEX-3c, D810 etc). In most items as it is now, it
is just a doublicate of the internal .. GX7 falls in this last case .. In fact it
is a copy of the D65 matrix that exists in adobe's standard dcps.
It can not substitute a dcp/icc profile as the last are generally better/more detailed
than a simple 3X3 matrix. In the case a user use a dcp/icc profile (which is recommended)
then this dcp/icc is used for color conversion and the internal or in camconst.json
matrix is only used for WB calculations.
Reported by iliasgiarimis
on 2014-10-24 14:59:34
I added the cameras from c#41 to RELEASE_NOTES.txt
Reported by entertheyoni
on 2014-10-24 18:56:26
Update according to
- new measures .. Many thanks to Charlyw for Canon 7DII and www.charlottesattler.de
for Nikon 700D
some more detailed per ISO WL will be included in a new camconst.json in next days
.. not a significant difference less than 2%
- new DNG_v8.7/v8.7.1 data
- Canon EOS 7D Mark II
- Canon PowerShot G7 X
- Canon PowerShot SX60 HS
- Fujifilm X30
- Fujifilm X100T/X100S
- Leica C (Typ 112), (same as Panasonic LF1)
- Leica D-Lux (Typ 109), (same as Panasonic LX100) .. not sure if it also needs adobe_coeff
- Leica V-Lux (Typ 114) (same as Panasonic FZ1000)
- Olympus PEN E-PL7
- Panasonic DMC-CM1 (sensor WL/BL copied from FZ1000)
- Panasonic DMC-GM5
- Panasonic DMC-LX100 Bug or change of policy in DNG conversion makes WhiteLevel tag
unusable
needs adobe_coeff.patch to always use camconst.json data instead of DNG exif data ..
- Samsung NX1 (only DNG support, waiting next Dcraw to decode SRWs).
While Dcraw/RT normaly supports DNG files, this entry is needed to consistently use
the color matrix of DNG_v8.7 even on DNGs converted by Samsung's utility (which uses
a bad color matrix i.e. DNG8.7 D65 ->DE2000 = 2.2 Samsung DNG D65 -> De2000 = 3.3
This should be paired with adobe_coeff.patch
- Sony ILCE-5100
- Sony ILCE-QX1 (same as ILCE-5000)
- Sony ILCE-A7M2
Companion to this camconst.json is adobe_coeff.patch for SammyNX1 & Panny LX100 to
always rely on camconst.json even for their DNG converted files.
Reported by iliasgiarimis
on 2014-12-12 22:06:49
Hi Ilias, thanks for the patches. For further dcraw.cc patches please use an editor
which uses LF instead of CRLF line endings. But it's also no problem for me to change
that :-) I did that for your current dcraw patch and committed the two patches along
with the dcraw.patch file to revision 8b2d6a92b59b
Ingo
Reported by heckflosse@i-weyrich.de
on 2014-12-12 23:39:44
New camconst.json update from iliasG from https://code.google.com/p/rawtherapee/issues/detail?id=2759#c16
Attached is the camconst.json file from there but I stripped some weird trailing whitespace
and [CR] (0x0D) characters, my guess is they got there by copying and pasting code
from a source which used different line endings, and a diff.
Reported by entertheyoni
on 2015-05-14 12:00:05
Originally reported on Google Code with ID 2348
Reported by
iliasgiarimis
on 2014-04-18 00:45:03