RAW files and DNGs obtained from them give different results at identical settings.

[daicehawk]

RawTherapee 5.3 ec0f7936 (older 4.2.1148 shows the same behaviour) downloaded from http://rawtherapee.com/downloads. Win 7 professional x64, CPU intel pentium n 3700 1.6 GHz, 8 Gb RAM. ORF file from Olympus E-500 and DNG file converted in Adobe DNG Converter 9.9.0.718: https://drive.google.com/open?id=0B_gJzl0-HJ6RaFozbHFTRGNtaDQ https://drive.google.com/open?id=0B_gJzl0-HJ6RZ1RXdldaR0haSzg Even the Neutral profile already looks different for the ORF and DNG, then the Auto mode calculates different Exposure Compensation and Black\Shadow compression values. This difference also is observed with ARW Sony ILCA-68 and DNG files obtained from it. Cannot provide an ARW, but here the jpegs developed from ARW and DNG with the same settings and the same matrix-only DCP color profile: https://drive.google.com/open?id=0B_gJzl0-HJ6Rc2hvVDFiYjBCZEU https://drive.google.com/open?id=0B_gJzl0-HJ6RZUJ6MFZOQlRXSkU Considering DCP profiles are made in Adobe DNG profile Editor (and tweaked by eye) from DNGs which are supposed to work as good with native RAWs in RawTherapee I see this as a major bug.

[daicehawk]

Under the same conditions as above, profile parameters cannot be copy-pasted between two instances of RT 5.3.

[Benitoite]

Are there .pp3 files associated with the processing parameters?

[daicehawk]

Should I share a one source Auto obtained either from RAW or DNG and copy-pasted to the other format to compare or two PP3 after Auto mode enabled for each format? Basically, as I posted, even the Neutral profile (default As Shot WB, camera profile and zeroed in sliders, i.e. no PP3 needed AFAIU) gives different results in both RT view and in render.

[Benitoite]

@daicehawk Any generated .pp3's with matching associated input (RAW/DNG) and output (RT-generated jpg) would be good.

Could this be any relation? https://github.com/Beep6581/RawTherapee/issues/2809

[daicehawk]

@Benitoite Excuse me, we need to decide whether we just explore the difference between the Auto mode or just how different the outputs are from the start i.e. Neutral? Or should we take one format as a reference, then from the Neutral profile enable Auto as a good-enough chain to illustrate the difference, copy-and paste to the other format and present the Jpegs along with pp3s? It might be related to the basleline exposure, I am not sure. Several points on this: 1) Moving the Exposure slider higher-lower in any of the opened formats comparing to the other i.e. each one is taken as a reference for a time and the Exposure is shuffled around in the other one does not get the same result as the reference for both DNG and RAW references. There is a clear HUE difference, especially on sky gradient and skin tone with a matrix-only DCP profile. 2) The Auto-mode causes a difference not only in Exposure, but the Black\Shadow compression and probably other values. What I basically want is to be able to open a DNG in the Adobe DNG Profile Editor, tweak the matrix and get the same results in the RT as I would with the same DNG in the DNG Profile Editor as long as I do not have profile matrix tweaking options in RT.

[agriggio]

Even the Neutral profile already looks different for the ORF and DNG

They look exactly the same here, when applying the same pp3. I tried both with neutral and with auto-adjustment. (Compared with GIMP, not just by eye). Can you share your pp3? maybe there's something in there...

[daicehawk]

there are the PP3 files: https://drive.google.com/open?id=0B_gJzl0-HJ6RX0VaeWpUZ1ZpcHM https://drive.google.com/open?id=0B_gJzl0-HJ6RWWFNLU8xTGNFNDQ https://drive.google.com/open?id=0B_gJzl0-HJ6RUjZoMWlxWjRud28

[daicehawk]

BTW the path to the PP3 could be displayed in a more human way (as a slashed copiable path on click in the Path field, not just the stupid uncopiable level buttons). Consider this as a bug report.

[Benitoite]

@daicehawk I see you have ORFNeutral.pp3 and ORFAutoRTMatix.pp3 as well as DNGAuto.pp3 The [Color Management] InputProfiles are different between the two ORF pp3's. ORFNeutral is set to (cameraICC), whereas ORFAutoRTMatrix.pp3 is set to (camera). DNGAuto.pp3 has (camera) as well. Other differences. The ORFNeutral.pp3 has no Compensation, Contrast, etc. in [Exposure], whereas ORFAutoRTMatrix.pp3 has values.

[daicehawk]

It's obviously RT's fault since the Default camera profile radio button on the ICM tab stays on both with the Neutral profile and when you click the Auto button. No external DCP\ICM\ICC profile was used for those PP3. Please check the behaviour yourself with the RAWs \DNGs provided since you can do it before just stating the difference and throwing it to be on my side.

[agriggio]

Thanks for the pp3. DNGAuto has auto exposure on, whereas ORFAutoMatrix has it off. Despite that, here's what I did:

$ rawtherapee-cli -j -p ORFAutoRTMatrix.pp3 -o orf.jpg -c P1015545.ORF

$ rawtherapee-cli -j -p DNGAuto.pp3 -o dng.jpg -c P1015545.dng

Then I loaded orf.jpg and dng.jpg as layers in GIMP, set the layer mode to "Difference", and I still get an essentially uniformly black result. You can find the GIMP file here:

https://filebin.net/fmx5ryq7cwwuq1k1

Maybe I'm just misunderstanding what you are seeing, or maybe it's just not reproducible here. Can you also try the above and post the output? Thanks.

[daicehawk]

Guys, you seem to not see the forest beyond the trees. You should not compare AutoORF with NeutralORF since they of course will be different. You might compare ORFAuto with DNGAuto to see the difference. These two PP3 were obtained each from the Neutral profile by only clicking the Auto butoon on the Exposure tab. The ORF Neutral is just the reference for what it is when you open the ORF and choose the Neutral profile.

[agriggio]

Guys, you seem to not see the forest beyond the trees. You should not compare AutoORF with NeutralORF since they of course will be different. You might compare ORFAuto with DNGAuto to see the difference.

Please reread more carefully what I wrote, thanks!

[daicehawk]

I mean, no sense in comparing ORF to ORF. Just take DNG and ORF yourself, open it, tweak however whatever format (DNG or ORF, or even convert ORF to DNG yourself) you like, then copy-paste the settings to the other untweaked, and see the difference. That is my issue.

[daicehawk]

DNGAuto has auto exposure on, whereas ORFAutoMatrix has it off.

I clicked the Auto button for both, any way, you can just load the files yourself and reproduce it - select the Neutral profile for both, and then click the Auto button.

[agriggio]

... and see the difference.

The thing is, I see (essentially) no difference. I tried to document what I did and what I meant with that in my post above. This is as precise as I can go, and I did my best to make sure that my experiment above could be reproduced on your side. I'd be very grateful if you tried and posted the resulting xcf file, so that we can take a look. Otherwise, I just don't understand what you mean, sorry...

[Benitoite]

@daicehawk I compare the ORF with the DNG with auto levels and get a 0.07 stop difference in exposure compensation values. They are fairly close, but is that the issue you are seeing?

[daicehawk]

OK, maybe my exact conditions will bring us closer to the root of the problem I am experiencing. Provided:

1. an original ORF and DNG obtained from the ORF as said above (you can take the ORF and convert in the adobe DNG converter yourself).
2. a valid monitor profile installed and specified in the RT
3. Load the ORF, choose the Neutral profile.
4. load the DNG, choose the Neutral profile. Do they look identical in RT? Do the renders to whatever colorspace look the same (in RT, PS or LR)? We ll even skip the Auto button and external matrix profile for now.

[Benitoite]

@daicehawk For me they look very identical in RT.

[daicehawk]

@Benitoite sorry to ask, do you have your monitor calibrated\profiled? Will my render with the Neutral profile for DNG and ORF for checking in LS, PS or RT be enough?

[Benitoite]

@daicehawk I'm on MacOS using the system profile generated by the standard MacOS cal routine. I don't understand the second question: which render was that, what do you mean by "checking in", be enough for what?

[iliasg]

A small difference exists between e-500 ORF and DNG in WhiteLevel used. Dcraw.cc uses 4095 as WhiteLevel for the ORF version while for DNGs reads the exif tag and applies WL=3967. This explains the smal exposure difference we see. To normalize this we can insert the E-500 item in camconst.json

    { // Quality B, white level correction
        "make_model": "OLYMPUS E-500",
        "dcraw_matrix": [ 8136,-1968,-299,-5481,13742,1871,-2556,4205,6630 ],
        "ranges": { "white": 3967 } // DNG 
    },

But still a strong difference exist in the calculated AutoWB temp/tint values !! (ORF:7217/0.89 - DNG:6465/1.05) and at the calculated AsShot temp/tint. I wonder if the DNG version has changed something under the hood or is it RT in fault .. I'll dig in raw data and report later ..

[iliasg]

After digging .. no RAW values' difference between ORF and DNG .. After inserting the camconst.json item, no difference should exist for BlackLevel (both read exif tags and set it at 63) nor WhiteLevel which is set at the same value WL=3967 either from exif tag (DNG) or camconst.json (ORF)
So we need a guy who can read code and examine if ORF and DNG are handled the same way .. a good start is the pre_demosaic AutoWB calculation .. ;)

[agriggio]

@iliasg thanks for your analysis. Now I see the problem as well -- indeed the white balance gives very different results. I'll try to investigate to see if I can spot something obvious, although I'm probably not the best person for doing this...

[agriggio]

@iliasg using your camconst.json entry the differences disappear completely in the output. ORF and DNG are handled differently in dcraw, and it is true that you get a different combination of temperature/tint for the white balance, but the outputs seem to match perfectly. Regarding why you get those differences in WB, I suspect it's because the matrices computed for ORF and DNG are (slightly) different. Indeed, if you apply this patch:

diff --git a/rtengine/dcraw.cc b/rtengine/dcraw.cc
--- a/rtengine/dcraw.cc
+++ b/rtengine/dcraw.cc
@@ -9542,7 +9542,7 @@
   if ((use_camera_matrix & (use_camera_wb || dng_version))
    && cmatrix[0][0] > 0.125) {
     memcpy (rgb_cam, cmatrix, sizeof cmatrix);
-    raw_color = 0;
+//    raw_color = 0;
   }
   if(!strncmp(make, "Panasonic", 9) && !strncmp(model, "DMC-LX100",9))
    adobe_coeff (make, model);

then you get consistent results. But of course you do not want to use the above patch in production... I know too little about the dcraw code to actually figure out what to change (if anything). Maybe @heckflosse can take a look?

[agriggio]

Here's a slightly different (better?) patch to let RT use camconst.json also for DNG files. As before, I'm still not 100% sure about what I'm doing (to put it mildly :stuck_out_tongue: ) so better wait for @heckflosse to take a look...

diff --git a/rtengine/dcraw.cc b/rtengine/dcraw.cc
--- a/rtengine/dcraw.cc
+++ b/rtengine/dcraw.cc
@@ -9539,7 +9539,7 @@
     }
   }
 dng_skip:
-  if ((use_camera_matrix & (use_camera_wb || dng_version))
+  if ((use_camera_matrix & (use_camera_wb /* RT || dng_version */))
    && cmatrix[0][0] > 0.125) {
     memcpy (rgb_cam, cmatrix, sizeof cmatrix);
     raw_color = 0;

[iliasg]

@agriggio Alberto, the color matrices are exactly the same in DNG colormatrix2 (which is the one used by dcraw/RT) and camconst.json (I just copied it from there). Have you checked at AutoWB algorithm is the averages calculated are the same ?. For substituting the DNG exif tags with camconst.json I would prefer

if(!strncmp(make, "xxx", x) && !strncmp(model, "xxx",x))
    adobe_coeff (make, model);

[agriggio]

Alberto, the color matrices are exactly the same in DNG colormatrix2

Ok, I dug a bit deeper. With the "plain" RT, using your camconst.json, the difference between the two seems to be due to different pre_mul matrices (in rawimagesource.cc, computed by dcraw):

• For ORF, I get pre_mul = {1.83844733, 0.946108758, 1.11184323, 0}

• For DNG, I get pre_mul = {2.04544663, 0.946108758, 1.13130164, 0}

This is why the WB code computes different values. As to why we get different multipliers, I'll follow up once I have more info...

[agriggio]

Alberto, the color matrices are exactly the same in DNG colormatrix2 (which is the one used by dcraw/RT) and camconst.json (I just copied it from there).

Ok, I think I might have found the culprit. The DNG file has a non-trivial camera calibration matrix, here's what ExifTool says about it:

Color Matrix 2                  : 0.8136 -0.1968 -0.0299 -0.5481 1.3742 0.1871 -0.2556 0.4205 0.663

Camera Calibration 2            : 0.8988 0 0 0 1 0 0 0 0.9828

So, if I understand correctly, we should calibrate our matrix in camconst.json. Which would mean to use the following:

    { // Quality B, white level correction
        "make_model": "OLYMPUS E-500",
//        "dcraw_matrix": [ 8136,-1968,-299,-5481,13742,1871,-2556,4205,6630 ],
        "dcraw_matrix" : [7312, -1768, -268, -5481, 13742, 1871, -2512, 4132, 6515], // DNG ColorMatrix2 with CameraCalibration2 applied 
        "ranges": { "white": 3967 }
    }

With the above, now I get this:

• For DNG:

Raw As Shot White balance: temp 5142.414093, tint 1.154740
Raw Reference (auto) white balance: temp 6465.305328, tint 1.050736, multipliers [2.075093 1.974108 2.062431 | 2.045447 2.034069 1.995657]

• For ORF:

Raw As Shot White balance: temp 5144.199371, tint 1.154721
Raw Reference (auto) white balance: temp 6467.983246, tint 1.050441, multipliers [2.074108 1.974064 2.063015 | 2.045012 2.034458 1.995657]

So, essentially the same. What do you think @iliasg? Does this make any sense?

[daicehawk]

If you do the correction, please release a patch or a file that can be used in 4.2. The new 5.3 is very slow, I am quite satisfied with 4.2. One minute just to open two instances of 5.3 compared to 30 secs for 4.2.

[agriggio]

@daicehawk please understand that we do not have the resources to maintain old versions. In this specific case, you might try adding the above entry to your local camconst.json file and see whether that is enough (more info here: https://raw.githubusercontent.com/Beep6581/RawTherapee/dev/rtengine/camconst.json) But in any case, I'm afraid you are on your own if you want/need/prefer to use an older version of RT, sorry.

[Benitoite]

Regarding 4.2 and camconst.json: create a "camconst.json" file next to your personal "options" file. Its values will then override and/or complete the ones of this file. (from camconst.json) That should work for the 4.2 installation as well.

[daicehawk]

The same happens I suspect to many other cameras. At least with Sony ILCA-68 too.

[iliasg]

@agriggio

So, essentially the same. What do you think iliasg? Does this make any sense?

Nice catch !!. So for DNGs, RT makes use of the calibration_matrix2 for WB temp/tint calculations !! A nice side effect is that after matching the WB calculations for ORF/DNG all auto actions in RT (WB, Levels) also match !!

As I understand it, this matrix is in fact a WB multiplier calibration used by Adobe to better match it's calculation of WB temp/tint with manufacturer temp/tint data. I think that RT also benefits from this as the caculated temp/tint values look more reasonable (although tint is a bit high ..).

I think the way to go is not changing the "dcraw_matrix" because this also affects the color appearance for models we do not have dcp/icc profiles**. I think it's better to add a "wb_calibration" tag in camconst.json and fill it when needed

**Well .. we have to find some way to measure color fidelity of some models in this category (a substantial calibration matrix) using both matrices .. if @daicehawk can provide some shots of magbeth chart http://rawpedia.rawtherapee.com/How_to_create_DCP_color_profiles#Shooting_the_color_target with E-500 would be nice .. ;) If the "calibrated WB" gives better results then we can use this calibrated matrix as you did in "dcraw_matrix" : [7312, -1768, -268, -5481, 13742, 1871, -2512, 4132, 6515], , else we have to use the dcraw_matrix for color and use the calibrated values for WB calculations only.

Would be nice to have the input of experts on this @atorger @Desmis

[daicehawk]

https://drive.google.com/open?id=0B_gJzl0-HJ6RbGNJd0NFeVVWTUU https://drive.google.com/open?id=0B_gJzl0-HJ6RTHJMVEN2MGp5MVE It's not a CC24, it's SpyderChekr24, quite close color and spectrum, but different patch arrangement. If required, I might try to find the XYZ or LAB reference file, or measure anew. Don't have the flatfield, but the illumination on that one is quite even.

[agriggio]

I think the way to go is not changing the "dcraw_matrix" because this also affects the color appearance for models we do not have dcp/icc profiles**. I think it's better to add a "wb_calibration" tag in camconst.json and fill it when needed

If there's consensus about this, I might give it a go.

The same happens I suspect to many other cameras. At least with Sony ILCA-68 too.

I just checked, and it happens also with my Sony A6000. I never noticed before because I normally don't use DNGs. But indeed, I just tried converting one ARW file to DNG to check the WB behaviour. The picture was taken in daylight, on a partially cloudy day: RT gives a temp of 5062K with tint of 0.875 for the ARW, and of 5022K with tint 1 for the DNG. To me, the DNG value seems more reasonable... (The calibration matrix as reported by exiftool is 0.9368 0 0 0 1 0 0 0 0.9444)

[agriggio]

Hi again @iliasg,

So for DNGs, RT makes use of the calibration_matrix2 for WB temp/tint calculations !! A nice side effect is that after matching the WB calculations for ORF/DNG all auto actions in RT (WB, Levels) also match !!

As I understand it, this matrix is in fact a WB multiplier calibration used by Adobe to better match it's calculation of WB temp/tint with manufacturer temp/tint data. I think that RT also benefits from this as the caculated temp/tint values look more reasonable (although tint is a bit high ..).

I think the way to go is not changing the "dcraw_matrix" because this also affects the color appearance for models we do not have dcp/icc profiles**. I think it's better to add a "wb_calibration" tag in camconst.json and fill it when needed

I re-read the above, and double checked the code again. As I understand it, the calibration matrix is not used just for WB calculations, or at least not explicitly. It is in fact used to compute the actual cam_xyz matrix. So in fact this would actually be the same as modifying the dcraw_matrix in camconst.json...

[iliasg]

@agriggio .. strange !! Because a bit different matrices (calibrated vs non calibrated) should give a bit different color reproduction ..

As we can easily find colorchart shots from a6000 I will try some measures tomorrow.

[agriggio]

great, thanks! and remember that I know very little about camera colour reproduction and all that, so I might be totally off... I was just reading the code and trying to make sense of it :-)

[agriggio]

also, I was interpreting the description here: http://www.exiv2.org/tags.html as meaning that the two matrices should actually be combined (ie multiplied) to get the desired color space transformation. but again, I might be plain wrong

[iliasg]

@agriggio Thanks for investigating, and sharing info. This exiv2 link looks usefull.

My understanding about the use of calibration matrix is low level .. I just guessed so because there would be no reason for Dcraw to not use it for all models in native raw format.

BTW the a6000 samples at Dpreview https://www.dpreview.com/reviews/image-comparison?attr18=lowlight&attr13_0=sony_a6000&attr13_1=canon_eos5d&attr13_2=canon_eos5d&attr13_3=canon_eos5d&attr15_0=raw&attr15_1=raw&attr15_2=raw&attr15_3=raw&attr16_0=100&attr16_1=100&attr16_2=100&attr16_3=100&normalization=full&widget=1&x=0&y=0 converted to DNG report a different calibration matrix than yours (both on daylight and tungsten illumination) 0.9157 0 0 0 1 0 0 0 0.9533 vs 0.9368 0 0 0 1 0 0 0 0.9444 Now this is a reason to not have a fixed constant matrix .. maybe this is why D.Coffin didn't made use of it for native RAWs

Anyway .. I think there are reasons the DNGs have separate colormatrix and calibration tags and it is reasonable for RT to follow this tactic in camconst.json

[daicehawk]

Wait till @atorger chimes in. And I would not trust any daylight simulation shots-derived data. There are yet no light with the quality of the natural daylight.

[Desmis]

I do not know why Adobe changes the constructor settings when converting to DNG. Probably to standardize its treatments inside its software (Lightroom, Photoshop).

I had not noticed this difference until today, but I tried on 1 photo made with my SonyNEX6

• on the first version (arw), I get the following main settings : temp=5561 Tint(green) = 0.948 exposure compensation=1.48
• on the second version (dng), I get the following main settings : temp=5155 Tint(green) = 1.055 exposure compensation=1.46

If I compared the 2 images there are no significatives differences If in the 2 cases, I used "auto wb", the settings are differents, but the result is same image.

Nevertheless there are (at least) 2 problems that need to be solved or not: 1) portability of pp3 (partial paste, etc.) 2) CAT02 for CIECAM

If we want to solve, the only way is to modify DNG settings, by replacing the values with those of the file origin, but this supposes to have them, or else it is necessary to find the manipulation that realized Adobe and to do it in reverse.

That's what I think Alberto started :)

Jacques

[agriggio]

@iliasg:

BTW the a6000 samples at Dpreviewconverted to DNG report a different calibration matrix than yours

Thanks for the remark. Indeed I checked other pictures of mine and I get different calibration matrices... I suspect Adobe tries to match the WB in-camera settings (the a6000 lets you tweak the AWB algorithm in-camera). So now I agree that we should not "bake in" any of those.

@Desmis, thanks for your analysis! You say:

If we want to solve, the only way is to modify DNG settings, by replacing the values with those of the file origin, but this supposes to have them, or else it is necessary to find the manipulation that realized Adobe and to do it in reverse.

If we want to avoid the use of the CameraCalibration matrix that is embedded in the DNG, that should be relatively easy to do. Assuming the other parameters (white and black levels, dcraw_matrix) are correct, then we should get a consistent behaviour.

This is a patch that implements the above, if this is what we want (I trust your opinion -- both of you -- about this point...):

diff --git a/rtengine/dcraw.cc b/rtengine/dcraw.cc
--- a/rtengine/dcraw.cc
+++ b/rtengine/dcraw.cc
@@ -6170,11 +6170,15 @@
    break;
       case 50723:          /* CameraCalibration1 */
       case 50724:          /* CameraCalibration2 */
+        /* RT
    for (i=0; i < colors; i++)
      FORCC cc[tag-50723][i][c] = getreal(type);
+        */
    break;
       case 50727:          /* AnalogBalance */
+        /* RT
    FORCC ab[c] = getreal(type);
+   */
    break;
       case 50728:          /* AsShotNeutral */
    FORCC asn[c] = getreal(type);

[Desmis]

@agriggio

Hello Alberto I just tried your patch.

It solves the problem, now for my SonyNEX6 no differences between "ARW" and "DNG"

Very good job :)

Jacques

[atorger]

Lots of comments, haven't read them all, but a quick comment on the main issue that RAWs and DNGs lead up to different results. I have not been actively involved in RT development for a few years now meaning that getting up to speed means a few days of just reading code, so I'm not so much an "expert" in terms of RT code on a daily basis. That is I can't really comment on the specific issue that well. Anyway, back to the generic comment:

DNGs are usually made by Adobe DNG converter (unless natively from camera). This means that the DNG converter in some circumstances adds information that doesn't exist in the orignal, like white levels, and maybe color matrices. Adobe themselves sort of always convert to DNG "under the hood" in their products and the raw conversion will thus see the same input file, as Adobe DNG converter is the de-facto standard conversion tool. Their internal conversions under the hood of course uses the exact same code as the Adobe DNG converter has, thus yielding the exact same results.

For everyone else than Adobe it's different, as we don't have access to their proprietary code, and even if we had perhaps we want to make some things differently. Making it work the same way as in Adobe products would mean that we need to make a reference conversion in the Adobe DNG converter and copy its values into the raw code. Another way would be to strip away all extra DNG information and let the hard-coded values always override, but that too has some drawbacks. What is the value of a "standard" raw format it some parameters are ignored? Oh, in the (rare) cases of raw formats with calibration data on the side (common with medium format cameras), the Adobe DNG converter actually applies and removes that and thus changes the contents.

Personally I don't mind if results are a bit different, although each issue should be looked at separately (as said haven't had time to get into this specific problem here). As a user either you decide with a DNG workflow centered around the Adobe DNG converter, which indeed has many advantages, or you use raw directly and accept that unless you use Adobe products end-to-end those two formats are not 100% interchangeable.

However, and important note is that those differences should be negligible in 99% of the cases if there is a RT color profile available for the camera used, as the color profile is what makes the big difference in interpretation of the raw data. I'm not 100% sure how RT does it now as I've been away from the scene a while, but the "normal" way for a raw converter to do it is that if it has a bundled camera profile it uses that and overrides any profile in the DNG/raw, unless the user specifies otherwise through some setting.

However, if there is no bundled profile I don't know what happens now. I guess there's at very minimum a default matrix from dcraw which is used unless there is another in the DNG file, if so there can be a significant difference, but I think that is okay, I wouldn't want to make the "fallback matrices" to override DNG values, I think it's only valid if there's a full profile bundled. However if the current coders do want to use the fallback matrices for override too I wouldn't object.

[agriggio]

Dear @atorger, thanks a lot for your comment. I think it clarifies the situation, at least for me.

Based on my understanding, one important point is the following:

However, and important note is that those differences should be negligible in 99% of the cases if there is a RT color profile available for the camera used, as the color profile is what makes the big difference in interpretation of the raw data. I'm not 100% sure how RT does it now as I've been away from the scene a while, but the "normal" way for a raw converter to do it is that if it has a bundled camera profile it uses that and overrides any profile in the DNG/raw, unless the user specifies otherwise through some setting.

If I understand the code correctly (and that's a non-trivial if), the above is not a 100% accurate representation of how RT works now. In particular, it seems to me that the white balance code in RT doesn't use the specified color profile at all, but it always uses the dcraw matrix (either the built-in one, the one embedded in the DNG, or the one coming from camconst.json -- depending on the actual input format). This is also reflected in the following comment (file rtengine/dcp.cc, line 1341):

    /* Calculate what the RGB multipliers corresponds to as a white XY coordinate, based on the
       DCP ColorMatrix or ColorMatrices if dual-illuminant. This is the DNG reference code way to
       do it, which is a bit different from RT's own white balance model at the time of writing.
       When RT's white balance can make use of the DCP color matrices we could use that instead. */

Therefore, even if you use the same color profile, the white balance computed for the DNG converted file (temperature and tint) will be different from the one for the original raw file, and in particular copying/pasting the values between the two will result in significantly different looks. From what I understand, there seems to be a consensus that this is not what we want.

Since changing the WB code of RT is way beyond my skills, my proposal to "fix" this in the short term is to not use the calibration matrix for DNG files (or better, maybe only for those that come from the Adobe DNG converter), so that we get consistent behaviour. However, I'd like to hear the opinions of the experts here.

[atorger]

Ok. Yes it's true that white balance code in RT is its own, I think it was made before we integrated DNG/DCP at all. Again, it's something many raw converters have issues with, many are older than the DNG format itself and thus have their own white balance models. What those other converters seem to do is to always let their own WB model override the DNG way. I'm not really a fan of that Adobe did include white balance model in the DNG format, in effect it has made it feel more Adobe-specific and less general, but that's the way it is. Their model is actually not bad though so we could choose to simply adopt it, but that would probably be pretty disruptive. The current RT WB model has some issues with extreme temperatures if I remember correctly, in my most active days in the RT project I was actually a bit keen on replacing it with the DNG model which handles those situations better, but I never came around to actually debate it or do it. The DNG WB model is already used in RT when it comes to mixing dual-illuminant DCP matrices and lookup tables though.

Anyway, yes I agree that we should make sure white balance ends up the same regardless of DNG or not. I'm too rusty about the RT code so I can't comment on what the best method is though.

[agriggio]

Anyway, yes I agree that we should make sure white balance ends up the same regardless of DNG or not.

Ok, good! :-)

I'm too rusty about the RT code so I can't comment on what the best method is though.

I'm not sure what is the best way either, but here are two possible ways. The first is to simply discard the camera calibration matrix when reading DNGs coming from the Adobe DNG Converter:

diff --git a/rtengine/dcraw.cc b/rtengine/dcraw.cc
--- a/rtengine/dcraw.cc
+++ b/rtengine/dcraw.cc
@@ -6244,6 +6244,20 @@
     free (buf);
   }

+  /* RT -- do not use CameraCalibration matrices for DNGs coming from Adobe
+   * DNG Converter, to make sure we get consistent WB values between
+   * DNG-converted and original raw files. See #4129 */
+  if (!strncmp(software, "Adobe DNG Converter", 19)) {
+      for (j=0; j < 4; j++) {
+          ab[j] = 1;
+          for (i=0; i < 4; i++) {
+              cc[0][j][i] = i == j;
+              cc[1][j][i] = i == j;
+          }
+      }
+  }  
+  /* RT end */
+
   for (i=0; i < colors; i++)
     FORCC cc[cm_D65][i][c] *= ab[i];
   if (use_cm) {

This however assumes that the color matrix in the DNG matches precisely the color matrix that RT will use for the orginal raw file. I suppose that is typically the case, but it is not guaranteed by the above patch, and so we might still see some differences for some raw formats.

The second alternative takes care of this by simply overriding the DNG color matrix with the "non-DNG" one used by RT (i.e. either hardcoded in dcraw or coming from camconst.json). Again, this only happens when the DNG file comes from the Adobe DNG Converter:

diff --git a/rtengine/dcraw.cc b/rtengine/dcraw.cc
--- a/rtengine/dcraw.cc
+++ b/rtengine/dcraw.cc
@@ -6260,6 +6260,9 @@
   }
   if (!use_cm)
     FORCC pre_mul[c] /= cc[cm_D65][c][c];
+
+  RT_from_adobe_dng_converter = !strncmp(software, "Adobe DNG Converter", 19);
+
   return 0;
 }

@@ -9540,7 +9543,13 @@
   }
 dng_skip:
   if ((use_camera_matrix & (use_camera_wb || dng_version))
-   && cmatrix[0][0] > 0.125) {
+   && cmatrix[0][0] > 0.125
+        && !RT_from_adobe_dng_converter /* RT -- do not use the embedded
+                                         * matrices for DNGs coming from the
+                                         * Adobe DNG Converter, to ensure
+                                         * consistency of WB values between
+                                         * DNG-converted and original raw
+                                         * files. See #4129 */) {
     memcpy (rgb_cam, cmatrix, sizeof cmatrix);
     raw_color = 0;
   }
diff --git a/rtengine/dcraw.h b/rtengine/dcraw.h
--- a/rtengine/dcraw.h
+++ b/rtengine/dcraw.h
@@ -58,6 +58,7 @@
     ,RT_whitelevel_from_constant(0)
     ,RT_blacklevel_from_constant(0)
     ,RT_matrix_from_constant(0)
+    ,RT_from_adobe_dng_converter(false)
    ,getbithuff(this,ifp,zero_after_ff)
    ,ph1_bithuff(this,ifp,order)
    ,pana_bits(ifp,load_flags)
@@ -154,6 +155,7 @@
     int RT_whitelevel_from_constant;
     int RT_blacklevel_from_constant;
     int RT_matrix_from_constant;
+    bool RT_from_adobe_dng_converter;

     float cam_mul[4], pre_mul[4], cmatrix[3][4], rgb_cam[3][4];

Opinions?

[atorger]

Hmmmm.... CameraCalibration should be pretty exotic stuff, as far as I understand that's used not for making a profile for a camera model, but for a specific camera. As cameras don't really vary that much instance to instance for the same model I didn't think anyone was using those tags? And if you actually want a specific profile for your camera the way everyone(?) does it is to render a whole custom profile and load that, rather than trying to tune the existing generic one with a calibration matrix.

Anyway, those CameraCalibration tags should only be applied if the bundled profile has the same signature, citing the 1.4.0.0 spec:

"This matching is done by comparing the CameraCalibrationSignature tag and the ProfileCalibrationSignature tag for the selected camera profile. If they match, then use the camera calibration tags. If not, then use identity matrices."

As far as I understand in RTs case we should then never use Camera Calibration tags. I would exclude them permanently for all DNGs and add them in if someone really needs them and provides a use case and test files with it.