RAWs from Canon 7d have pinkish highlights

[maboleth]

I imported some of my old Canon 7D RAW files into DT and upon opening them in darkroom all of them had pink highlights that I simply couldn't turn off.

The first line is how the images should look like, the 2nd row is after opening them in DT.

Normally, I've seen this behaviour after messing with filmic rgb and/or highlight recovery. Or stretching out whitebalance. However, here are out of the box settings - white balance set to "as shot", no HL reconstruct, no filmic rgb.

In fact, HL recon. set to "reconstruct in LCh" does seem to ease this color off, but it's still not entirely.

Latest DT master git, arch linux.

[junkyardsparkle]

It's completely normal for blown highlights to be magenta when the highlight reconstruction is turned off, that's why it's on by default. :-)

After hitting the reset button on that module (which should set "clip highlights" mode with threshold of 1) is the color cast still there?

[maboleth]

After hitting the reset button on that module (which should set "clip highlights" mode with threshold of 1) is the color cast still there?

Yes, it's going from purple (hl recon. off) to pink (hl recon. on). The only thing that removes it more or less successfully is lowering the threshold to 0.65 or so. However, that's also clipping some light highlight details in the sky.

[junkyardsparkle]

Hmm... a couple of random thoughts: I believe there's an "enhanced" input profile for that camera? If it's selected in the input color profile module, does changing to "standard matrix" make any difference? If the base curve module is enabled, does changing the chroma preservation mode make any difference? Also, I notice the difference in the thumbs you posted seems to correlate to ISO setting... does this only happen with certain ISO values?

[maboleth]

Hmm... a couple of random thoughts: I believe there's an "enhanced" input profile for that camera? If it's selected in the input color profile module, does changing to "standard matrix" make any difference?

That's correct and that doesn't make any difference in this case.

If the base curve module is enabled, does changing the chroma preservation mode make any difference?

That actually did it! Though not chroma preservation but turning it to 'none' altogether. That did the trick. However, on really pinkish/purple images it still displayed slight pink hues. Like 1%.

Also, I notice the difference in the thumbs you posted seems to correlate to ISO setting... does this only happen with certain ISO values?

It is a bit. ISO 320 is more pronounced than ISO 250. I noticed a similarity though with different model, but still Canon. Interestingly, Canon 5d MKIV drastically changes RAW b/w values in DT from ISO 250 to ISO 320. It's only 1/3 step difference, but it is a night and day in module's values.

Overall, can this whole issue be somehow subdued or explained, why does it happen and what us, users can do when it happens (curves, hl. reconstruct etc.)?

On a more advanced level - can it automatically be corrected by DT in the future?

[junkyardsparkle]

Though not chroma preservation but turning it to 'none' altogether. That did the trick.

Well, normally the default "luminance" mode should be pretty safe even with blown highlights, so if you're still seeing some magenta cast even with "none" then something is off somewhere else... maybe the RAW b/w points need re-evaluating? I don't have much experience there, never really needed to worry about them for my cameras. :-)

[maboleth]

Well, normally the default "luminance" mode should be pretty safe even with blown highlights, so if you're still seeing some magenta cast even with "none" then something is off somewhere else... maybe the RAW b/w points need re-evaluating? I don't have much experience there, never really needed to worry about them for my cameras. :-)

What worries me in this case(s), I cannot have base curve disabled. Simply put, if I import my image with only highlight reconstruction and white balance enabled (excluding always-on modules like demosaic, raw bw point etc), the images come out with horrible pink highlights. So horrible that they look like this:

And its histogram:

This is the imported Raw file with reset raw bw module to default, white balance set to 'as shot' and highlight recovery to 'clip highlights'. Nothing else. Also I cannot have linear base curve - I need to have it curved (brightened the image) to push those pink hues out. So effectively, base curve isn't remedying this by itself, it's literally pushing the pink out of the highlight histogram values when the line is heavily curved.

Same is true to any exposure-altering module - simply making the exposure overexposed or bright, will make pink disappear by pushing it out.

[jenshannoschwalm]

You should really share the raw file and sidecar for anyone to analyse

[maboleth]

Alright, thought about it but wanted to be sure if there's something to be done on my end.

Here it is. You don't need a xmp file, just import it and see in the darkroom.

_MG_4771.zip

[RawConvert]

@maboleth I'm wondering if dt has the wrong white values for your camera. If you go to the "raw black/white point" module and reduce the white to around 13,000 (from default 16,300 on my system) the pink/magenta disappears. This white point setting depends on ISO usually - certainly with my eos 6D - and dt should set it appropriately as long as your camera is supported.

[jenshannoschwalm]

ok, the red channel is vastly out of bounds due to overexposure. As you said you can set the clipping threshold in highlight reconstruction to something like 0.6. The white balance shows a very strong red channel too so the "overblowing" is so heavy.

It could be also a problem of the rawspeed decoder getting wrong white points ...

[jenshannoschwalm]

maybe @LebedevRI knows

[wpferguson]

Here's a couple of xmp files on how to deal with it. If you switch on the raw over exposure you see the red and blue channels are both blow out.

4771.xmps.zip

[RawConvert]

@maboleth , I haven't looked at the XMPs above, however, to add to what I said previously, I remembered there is a parameter file used by Rawtherapee to derive the white point, camconst.json. I've changed the name to .txt in order to upload it here. This has values for both 7D models. For the ISO 250 you were using, the white point RT uses is 15,200, not the 16,300 my dt had. camconst.txt

[parafin]

It does look like incorrect white level. Probably these intermediate non-real ISO values are affected. @LebedevRI should take a look.

[LebedevRI]

Yep, that's pretty typical indication of wrong white level. As it can be seen, for that camera, there is no specific white level specified for ISO250: https://github.com/darktable-org/rawspeed/blob/develop/data/cameras.xml#L897

The simple solution is for someone who owns that camera to take one shot (even body cap on) per every ISO level that could possibly be selected on that camera, and i'll adjust the values.

The better solution is to fix RawSpeed to read those whitelevels from CR2 automatically.

In both cases, the existing developed images will regress, much like in the case with changing color matrix.

[parafin]

I think they won't regress, since rawprepare values should be part of the history stack nowadays. Maybe some old XMP files before that change will regress.

[wpferguson]

@LebedevRI I have a 7D and I've shot the back of my lens cap from ISO 100 to 12800. Do you want the raws, and if so where, or do you just want the EXIF information, and if so what?

[LebedevRI]

@LebedevRI I have a 7D and I've shot the back of my lens cap from ISO 100 to 12800

... including every sub-iso (both 1/3 and 1/2 increments, if any) and extended iso?

[wpferguson]

A 7D does 1 stop or 1/3 stop. I have 100, 125, 160, 200, 250, 320, 400, 500, 640, 800, 1000, 1250, 1600, 2000, 2500, 3200, 4000, 5000, 6400, 12800

[LebedevRI]

Aha. Well then, if you have adobe dng converter present, run it on all those files (copy them into clean dir first?), and then run https://github.com/darktable-org/darktable/blob/master/tools/dngmeta.rb on those dng's (it //should// work, but it's been a while since i used that last..) and either PR the result to https://github.com/darktable-org/rawspeed or paste it here.

[wpferguson]

I can also do a 10D and a 7D mk2 if you would like those.

[LebedevRI]

Sure, if you want to.

[wpferguson]

invsensors {[2048, 13600]=>[100, 125], [2048, 12600]=>[160, 320, 640, 1250, 2500], [2049, 15400]=>[200, 250, 400, 500, 800, 1000, 1600, 2000, 3200, 4000, 5000, 6400, 12800]}

sensors [[100, [2048, 13600]], [125, [2048, 13600]], [160, [2048, 12600]], [200, [2049, 15400]], [250, [2048, 15400]], [320, [2048, 12600]], [400, [2048, 15400]], [500, [2048, 15400]], [640, [2048, 12600]], [800, [2048, 15400]], [1000, [2048, 15400]], [1250, [2048, 12600]], [1600, [2048, 15400]], [2000, [2048, 15400]], [2500, [2047, 12600]], [3200, [2049, 15400]], [4000, [2047, 15400]], [5000, [2048, 15400]], [6400, [2048, 15400]], [12800, [2049, 15400]]] mostfrequent [[2049, 15400], [200, 250, 400, 500, 800, 1000, 1600, 2000, 3200, 4000, 5000, 6400, 12800]] invsensors {[2048, 13600]=>[100, 125], [2048, 12600]=>[160, 320, 640, 1250, 2500]}

<Camera make="Canon" model="Canon EOS 7D">
    <ID make="" model="">Canon EOS 7D</ID>
    <Sensor black="2049" white="15400"/>
    <Sensor black="2048" white="13600" iso_list="100 125"/>
    <Sensor black="2048" white="12600" iso_list="160 320 640 1250 2500"/>
</Camera>

[LebedevRI]

That seems mostly consistent with previous messages stating that ISO250 should result in whitelevel of ~15200..~15400. Please PR :)

[RawConvert]

Out of curiosity, are those black and white values in the CR2, or is DNG Converter adding them to the DNG based on info Adobe has assembled?

[LebedevRI]

They are already there, and as i have already said,

The better solution is to fix RawSpeed to read those whitelevels from CR2 automatically.