Full support of baseline exposure

[Beep6581]

Originally reported on Google Code with ID 2829

DNG files made with Adobe DNG converter sometimes have a none-zero "Baseline Exposure",
for my Canon 5D mark II it's 0.4 stops for example.

What this tag means is that the base exposure should be adjusted +0.4, and then the
DNG profile can adjust this further with its baseline exposure offset tag.

With the recent DNG profile patch RT now supports the baseline exposure offset tag,
but not baseline exposure in the raw DNG file. This means that DNG profiles designed
for Adobe Camera Raw will render 0.4 stops too dark in RT for my Canon.

It's really easy implementing support for this and the highlights will be recoverable
with negative exposure settings, so I could be nice to have in.

However it will break rendering for those that did use Adobe's DNGs before and have
adjusted exposure accordingly, and it will cause an Adobe DNG conversion of a file
possibly render brighter/darker than if you open raw.

My suggestion is that we have this disabled per default, but in the "Raw White Points"
drop-down in the raw tab add a checkbox "Use DNG baseline exposure", so it can be enabled
for those that want the specified DNG behavior. The baseline exposure would be implemented
as an offset to the white-point correction slider, so it's a logical place for the
checkbox.

What do you think?

Reported by torger@ludd.ltu.se on 2015-07-07 09:10:33

[Beep6581]

It would be nice to have and a step forward for those who want an out-of-the-box decent
looking image, along with your recent DCP additions.

The raw tab is not the wrong place for it, but it seems not very practical. I think
it would be better, more clear to the user, if there was a "Use baseline exposure"
checkbox added to the Exposure tool which would:
- be grayed-out when the raw file lacks this tag,
- set the Exposure slider to this value when enabled and when not using Auto Levels,
so not only does it have an effect but the user sees the value,
- get disabled the moment you adjust exposure manually, the same way the Auto Levels
button sets a value and gets disabled the moment you touch the Exposure slider.

It could then be disabled in the (Neutral) profile but enabled by default otherwise.

The PP3 version may need to be bumped so that older versions are unaffected - i.e.
Baseline Exposure stays disabled when loading an older PP3.

Reported by entertheyoni on 2015-07-07 09:22:27

• Labels added: Type-Enhancement
• Labels removed: Type-Defect

[Beep6581]

Although I see some advantages of your suggestion, I still prefer mine. Not only because
it's easier to implement ;-).

The reason is that I think this is a setting that you'd either never use or always
use, and having it in the main view would just bloat it. It's a setting which is would
typically be used by users that like a DNG-workflow and are moving from Lightroom and
want to retain the profiles and base look from there.

When it comes to native RT profiles I don't think using baseline exposure is a good
idea, actually I don't like the tag at all, but as Adobe has put it there it's something
we'd want to optionally support to make RT compatible with various workflows. The problem
with the tag is that it's set by the DNG converter as a design decision by Adobe, it's
no information in the original raw file. That is if I open a .CR2 I won't get any baseline
exposure as I get when I open a .DNG converted by Adobe's DNG converter. It's then
much better to put in all exposure offset into the profile itself, which indeed is
supported (and now implemented in RT) as it will then work also for the original raw.
I have no idea why Adobe chose the approach of having two tags which just complicate
things.

So as this is not really a setting we want to put into use for the main recommended
RT workflow I think it's better to tuck it away in the raw tab to be used by those
that really wants it, or need to replicate Adobe Camera Raw's look for some reason.

(I want this feature myself for some Lightroom<->RT exchange stuff)

Reported by torger@ludd.ltu.se on 2015-07-07 10:17:30

[Beep6581]

For the record, I do convert all my PEFs to DNG, and in an unrelated thing I also am
tasked with replicating ACR's look of a CR2 in RT :)

Reported by entertheyoni on 2015-07-07 10:39:42

[Beep6581]

I'm making a patch now, with the simpler solution, so we can test and think about it
for a while how we want it. I hope to be able to post a patch later today.

Reported by torger@ludd.ltu.se on 2015-07-07 10:51:53

[Beep6581]

Here's a patch. It also includes the white level cleanup where the obsolete highlight
preservation slider is removed from the GUI, and an additional small "Adobe compatiblity"
patch, meaning that Adobe profiles (checking copyright string) which lack tone curve
get their default tone curve assigned, just as in Adobe Camera Raw / Lightroom. This
way you can finally use "Adobe Standard" profiles.

Now it's only one Adobe thingy left, the black render tag. It's less important though
compared to the other things, but maybe we should look into supporting that too.

Reported by torger@ludd.ltu.se on 2015-07-07 14:04:31

• Status changed: PatchSubmitted

---

• Attachment: wp-patch1.txt

[Beep6581]

Oops, forgot including baseline exposure for ciecam02, fixed in this patch

Reported by torger@ludd.ltu.se on 2015-07-07 14:15:46

---

• Attachment: wp-patch2.txt

[Beep6581]

Why not make it conditional based on the RT version, to allow backwards compatibility?
No UI, it would simply work.

Reported by michaelezra000 on 2015-07-07 15:33:53

[Beep6581]

#7, I'd be okay with that, but the drawback is then that there can be different rendering
result if you use a DNG and if you use a native raw. Native raws don't have baseline
exposure, the value is hard-coded in Adobe's DNG converter and added by it. I'm not
so sure we'd like to be dependent on Adobe's choice of baseline exposure.

The purpose of the baseline exposure tag is to make the default exposure match the
camera's auto exposure function, that is when you open the file for the first time
the brightness of the photo becomes about the same as the brightness of the in-camera
preview. Adobe further adjusts that exposure with their DNG profile curve (which has
a quite strong brightening component), and sometimes a baseline exposure offset tag
in the DNG profile. That is the baseline exposure is also closely related to how the
DNG profile is designed, meaning that Adobe's baseline exposure is matched for Adobe's
DNG profiles.

It may be a good idea to adopt the baseline exposure concept for RT as well (to be
able to get a better "camera-like" default rendering), but storing it in the raw file
is not the method I'd choose. Having it camconst.json would be better as it then would
be applied on both native raws and DNGs, and we would have control over the value so
we can design it to match our own profiles rather than Adobe's.

I consider using the DNG-embedded baseline exposure tag an Adobe compatibility feature
(in the case you're probably using a DNG profile from Adobe as well, or you use your
DNGs both in RT and in Adobe Camera Raw and want the same baseline result), not something
we should make a default behavior for RT. I think it's valuable to support exchange
with Adobe products, but I don't want to adopt any aspects as default if it's not in
line with RT's design.

Therefore I think having it "hidden away" in the raw tab like an advanced user setting
is the right trade-off.

What do you think?

Reported by torger@ludd.ltu.se on 2015-07-07 16:28:23

[Beep6581]

I understand the reasoning, one more checkbox in the raw tab would not hurt much:)

It would be nice to have DNG and raw open identically in RT. Otherwise, it would lead
to impression that something is broken. Having the baseline exposure/offset set in
camconst.json would be a nice addition to support that along with this new checkbox
to enable/disable the effect.

Reported by michaelezra000 on 2015-07-07 17:15:00

[Beep6581]

It's actually not one more, the same patch removes the obsolete highlight preservation
slider in the same panel, so it's +/-0 :-)

Reported by torger@ludd.ltu.se on 2015-07-07 17:36:24

[Beep6581]

Putting baseline exposure into camconst.json is probably a good idea, but I think it
can wait post 4.3.

The baseline exposure thing is intented to be used with profiles that provide a curve,
and in combination match the brightness/contrast of in-camera JPEGs. RT's native profiles
are colorimetric without curves, so they will render naturally dark. If we add baseline
exposure to camconst.json we must start thinking about default curves too. My own DCamProf
can prove valuable to RT concerning making profiles, but it's too in a turbulent development
state still... when things have settled a bit I think we can start think more about
those things.

Reported by torger@ludd.ltu.se on 2015-07-07 17:41:06

[Beep6581]

Hmm, I wrote a response but it seems I did not send it.

I don't mind having it in camconst.json or in the raw tab, though I am against having
it forcibly automated i.e. not being able to easily turn it on or off.

Please reconsider my suggestion in #1. Not just because it's mine and I'm biased about
my suggestions being generally good, but also because I believe it would be the most
clear solution to the user. I don't think it would bloat anything, and there is space
for one more checkbox in the Exposure tool without requiring more vertical space http://i.imgur.com/fZgLteQ.png
The exposure changed by this tag is no different in effect to the exposure changed
by the Exposure Compensation slider (correct?) so it would be good to use the same
slider.
It would have the added benefit that using Auto Levels would not conflict as Auto Levels
would hijack the Exposure Compensation slider.

Of course I'll change my opinion in an instant if there is a good argument to be made
against my suggestion, but please treat this addition as something actually useful
which should be clear to the user, and not as something which should be pinned on to
the rear end of the program where intentionally few will see it - then what's the point
of adding it at all.

Reported by entertheyoni on 2015-07-08 21:18:28

[Beep6581]

The idea of adding it is to make it possible to use Adobe Camera Raw DNG profiles and
get the same rendering as in Adobe Camera Raw (nothing more, nothing less), which depend
on baseline exposure being Adobe's value to work right. I think this is poor design
by Adobe (that the profile depends on an outside value to render the intended way),
but hey, it's not my fault. Good design would be to not have any baseline exposure
embedded in the DNG but only have it in the profile -- which indeed DNG profiles supports
now.

I don't think RT should redo the same mistake. That is if we want to use baseline exposure,
we should embed that in the profile only, which in RT's case is two-fold, camconst.json
for those that do not have a profile and overridden in the DNG profile for those that
have. I don't think we should depend on the Adobe-specific DNG value, that would mean
that we would need to adapt to Adobe's DNG converter releases all the time. That is
if we start supporting a camera really early with only it's native raw and Adobe later
release DNG support with a baseline exposure offset we would have to change our offsets
to match that. I don't like that kind of dependency.

I like supporting profiles the big commercial raw converters though, those that many
use. That's why I've put so much effort into both shaping up DCP handling to handle
all features used by Adobe profiles and Capture One ICC support. Adding suport for
Adobe's baseline tag is one such extra thing. I don't like third party support take
over and shape the design of RT's native workflow though.

For RT's native workflow I don't think baseline exposure should be on the front page,
as described above I think it is a profile value, like the "use tone curve" checkbox
we have in the color management box, but as the value is in the raw file and affect
white level the same way as "white-point correction" I think it's most logical to have
it there.

Then we would simply document in the color management section of the documentation
how to use third party profiles, I think we should structure it a bit better than currently,
with one headline per third party type, under Adobe Camera Raw / Lightroom profiles
we would document the thing with that you have to use DNG files and enable baseline
exposure to get exposure at the designed level.

So I still think my suggestion suits RT better.

However, say that we do decide that all our cameras should have a baseline exposure
value in camconst.json and all our DNG profiles should have one, that is RT starts
using baseline exposure in it's native design then I think it makes more sense to have
it on the front page, so that we in RT's way make the user aware that exposure is adjusted.
That value would then always be the RT native value though, not the DNG-embedded value
(otherwise we would have to match our values with Adobe's all the time), so we would
still want to have a separate switch to use the DNG-embedded value.

Does all this make any sense to you?

Reported by torger@ludd.ltu.se on 2015-07-09 09:31:09

[Beep6581]

To add a general comment -- DNG is often viewed upon as a manufacturer-agnostic standard
raw format just like JPEG is an bitmap format, and indeed it is that in many aspects,
but Adobe has still put in several Adobe-specific tags in it, they have tags for controlling
several Adobe Camera Raw functions, noise, sharpening, green equalization etc, without
documenting them properly in their specification, they're clearly just "slider values"
for their native tools. Their attempt to make it a standard format for all is thus
in some aspects half-hearted I think that has hurt the adoption of the format. Many
view it as an Adobe-specific proprietary format and indeed there are some strong arguments
that it is.

The baseline exposure is one of these tags that really shouldn't have been there. There
is a "baseline exposure offset" in the DNG profile itself, and it makes no sense whatsoever
to combine both these tags, but that is what Adobe does unfortunately. I think Adobe
have realized their mistake because the DNG profile value was introduced in DNG spec
1.4, while earlier there was only the DNG tag.

Reported by torger@ludd.ltu.se on 2015-07-09 09:42:14

[Beep6581]

One more alternative: if DNG profile == "copyright Adobe Systems" then apply baseline
exposure from DNG otherwise not. No visible setting.

(I'm about to add another Adobe specific, if no curve and Adobe profile then add Adobe
Camera Raw default curve like adobe does.)

Reported by torger@ludd.ltu.se on 2015-07-09 14:23:31

[Beep6581]

Re #13: It all makes sense and I agree with it, though I don't see how any of it conflicts
with my suggestion. If you feel strongly about putting this setting in the "Raw" tab
then go ahead with that, though I vote for keeping the button to enable the baseline
_exposure_ in the "_Exposure_" tab, regardless whether RT gets this baseline exposure
value from the DNG, from the DCP, from camconst.json or elsewhere, and I think it would
be both informative and easy to understand if the checkbox to enable baseline exposure
moved the Exposure slider to the appropriate place, e.g. from 0 to +0.3EV, from where
the user can move it anywhere else if (s)he wants to adjust exposure manually. If there
is a baseline exposure value available, regardless of its source, the checkbox is clickable;
if not, it's ->set_sensitive(false);

Re #15: As in #12, I don't think adding exposures and curves which have no visible
setting is a good idea or the RT way.

Reported by entertheyoni on 2015-07-09 17:00:26

[Beep6581]

#16 okay, I'll look into it. What makes me change my mind is that we having it visible
in the exposure slider is very logical in an "RT way", we have a well-defined zero-point,
ie no negative exposure no clipped data visible, so it's nice when a baseline exposure
is enable we actually see how much real range we push out.

I will include also the baseline exposure offset to this value.

I also checked and although not defined in the spec where or how baseline exposure
should be applied, the DNG reference code applies all baseline exposure after the first
huesatmap, so indeed, it makes most sense to apply the baseline exposure at the same
time as the exposure slider is applied rather than in white-point correction step.

Reported by torger@ludd.ltu.se on 2015-07-09 18:50:45

[Beep6581]

Reported by torger@ludd.ltu.se on 2015-07-09 18:55:20

• Status changed: Started

[Beep6581]

I really dislike the mess of splitting baseline exposure between DNG file and DNG profile,
so I'm thinking of doing like this:

- Always ignore baseline exposure tag in the DNG
- Add adobe_baseline_exposure tag to camconst.json, which is only used when an Adobe

  DCP is loaded.
- For all other DCPs baseline exposure is always 0, meaning that 100% of the 
  baseline exposure is decided by the DCP itself through its baseline exposure 
  offset tag.

The rationale is that if someone wants to use Adobe DCPs they need to enter DNG offset
value into camconst.json (if someone hasn't already done it) which then will make the
Adobe DCP work as intended for both DNGs and native raws. And for our own profiles
we don't need to mess with this totally unnecessary split between profile and raw file.

Otherwise it would work as said, with checkbox in the exposure panel, only sensitive
if the profile has a baseline exposure offset.

Reported by torger@ludd.ltu.se on 2015-07-09 22:00:10

[Beep6581]

This got hell to implement... checkboxes which affect other sliders and boxes and batch
mode on that, and depends on data here and there, it's GUI hell. I'm close to giving
up this feature...

Reported by torger@ludd.ltu.se on 2015-07-10 11:52:08

[Beep6581]

I'm giving up, costs too much to do for me that haven't got full insight into how the
GUI works. I'm not spending hours and hours on trial-error on GUI for this narrow feature.

Currently if someone wants proper baseline exposure with their Adobe profiles one can
check what it is in a DNG file and then make your own pp3 with exposure slider set
to that value. The baseline exposure offset on the DCP is already implemented.

If someone implements this feature we should remove the DCP baseline exposure checkbox
and just have the baseline exposure checkbox (or toggle button?) in the GUI, calculate
the total exposure offset and mirror that in the slider.

For now it's an Adobe-only feature (RT's profile don't use baseline exposure), so before
spending time on this we should probably intend to use baseline exposure ourself in
our own native RT profiles.

If we want to provide a good "camera-like" look on default opening a file, then use
of baseline exposure will be natural. So far that has not been the goal of RT which
is more into the auto-levels thing, we should probably know where we want to go before
we put any huge effort into this type of feature.

I suspect that for Hombre implementing this is easy though as he knows the GUI much
better than me.

Reported by torger@ludd.ltu.se on 2015-07-10 12:14:48

• Status changed: New

[Beep6581]

Anders, this is a great idea - why not simply incorporate this into auto-levels exclusively?
This is auto- functionality, so it will just enrich the auto-levels. 

Although, do auto-levels even need this tag? Current implementation is based on dynamic
image analysis, not image metadata or camera profiles.

Reported by michaelezra000 on 2015-07-10 13:34:20

[Beep6581]

Auto-levels doesn't need this tag as it makes dynamic image analysis. The purpose of
baseline exposure is to make a static adjustment which together with a profile with
curve makes the default output match in brightness and contrast the in-camera JPEG.
That is if your camera has a decent auto-exposure you get a good first render without
moving a single slider.

At some point I'd like to have such functionality in RT. You can achieve it already
today by making your own pp3 with an exposure offset, profile and curve (I have default
pp3 for my cameras for this reason) so it's no critical feature but I think it would
make RT much more user-friendly as every commercial raw converter out there work this
way, ie default render when you first open a file roughly matches the in-camera JPEGs.

Reported by torger@ludd.ltu.se on 2015-07-11 07:17:07

[Beep6581]

Anders, how do I get this baseline exposure value in the code so I can try implementing
it in the GUI when I have time?

Reported by entertheyoni on 2015-07-14 12:19:33

[Beep6581]

One value is in dcp.cc, the baseline exposure offset. It's applied when the looktable
is applied now, but it should be moved to an exposure slider offset as well.

The other value you get from the metadata.

baseline_exposure = root->getTag ("BaselineExposure")->toDouble ();

You need to patch rtexif though according to the posted patch early in the thread.

Reported by torger@ludd.ltu.se on 2015-07-15 15:26:35