RFC: Auto-enable lens module

[jenshannoschwalm]

Freshly imported images don't have lens correction applied To automatically enable lens correction i have to define presets for the cameras / lenses i am using. That leaves out images of combinations i don't regularly use.

Describe the solution you'd like I would suggest we add the lens module (if available) to the modules automatically applied.

1. Applied to all raws and true monochromes. (We can't safely do for other files as they might already have correction applied)
2. This should be done for all workflows. (scene, display, none)

Alternatives Keep as is

[parafin]

What is the problem with creating a general auto-applied preset, for all cameras/lenses?

[jenshannoschwalm]

At least i was not able to create such a preset :-( (Any hint appreciated ...)

[parafin]

Start with some fresh image, enable lens iop, do not touch any controls, create an auto-applied preset out of it (do not limit it to some camera and lens). This way worked before.

[jenshannoschwalm]

Got it :-) Thanks for hinting -- leaving me with open eyes and these questions:

1. can we expect a "normal" user to know about this?
2. Wouldn't it be better to offer this as default?

[jenshannoschwalm]

There is also one issue: doesn't work well with the new embedded data mode ...

[victoryforce]

Got it :-) Thanks for hinting -- leaving me with open eyes and these questions:

My opinion on this matter:

1. can we expect a "normal" user to know about this?

No. A novice user who has not yet mastered the program cannot know this.

This is something that a user who is determined to master the program can, of course, find in the settings and eventually use. But simple things should be done simply.

2. Wouldn't it be better to offer this as default?

Definitely yes.

[parafin]

Documenting this definitely wouldn’t hurt. As for “on by default” my vote would be against, I don’t use this module like at all, and I’m not sure we can guarantee quality of the lens DB, so I would rather user enabled this feature by himself.

[jenshannoschwalm]

Ok, i accept your point about lensfun quality validation although we are proceeding using existing lens correction data extraction. One more point would be the issue of not all lenses reported properly.

Can we yet proceed further? Proposals from my side:

1. Just document the "magic way" at a good place in the manual (maybe some reference in the module)
2. Add a preset in the module that "just simply works" in addition to 1)
3. Add another option in preferences "autoapply lens correction" (although that would again add another option which i personally dislike a lot) leaving out the question to be on or off by default.
4. Auto-apply in all cases and place the burden of disabling lens correction via a preset on user's shoulders

[parafin]

1. This the minimum. Not really discoverable.
2. Not sure how it would help, since it’s the same as just enabling the module. It’s a big leap for a user to guess that you are suggesting to create user auto-applied preset out of it, and would lead to 2 presets in the list, which is excessive IMHO.
3. Adding a preference that just duplicates existing functionality is kinda stupid. Better fix discoverability of auto-applied presets, it would be more general solution.
4. There will be cases with bad output, which user will rightfully blame on darktable. E.g. some cameras has optional lens correction (even with grades for strength of correction) for raw files, which makes it impossible to avoid double-correcting. IMHO it’s better not to correct, than over-correct (since this correction destroys part of data).

My suggestion is to try to fix discoverability of auto-applied preset feature.

[jenshannoschwalm]

3. Adding a preference that just duplicates existing functionality is kinda stupid

I don't see the argument here. We have lots of "features" that can be modified via preferences or by hand-tuned presets. This would be about "an easy to use one click" thing.

My suggestion is to try to fix discoverability of auto-applied preset feature.

Any idea on this? How and where to detect? Not obfuscicating or overwhelming?

[TurboGit]

I agree with @jenshannoschwalm discoverability is nice, but remember that dt initial design was no menu. We have to bear with that and we certainly don't want adding more stuff on the main display for people refusing to read the documentation. Not reading the documentation means that many features are missed by the users (and we have complains / questions about this).

[TurboGit]

Just thinking out loud maybe we need to add a way to reset an auto-applied preset to ensure that it read it's context and set itself with proper values. That is, as if it the module was activated by the user by clicking on the button.

[parafin]

Existing preferences are more about minimal required set of modules to get a useful output or just plain legacy. Lens correction for most cases doesn’t fall into those categories (e.g I hardly ever used it), it’s much closer to “let’s add a setting for everything”. My argument is simple - just don’t add any more preferences, we have enough already for a couple of applications.

As for discoverability - I myself don’t see an easy fix - auto-applied presets are quite flawed, e.g. see #13414 which is a mess to understand, and this auto-detection of module settings is especially so, I’m pretty sure there are a couple of old open bug reports here about this already.

[SoupyGit]

I think we should ask,

1. Do the majority of users always apply lens correction?
2. Do lens corrections give an improved result most of the time?

If answer to both is yes, then I think enabling by default is a good idea. For me personally, answer to both is yes, but I am just one.

There is a third question to ask,

3. For those who do not use, would there be an easy way for them to disable auto lens corrections? Currently, there is an easy, although undiscoverable, way for we who like lens corrections to apply it automatically (via preset). Doing this means we don't have to turn it on every image. If instead lens corrections were auto applied, would the same ease to turn it off for all pictures exist for those who dont want it? If answer to this question is yes (and that is the third yes) then this feature is good. If answer to this question is no, then probably have to leave things as they are, or people will be annoyed having to always turn lens corrections off.

Just my 2c.

[elstoc]

Do the majority of users always apply lens correction

I would guess the majority of users don't frequent github

[ptilopteri]

as far as that goes, anyone should be able to figure out how to make a preset that automagically applies a module. the lens correction module and making a preset work pretty much the same through out all of dt. the new user with little technical skills is the one who will find difficulty, and making a preset to always apply lens correction will just one among very many

I would vote to leave it unset.

[SoupyGit]

I would guess the majority of users don't frequent github

Well that much is obvious. But why not those who are here answer the questions? Discuss their mode of working? Or why not browse through the playraws on pixls to determine how often lens correction is applied? With not much effort, there are ways to get an indication on these things, rather than just take the first answer that comes. That is no offense to parafin, it is good he replied with his preference. Everyone should. This not about me getting my way, or he his. Personally, it doesn't matter to me if it's not applied by default since there is a workaround with the preset. I'm just trying to come up with a reasonable objective method to determine what the default should be. Let's not take it personally.

as far as that goes, anyone should be able to figure out how to make a preset that automagically applies a module.

Should, yes. But I've been using dt for years now and didn't know. I guess I'm not alone. Why? Probably because when we save presets in other modules we are typically saving set parameters. Eg. A preset in colorbalance with the chroma slider set to 25% will always be 25% in every image. But Lens Correction is different, in that a correction for Nikon camera will not always be a correction for Nikon camera, rather, a correction for whatever camera is found in exif. I didn't realise I could save a preset for my Nikon, and have that preset work on other cameras, even if it seems obvious now.

But it doesn't matter whether anyone should be able to figure it out or not. It's about the best default. The best default might indeed be something people should be able to figure out. Eg. By default certain workflows apply a tonal curve. Should anyone be able to figure out a tonal curve is required? Yes. Does that stop us from turning it on by default? No. Why? Because in vast majority of cases, it is good to have. Same with exposure adjustment, and demosaicing, and orientation.

[parafin]

Another potential problem with lens correction being on by default:

1. user has outdated lens DB (very common situation)
2. he imports an image for which there is no lens correction in his DB
3. develops a picture, e.g. crops it however he likes
4. later he updates the lens DB, now lens correction for that picture is available
5. after that the edit of first picture is changed, isn’t it?

For me this totally breaks the guarantee of image edit history not breaking. Note that if user enables the lens correction manually he will most likely disable it after he sees that no correction is available and the problem is avoided. For auto-applied preset not so much, but at least if user consciously created it we could consider he knew the risks (and should document them).

Up until recently lens correction module even was optional in build (it was me who argued to change that), now suddenly it should be on by default?:) I think it will create more problems than it will solve. I suggest time is better spent by improving it, e.g. better communicating that by default settings are set to auto-detect, not particularly Nikon, Canon, whatever.

[pehar1]

... been using dt for years now and didn't know. I guess I'm not alone.

Definitely you are not alone, I use dt since version 2.4 and have tried several times to create such an "auto-detect" preset for lens correction. Without success... I use the module in more than 95% of the cases, but I can understand the argumentation against a general automatic activation of the module.

better communicating that by default settings are set to auto-detect, not particularly Nikon, Canon, whatever.

Absolutely !

[SoupyGit]

Yes if implemented it would have to be in a way to not change old edits, by applying only to new imports or something. If that can't be done, I agree it's a deal breaker.

[jenshannoschwalm]

Agreed. Let's see if we get further with presets first, maybe come back then.

[jenshannoschwalm]

Closing due to #13432

[garrett]

Hi! Any chance to revisit this decision? darktable now has support for correcting using embedded lens data, with a fallback to lensfun... so the lens information should be higher quality now.

I, probably like most other photographers, always want lens correction (and noise removal — but that's a different discussion) on. JPEGs from cameras and all other raw editing programs that I'm aware of do this by default.

I did have a preset for auto-enabling lens corrections, but it no longer works in darktable dev builds (as it applies twice), so there's no way for me to do this anymore. Right now, it's a choice between no lens correction, having it applied twice, or manually enabling it for each and every photo... none of which are ideal, especially since darktable should probably be doing this without extra user effort.

[jenshannoschwalm]

I also have a preset for lens activation for all images. Please note the new presets thing for 4.4 making this a lot easier.

[parafin]

An existence of bug in auto-applied preset implementation is a reason to fix said bug, and not for anything else...

[garrett]

A user expectation to have lens corrections automatically applied is a reason to fix said bug.

[TurboGit]

Hi! Any chance to revisit this decision?

Fact is that if we activate automatically the lens correction module, people not wanting it won't be able to disable the module. Well it can be done, but you'll end up with an history stack with two entries, one for the auto-applied and one for disabling the auto-applied module (and again this is kind of advanced in dt, new comer won't be able to do that easily).

Keeping as it is today means that one can create an auto-preset to enable the lens correction module as he likes.

My personal case, I won't like the auto-enable lens for all my pictures, today I enable it only for some lens only.

So at the end I see no good/simple solution for everyone.

[TurboGit]

I did have a preset for auto-enabling lens corrections, but it no longer works in darktable dev builds (as it applies twice)

Can you share the RAW + xmp please? Maybe there is already an opened issue?

[garrett]

Fact is that if we activate automatically the lens correction module, people not wanting it won't be able to disable the module.

All camera manufacturers, by default when processing JPEG in-camera, use lens corrections. They actually design their own lenses to need in-software corrections, and have for many years. And virtually all other raw processing software applies these corrections by default too.

As much as I am usually against adding an option to settings in general, there are cases where it makes sense. This may be one of those, if people feel strongly that they do not want their lenses automatically corrected. There should probably be an option that's on by default in darktable's settings to automatically correct lenses; this would let someone can then untick it if they want. But enabling it by default would then give people what is actually expected (even from a manufacturer's point of view) from a camera + lens combo.

---

The raw + xmp is @ on Pixls.us, which I linked from another issue @ https://github.com/darktable-org/darktable/issues/14393#issuecomment-1533418402

1. download https://discuss.pixls.us/uploads/short-url/6SNZVfz4ahddbBWva55wFYEzPHM.RAF
2. use this XMP: https://discuss.pixls.us/uploads/short-url/4x2mSONurioOrVpglqCwjyJGbCW.xmp (make sure it's called the same as the filename — both should be named properly when downloaded, but double-check)

The preset is just "enable auto". I don't have access to it right now, but could provide it when I get time tonight or this weekend.

I'm also planning on working on a fresh temporary config with just the preset imported (and not all my other configuration from over the years which might be triggering a bug) to see if I can reproduce it in an isolated environment.

[wpferguson]

darktable, by default and design, only applies the minimum number of modules on import to display an image in the user's selected workflow (scene referred, display referred, none). This has led to countless discussion threads "Why is my image so bland|washed out|dull|etc".

Commercial raw developers try and provide a nearly finished image by making choices for the user and applying them. The darktable philosophy is that the user should choose the look, and thus the processing, not the software. darktable has styles and presets that allow the user to apply their choice of modules to the image to get closer to their desired starting point, if they so desire.

[garrett]

@wpferguson: Yes, but I'm not talking about that. I know; I've been doing photography for decades, use FOSS since 1996, and have been using darktable for a few years already.

This is not about artistic changes. This is about something that's literally part of the lenses as designed by their manufacturers. The people who make lenses intend for their hardware to be "fixed" in software. This is way more prominent in some lenses versus others.

darktable has long had "sharpen" as a setting, which is an artistic change. Just like white balance (correcting the colors), lens corrections are not an artistic change, but a basic and intended (and sometimes fully needed for some lenses) step in processing a photo.

[parafin]

It may be true for some lenses, but definitely not for all. I've never seen lens correction done by my camera for any of my lenses, but lensfun have correction available for them and if enabled by default it would look different from out-of-camera JPEG. So I strongly disagree that it should be enabled by default. And there is already a way to enable it for users who want it - auto-applied preset.

Clearly there are conflicting workflows, and in general it's more logical (or at least more darktable-way) to have less processing steps by default, not more. User has to make some effort to study the application he's using and adapt its settings to his liking.

[jenshannoschwalm]

Well, we (devs) have agreed on not enabling by default. Also it's now very easy to enable by preset without any understanding of dt internals.

So good reasons to keep this issue closed.

I've never seen lens correction done by my camera for any of my lenses, but lensfun have correction available for them and if enabled by default it would look different from out-of-camera JPEG

This is something i would never have expected and is the exact reason why lens correction is not done by default. My personal experience is completely opoosite,, never had a camera that does not correct for jpegs :-) (Oly, Leica)

[garrett]

All of my cameras always correct for lenses when they render JPEGs:

• Ricoh GR IIIx
• Fujifilm X-T2
• Fujifilm X-Pro1
• Panasonic GF1
• Canon 5DmkII (it's called "peripheral illumination correction"; newer bodies also do additional lens corrections)

Same for phone cameras:

• Google Pixel 6 Pro (Google app does some processing before the DNG, but OpenCamera and others don't; some adjustments are still needed no matter what)
• OnePlus 6T (it has native DNG support)
• Google Nexus 6P (same as the Pixel; Google app does extra processing before DNG, but other apps do not)
• Same for older phones too; the apps that can save raw files (basically all DNGs) desperately need the lens corrections.

The one that doesn't (afaik; although it might):

• Canon D60 (From 2002, not the 60D; it's so old it doesn't have it as a setting. Still, it might've corrected it without a setting in JPEG files, but I mainly shot raw.)

Notes:

• Micro 4:3 (such as the Panasonic GF1 mentioned above) actually have correction as a requirement in the M43 spec. This blog post talks about it, with some example images: https://m43photo.blogspot.com/2011/03/geometric-distortion-correction.html
• Phone lenses (and sensors) are often atrocious and are adjusted even more with computational photography. But even doing lens corrections helps quite a bit (when it's not baked into the file ahead of time).

[parafin]

All of my cameras always correct for lenses when they render JPEGs: [....]

• Canon 5DmkII (it's called "peripheral illumination correction"; newer bodies also do additional lens corrections)

At least this one is optional (can be disabled in camera menu), so "always" is not factually correct. Furthermore I'm mostly concerned with doing geometric corrections by default, not shading/flat-field. And lensfun profiles AFAIK always include geometric corrections, while shading is optional.

Notes:

• Micro 4:3 (such as the Panasonic GF1 mentioned above) actually have correction as a requirement in the M43 spec. This blog post talks about it, with some example images: https://m43photo.blogspot.com/2011/03/geometric-distortion-correction.html
• Phone lenses (and sensors) are often atrocious and are adjusted even more with computational photography. But even doing lens corrections helps quite a bit (when it's not baked into the file ahead of time).

I may agree that if there's a specification that lens correction has to be applied (like OpCodes in DNG spec), that it would make sense to apply it if not mandatory, but at least by default. But that's definitely not the case with ordinary raw files and lensfun profiles. And we have no way to differentiate these cases right now, so it's up to the user to create such configuration using auto-applied presets.

[HSUfineprint]

Seems you have consensus. Just here to add my point. Digital lens corrections are part of optical lens design in these days. Some lens even can't work without it, and users can't turn it off in camera. For those cases, it's not an editing process, but necessary process to obey the lens designs and how they work.

For example, Canon RF 16mm f/2.8. Canon push it to the limit, the image circle even cannot cover the sensor. Similar situation on Canon 14-35mm f/4 "L" lens. This will gradually become the norm.

[wpferguson]

This issue isn't about some lenses requiring correction. Agreed, they do. For those lens and camera combinations, set a preset to automatically turn on lens correction.

The issue is about having lens correction automatically turned on for ALL users even if they don't own or use one of these newer lenses that require correction in order to be useful.

[garrett]

set a preset to automatically turn on lens correction

The problem is that you're asking a user to find out that their lens is basically broken in darktable by default, and then how to figure out how to fix it.

darktable (like any other software) shouldn't be broken by default.

We, users of darktable who are photographers, are asking darktable developers to not have our lenses broken in darktable. Some of us can jump through hoops to fix darktable with our settings (once we figure out how), but not everyone can.

That's the issue.

[wpferguson]

darktable (like any other software) shouldn't be broken by default.

My Caono 7D recognizes by Sigma 50-100 f/1.8 as a Canon 28mm f/1.8. If lens correction is on by default, then a wide angle lens correction will be applied to by zoom telephoto. My Sigma 100-400 is identified as a Canon 100-400. Close but not the same. My Sigma 17-50 f/2.8 is identified as a Canon 17-55 f/2.8. Again, close but not the same.

Here's an article about the problem, https://www.lensrentals.com/blog/2012/08/canon-illumination-correction-and-third-party-lenses/

It was bad enough that I wrote a script to correct it (correct_lens.lua).

If I use some of my old FD glass and shoot it adapted to my 7D, then there is no lens data unless I program it into the adapter, but even so there is no correction data.

There are too many instances where lens correction wont work, or will work incorrectly, when turned on by default which goes back to your "darktable (like any other software) shouldn't be broken by default".

[TurboGit]

darktable (like any other software) shouldn't be broken by default.

I fully agree with you... But after saying that there is quite a lot of effort to reach this goal. First darktable is using lensfun for correcting lens issue and lensfun is based on data uploaded by users. The data range from very good to poor or missing part (only distortion but no vignetting for example).

We (darktable team) don't have the money to buy every camera/lens in the market to do the work. And even if we had the money for this testing all combination would take a huge amount of time.

So, as I said, even if I agree with you I see no proper solution.

Let's move forward, we don't enable lens by default because we know there is issue for some camera/lens and we even know that the lens database is missing some camera and lens. That's the current state.