Support .dcp color input profiles

[zygmund2000]

Hi,

Because I didn't find any topic about .dcp input color profiles I submit suggestions for adding this feature do Darktable.

[github-actions[bot]]

This issue did not get any activity in the past 30 days and will be closed in 7 days if no update occurs. Please check if the master branch has fixed it since then.

[abnormally-distributed]

I agree, DCP support would be greatly appreciated. I know darktable has darktable-chart for doing color calibrations, and supports ICC profiles, but DCP profiles are much more flexible and increasingly common. A simple search on google for darktable dcp support shows this is a commonly desired feature for darktable, so I think most users would find it a very welcome addition.

[dclecar]

Please do support .dcp color profile! It is very important!

[kanyck]

Second this. RT does support it for a while.

[github-actions[bot]]

This issue did not get any activity in the past 60 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[ccoenen]

This auto-close action is really annoying for feature requests. There is not much to discuss at the moment. That feature would be nice to have. It will remain "nice to have" until it is implemented.

[victoryforce]

@ccoenen Can you tell me what exactly annoys you so much about the actions of this bot? Perhaps I can change the text of the bot's message so that the purpose of its work is more clear.

[ccoenen]

This is not limited to darktable, I may have reacted somewhat harshly because I get auto-close-bot messages on a weekly basis from all kinds of issues I'm subscribed to.

I don't think auto-closing feature requests is a good idea in general (not just in this project). I can see the usefulness of auto-closing for some purposes. Issues raised that are actually support tickets and such. And I know from first hand experience that an issue tracker is no fun at all if there are thousands of open tickets.

For feature requests, though I think any of these would be preferable to some bot:

• tagging this with help wanted or up for grabs or whatever. If maintainers don't intend to implement this themselves but are generally open for this feature. This should exclude it from auto-closing at the same time or extend the times considerably beyond 60 days. It also communicates "this is not coming, unless someone steps up and does it" which may or may not help recruit other people.
• tagging this with next version or backlog or putting it on some future milestone - if the maintainers intend to work on this themselves. This should also exclude it from some auto-closing bot.
• closing the issue manually, if maintainers do not intend to work on this and also don't really invite outside contributions. For example, when this is not aligning with their vision of the project.

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[ccoenen]

Putting this issue toast back on the toaster, to make it less stale.

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[vorlif]

I think .dcp support would be nice.

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[kanyck]

It seems than the bot is the most active participant in this thread so far...

[TurboGit]

@kanyck : Do you want to take the lead on this project? The bot is active all the days and we are just a bunch of developers. If you have nothing more to offer please let the issue be closed by the bot.

[tpapp]

FWIW, I think that auto-closing issues (and the related warnings from the bot) is a bad policy. A lot of successful FOSS projects have outstanding issues that have been open for years, if not decades. This does not mean that they are irrelevant, and won't be fixed at some point. Even if the OP cannot fix them, someone may come along in due course, or find a host of related issues when doing a refactoring, etc.

I understand that having a lot of open issues can be disheartening as a maintainer, but it is effectively harmless when handled appropriately. Labeling, periodic culling and re-triage of issues is a much better policy. See eg this recent discussion at Kubernetes and their subsequent solution.

[fredguile]

Agree. And 👍🏻 for that feature request.

[gilaberticus]

DCP support would be great since camera "style" .dcp´s from Adobe are already available for almost all cameras (e.g. standard, vivid...)

[Donatzsky]

For all of you that find this so important and useful, it would probably be a good idea if you could actually explain why and how it would fit in your workflow (which none of you have done so far). Unless the developers see a clear benefit, it's unlikely to get implemented.

[tpapp]

@Donatzsky: because it saves work, as one could just grab an existing DCP profile for a camera without calibrating it manually. See eg the third reply by @abnormally-distributed above.

[fredguile]

I've bought specific camera profiles and these were delivered as DCP files.

[ccoenen]

I have a camera in my ZenFone 8, which can shoot in raw format with OpenCamera. The resulting DNGs do not look anything like the jpgs at all. I have created DCP files with the adobe tool and my hope is that this would make processing these DNGs much simpler.

[TurboGit]

As @Donatzsky said, you want it but where this could be supported in dt workflow? No one give a way to properly add support for DCP in current darktable. Maybe in color calibration ? A bit like what is done with "calibrate with a color checker" ?

[kmilos]

Maybe I'm wrong, but I don't think DCP is (readily) compatible with dt's current scene-referred workflow to begin with?

Perhaps w/ the legacy display-referred one, e.g. mapping ProfileToneCurve to base curve, and ProfileHueSatMapData and ProfileLookData to two LUT instances, but not sure dt even has an HSV working color space for those?

But even before thinking about DCP, dt doesn't support dual/triple illuminant input profile interpolation, which is a pre-requisite for it IMHO. That would in turn probably mean a big refactoring of color calibration (everything is relative to D65 currently)...

So, not a trivial request by any means.

[TurboGit]

@ccoenen

I have a camera in my ZenFone 8, which can shoot in raw format with OpenCamera. The resulting DNGs do not look anything like the jpgs at all. I have created DCP files with the adobe tool and my hope is that this would make processing these DNGs much simpler.

Have you tried to take a picture of a color checker and use the "calibrate with a color checker" in Color Calibration ? Should be a good alternative and already supported.

See https://docs.darktable.org/usermanual/4.4/en/module-reference/processing-modules/color-calibration/#extracting-settings-using-a-color-checker

[tpapp]

@TurboGit: the appropriate place for this in the workflow would be where the color matrix is applied, in input color profile.

Yes, people can use a color checker. But that has two downsides:

1. They might not have one available. Yes, they are not expensive, but not cheap either, and need to be carried to a scene.

2. You cannot reuse results for different light conditions, so you have to calibrate again and again.

Just grabbing the matrix from a DCP file is a quick & dirty solution to profiling your camera, at least as a first pass. But given that no input matrices were added in the last 7 years or so, it may be the best one for some people. RawTherapee also has a bunch of DCP files which Darktable users could just grab and use.

[kmilos]

Just grabbing the matrix from a DCP file

The input matrices for all currently supported cameras come from the DNG conversion, i.e. they are equivalent to the "Adobe Standard" DCP matrix already (the D65 one is used only).

no input matrices were added in the last 7 years

We're adding those all the time.

You're confusing this w/ custom calibration matrices. Those are not even used any more AFAIK.

[tpapp]

@kmilos, thanks. so the last column of this table is outdated/irrelevant?

[kanyck]

AFAIK DCP profiles have a clear advantage over ICC in this: ICC is a one-point measurement, so it must be re-calibrated every time the light conditions are changed (which is not only a lot of fiddling. The light may change very fast, so you may have to re-calibrate every few minutes!) From the other side DCPs are two-pointed, so there is an opportunity to extrapolate between two points (and somewhat beyond). Yes, it's not ideal, but from my POV is much better than 1-point anyway. ICCs are quite enough for studio, but for outdoor shooting DCPs fit much better. And yes, there are lots of broken profiles out there but this does not mean that there are no good ones. I use ART along with DT and find DT more convenient and feature-rich, but I have to use ART because of two things: DCP profiles and pixel-shift support. So to say, DT is my main tool (95+% of use) but when I need to get right colors under tough light conditions or the pixel-shift support, I need ART. But I'd like to shift solely on DT, because workflows are quite different between the two, and I have to adapt back and forth. And DT is much more familiar to me, I use it since v0.6.

[ccoenen]

I have a camera in my ZenFone 8, which can shoot in raw format with OpenCamera. The resulting DNGs do not look anything like the jpgs at all. I have created DCP files with the adobe tool and my hope is that this would make processing these DNGs much simpler.

Have you tried to take a picture of a color checker and use the "calibrate with a color checker" in Color Calibration ? Should be a good alternative and already supported.

See https://docs.darktable.org/usermanual/4.4/en/module-reference/processing-modules/color-calibration/#extracting-settings-using-a-color-checker

@TurboGit I have not, but I will take a look! Thank you for mentioning this, I haven't come across this particular way, yet!

[Donatzsky]

because it saves work, as one could just grab an existing DCP profile for a camera without calibrating it manually.

I kind of gathered that already, but you (any of you) still haven't explained the workflow you envision. Other than eventually managing to say that it would go in input color profile, all I have seen so far are fairly generic "it would be easier/better/prettier" statements. That doesn't help anyone to understand how it would, practically speaking, be useful or how it should be implemented in the scene-referred workflow. Note that what darktable-chart is doing is only really useful if you're still in a display-referred workflow.

To help you, here's a step-by-step you can follow:

1. Explain your current workflow.
2. Explain how it would differ if there was support for DCP profiles.

That leads me to a few general observations and recommendations:

1. If you're still working in display-referred, realise that darktable is all-in on scene-referred and as such you shouldn't expect the developers to make shiny new toys for you.
2. Some of you seemingly don't understand how to work efficiently with DT. This video will give you a good basis: https://www.youtube.com/watch?v=5CmsxxxsMDs
3. There are ways already to improve color accuracy in a color calibration centred workflow. Depending on the camera, the difference can be drastic. This video has all the details (it's also in the manual): https://www.youtube.com/watch?v=EteAhSN9W8s
4. If you want your photos to look good before you do anything, darktable is arguably not the right software for you. You should really try RawTherapee or ART instead.
5. DCP profiles are not inherently better than ICC profiles. There are different intentions behind them, and which is better depends on what you're trying to do and your workflow. I'm still going down that rabbit hole, so can't really explain it better than that for now.

@kmilos

You're confusing this w/ custom calibration matrices. Those are not even used any more AFAIK.

They are still available as enhanced color matrix. Color calibration ignores anything but standard color matrix, however, so I suppose you could indeed say they are no longer used.

@tpapp

so the last column of this table is outdated/irrelevant?

Yes. But more because they were often no good to begin with.

[gilaberticus]

Thanks for the videos @Donatzsky. Specially the second one about color calibration contained a lot of new information for me.

@Donatzsky

Explain your current workflow.

My current workflow is scene-referred with sigmoid. So using the typical linear modules and sometimes color zones and local contrast after that.

Explain how it would differ if there was support for DCP profiles.

Abode DNG Converter has two types of dcps available :

1. The 'Abode Standard' ones, which contain the color matrices that if I understand correctly map camera RGB to CIE in the input color profile module. Those are the ones that darktable currently uses as standard (converted to ICC with single illuminant). As I understand what DT does now is use the single illuminant color matrix, then adjust white balance afterwards in color calibration module. I am not sure how big the gains of using dual illuminant DCP with interpolated color matrices would be, but it seems to be the standard method in other software. I understand it is a very fundamental change and therefore very complex to implement. On top of that, being able to use abode DCPs would potentially make the process of supporting new cameras easier and allow users with unsupported cameras to use those directly if they are available.

2. Then there are 'camera style' DCPs from Abode that also contain HueSatDelta tables that can replicate the different OOC styles specifically for each camera (e.g. vivid, standard, landscape etc). In the current scene referred workflow I think that would fit in a module doing something similar to the color lookup table module or 3D LUTs, but before filmic/base curve/sigmoid. This would bring a lot of value to users that are looking to get the OOC look as baseline which is something that is frequently asked in DT forums. And would be achieved specifically for each camera and style model using tried and tested tables from a reliable source. Of course this is under the assumption that those tables do not have a tone curve baked in already

[tpapp]

@Donatzsky: My current workflow is as follows: I work in scene referred, using darktable master (or close). I auto-apply denoise (profiled), lens correction, I use sigmoid, and white balance is camera reference.

My goal is to get a "good enough" starting point for further corrections. Specifically, sometimes my images have a color cast, especially on Panasonic bodies (but for some, that has been improved by inclusion of the color matrix), and while color calibration is a supremely versatile tool, it practically requires that the user comes up with a of 3x3 matrix (this was made much easier by primaries #15113, but I am still learning that tool).

My understanding is that Bayer sensors have their own spectral sensitivity, and correction for this can be approximated by a 3x3 matrix for a particular illuminant only, as it is approximating an integral operator. While technically each illuminant would require its own matrix, my reading of the Adobe DNG standard is that it interpolates between two matrices obtained for different illuminants. I don't know how much this helps as I never used this in practice. I recognize that the ideal solution would be a color checker, but that is not always practical and the results do not generalize as a preset to different illuminants.

My (perhaps naive) idea is that since these color matrices are available in DCP profiles, Darktable could just use them, perform the interpolation, and correct accordingly. I do not have a strong preference about where exactly that would happen in the pipeline.

The advantages would be the same as the current matrix, but more general (ie less need for futher manual correction, faster workflow), and not having to manually add these matrices for each camera (less work for devs).

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.