Allow an option for DNG files to store most recent preview, edit history, snapshots, other metadata in DNG files in addition to database and/or xmp

[kosulin]

Is your feature request related to a problem? Please describe.

File portability and team work is not easy to achieve: team work on the same image is not easy - database cannot be shared, and xmp files are the only option for the collaborative post-processing. But xmp files do not store the most recent preview, and each time the file is sent for review (to those without Darktable) it have to be exported to JPG, a step that can be eliminated.

Describe the solution you'd like

I usually convert my RAW files to DNG on a shared drive (with original cr3 embeded inside), import to Lightroom, and apply all edits to this DNG. Another person imports this DNG to their Lightroom, applies whatever crops and watermarks they find most suitable, and requests the final review of the image by a 3rd person. If the feedback asks for any changes, we apply them to this DNG, and ask for another review. If not - we export to JPG for final delivery.

I am considering to switch from Lightroom to Darktable, and would strongly prefer to have an option of storing preview with all the changes applied, the edit history, snapshots, and metadata directly in DNG to increase the file portability and simplify collaboration. As a result, all steps, including reviews, can be done by opening a single DNG, and there is no need for a messy JPG export steps. Of course this would mean ability to sync changes, that were made to the DNG externally, to Darktable database.

Alternatives

XMP files are limited feature wise (no preview), and less convenient. Export to JPG is required.

Additional context

[ralfbrown]

I doubt you will get any devs to add that, as darktable's philosophy for its entire 15-year existence has been to never, ever, under any circumstances, modify the original image file.

What you might get is a Lua script which will make a copy of the original DNG as well as export to JPEG, and then embeds the JPEG into the copy of the DNG.

[kosulin]

I understand that, but DNG is not the original, the original CR3 is embedded (and stored separately as well).

[ralfbrown]

Maybe so, but as far as darktable is concerned, the imported file is the original - there are many camera models which generate DNG files as their raw images.

[victoryforce]

I understand that, but DNG is not the original, the original CR3 is embedded (and stored separately as well).

DNG is the original raw format of all smartphones and a certain number of cameras. Yes, you're using DNG as a non-original format in your particular workflow, but that doesn't make the format non-original in general.

As already mentioned, what you want can be implemented by means of Lua. Lua scripting was added precisely to implement such extensions that support specific workflows not very suitable for everyone.

I'm tagging our Lua guru @wpferguson here. Maybe he will add something. But I don't think you should hope for the inclusion of your option in the core of darktable.

[wpferguson]

In order to address this we need to separate what the workflow is from how it's currently implemented.

My understanding of the workflow is:

• an image is selected for processing and a local copy is made
• the image is edited
• the image is sent to another editor for final processing
• the final processed images is submitted for approval
• if changes are requested they go back to ? (one or both editors).
• requested changes are made and resubmitted.

The workflow is currently implemented using lightroom

• The DNG format is used/abused to be a container for the image, it's edits, and associated data
• The DNG is sent to another editor for final processing
• The resultant DNG is sent for approval ?
• The reviewer returns the DNG to the final reviewer ?
• The final reviewer imports the DNG over the old DNG?
• The final reviewer may return the DNG to the original editor, who imports it over the original DNG?

The workflow in darktable could look like:

• configure darktable to check for updated XMP files on start
• create a folder for processing the image
• copy an image into the folder
• import the folder into darktable
• open the image in darkroom
• open duplicate manager
• name the original as original
• create 1 or more duplicates and name them for editing. Since duplicates are just additional XMP files they take up little space. Or, you could skip the duplicating and just use the original image.
• Edit the image
• Use a zip file for the container. Zip up the directory and send it.
• The final editor receives the zip file and unzips it, then imports the directory into darktable.
• The final editor duplicates (or not) and applies the final image touches, the zips up the directory and sends it for approval
• The approval authority unzips the file, imports it into darktable and makes any changes/notes. With a zip file being the container, the approval authority could use a word processor and include a document in the directory to spell out the changes they want.
• They zip the file up and send it back to the final editor
• The final editor unzip the files over the existing directory
• When darktable starts it applies any updated XMP files. The requested changes are made, the directory zipped and sent for approval.
• The approval authority unzips the file over the existing directory. darktable picks up the changed XMP files when it starts and applies the edits.

An advantage of using the zip file format is security since it can be password protected or encrypted if necessary.

Certain parts of the workflow could be automated with a lua script:

• select the original image and the script could create the directory, copy the image, and do the import
• zipping for transport
• unzipping and reimporting