Create sidecar file on all editing action

[paolobenve]

Darktable 4.4.2, self-compiled, ubuntu

If your "write sidecar file for each image" XMP (in storage tab) preference is "after edit", you have actions that are actually editing ones that doesn't write the sidecar file.

One of these is when you set the gps values for the image: the gps values are stored in the db, but no sidecar file is created.

Another one is when you modify the photos date/time with the geotagging module.

Now, if you perform one of those actions and don't make any other editing, and move/rename the raw file with the file manager, you loose the gps values you have set.

But even if you shouldn't move/rename the raw file, you are in a situation where some change has occurred in the image (actually in the metadata), but only the db, and not the sidecar file, tracks them.

[wpferguson]

You've changed the metadata for the image. I think an "edit" consists of a history stack change, which isn't caused by editing metadata.

Perhaps you want a new feature (option) to write the sidecar file after any change to the image?

[paolobenve]

yes

[gi-man]

But isnt that the almost the same as on import ?

[paolobenve]

no, because with the proposed behaviour the sidecar is generated when the metadata are changed.

[gi-man]

Yes I know. Selecting a star value, writing copyright info, tagging, geotaging... would then all write an xmp in your proposal. So, what is the practical difference and/or benefit vs just creating on import?

[kmilos]

I think it's a good idea, as already brought up in https://github.com/darktable-org/darktable/pull/15145#issuecomment-1703072616 and https://github.com/darktable-org/darktable/issues/3632#issuecomment-1701345684

what is the practical difference and/or benefit vs just creating on import?

You can still have no sidecar if you haven't many ANY changes whatsoever (metadata or edit) with your raw file. One can then use that as a simple indicator to separate changed/edited vs untouched files for example...

[paolobenve]

what is the practical difference and/or benefit vs just creating on import?

Simply, you can decide to not create sidecar files at importing, and your workflow can begin modifying the metadata. It's a legitimate choose. If you doesn't modify the image, you won't have your metadata changes saved in the sidecar files, and you'll loose them when you move/rename the raw file.

[gi-man]

I'm just trying to understand the benefit of importing and not creating and xmp right away, vs creating it when there is a metadata change (star rating, copyright info, gps, etc). Saving 20kb? If all you are doing is culling, dt will delete the xmp when you delete the associated image. What else can you do with the imported image (sans xmp) in the proposed workflow?

[ptilopteri]

it does when you actually delete the image from storage, but maybe not if just removing from db.

[paolobenve]

What else can you do with the imported image (sans xmp) in the proposed workflow?

I see in the file manager that it hasn't any sidecar file and I know that I haven't done anything with it.

On the contrary: I see that the sidecar is there, and that tells me that I have begun to work with it.

It's true that a sidecar file is a little thing, but IMHO it's worth not to have it until it carries some info. A sidecar file created on import doesn't have any significant info until I perform some change in the image or its metadata.

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[Solarer]

I just ran into this issue today, when I had to reimport my latest shots because some thumbnails would not generate correctly. I had star ratings and rejection flags applied to about 800 pictures and lost all of that when I reimported the collection. Since nothing had been written to XMP even though I had the create XMP on edit setting active.

When I take a ton of pictures on vacation I only edit the best shots. That does not mean that I am willing to lose the information which of my images were rated 1-4 stars or rejected. I would still like that information to be secured in an XMP if I opted in for using these files. Any information that (currently) will be stored in an XMP after editing a picture should be written as soon as that information is updated (independent whether the picture was edited or not). Otherwise database and disk state are not in sync.

[ptilopteri]

• Jan Kaufmann @.***> [06-26-24 17:49]:

  I just ran into this issue today, when I had to reimport my latest shots because some thumbnails would not generate correctly. I had star ratings and rejection flags applied to about 800 pictures and lost all of that when I reimported the collection. Since nothing had been written to XMP even though I had the create XMP on edit setting active.

why was it necessary to "reimport"? where the accompanying xmp files not in the same directory? did you confirm the xmp settings were "on edit" why not "on import"

did you actually rate/star the images in dt or in camera or another application?

there is a setting for importing ratings contained within the raw image but I don't believe it is applicable for all cameras. is your's included?

When I take a ton of pictures on vacation I only edit the best shots.

understandable

That does not mean that I am willing to lose the information which of my images were rated 1-4 stars or rejected. I would still like that information to be secured in an XMP if I opted in for using these files.

that is a normal expectation and in my experience, what actually happens.

Any information that (currently) will be stored in an XMP after editing a picture should be written as soon as that information is updated (independent whether the picture was edited or not). Otherwise database and disk state are not in sync.

if the setting was on edit, and iiuc, assigning a rating in dt is and "edit" and should have been written to both the xmp and the library.

-- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet oftc

[Solarer]

• some thumbnails where showing skulls, others where black. I confirmed that the images were present and undamaged and tried to refresh the thumbnails by choosing different zoom levels in filemanager layout and deleting the thumbnail cache in .cache/darktable. Nothing worked, so I deleted the files from the catalog and added them again. That fixed the issues.
• The setting is on edit. No xmp files have been created so far since I was just in the process of culling the images. I only applied ratings and did not edit a single picture in the darktable yet - so nothing was generated.
• I did not chose on import because I did not see any advantage from it. Why should I generate thousands of small files in advance that clutter the directory when I can just generate them on demand as soon as I make my first edit? Of course I assumed that any changes to meta data would count as edit and thus generate the xmp.
• Rating was done only in DT
• Rating does not count as an edit with regards to xmp files as of now. There have been several other issues opened about this and recently tags were added. But geodata does not seem to be and ratings definitively are not (I just tested again). No xmp files are generated and since there is no xmp file to store the information in - it is dropped silently.

Added tags but no geolocation: https://github.com/darktable-org/darktable/pull/15373

Same issue as mine. Closed as partially fixed and discussion should continue here in #15330 https://github.com/darktable-org/darktable/issues/14913

[wpferguson]

Rating does not count as an edit with regards to xmp files as of now

If you have the default setting and import images, each image's xmp file has a xmp:Rating = "0" entry, provided the image didn't have a rating from the camera. If we consider assigning a rating an edit, then every image would have an XMP file, negating the "advantages" of after edit.

If we consider the XMP files as the last resort database backup, then why would a user want only a partial backup? I think that after edit causes more problems than it fixes.

[Solarer]

If we consider assigning a rating an edit, then every image would have an XMP file, negating the "advantages" of after edit.

Importing the rating of an image should not count as an edit. There is no need since the rating is already backed up in the image file itself. If I had to reimport the file, it would restore the rating just fine.

• import a 0 star image -> no xmp
• import a 5 star image -> no xmp
• import a 0 star image and assign 1 star -> generate XMP
• import a 0 star image and assign 1 star, then back to 0 star -> keep XMP and update it to 0 stars
• import an image and assign the same number of stars it already had -> also generate XMP just to be safe

[Solarer]

Any data created by darktable that will be stored in an XMP if the file exists should trigger an XMP generation if the file does not exist. At least that is what create XMP files: on edit implies.

That includes edits to the image, ratings, tags, geotags, time synchronisation, copyright information and others that I might overlook. If the metadata is embedded in the image file and was not changed by dt since import, then there is no need to create the XMP since the data is persistent in the image file.

Yes, that might mean that on edit will generate XMPs for all of my images if I assign a tag or copyright information to all of them and that there is no difference to the on import setting in that case. Which is fine. We just should not imply that any work in progress is stored in two places when it really is not. The default for a database snapshot is once per week - that means that I can lose 7 days of work of preselecting and rating images while using the on edit option.

[ptilopteri]

• Jan Kaufmann @.***> [06-29-24 08:27]:

  Any data created by darktable that will be stored in an XMP if the file exists should trigger an XMP generation if the file does not exist. At least that is what create XMP files: on edit implies.

  That includes edits to the image, ratings, tags, geotags, time synchronisation, copyright information and others that I might overlook. If the metadata is embedded in the image file and was not changed since import, then there is no need to create the XMP since the data is persistent in the image file.

  Yes, that might mean that on edit will generate XMPs for all of my images if I assign a tag or copyright information to all of them and that there is no difference to the on import setting in that case. Which is fine. We just should not imply that any work in progress is stored in two places when it really is not. The default for a database snapshot is once per week - that means that I can lose 7 days of work of preselecting and rating images while using the on edit option.

what relevance is an xmp file if it does not contain the "current" state of edits/ratings/... of it's image?

I have not heard of "default for a database snapshot is once per week" but I may be out-of-touch.

I believe that the only reason(s) for not creating an xmp on import is personal to the user and allows for loss of information and anyone choosing that option should be aware.

the space consumed by the xmp files is miniminal.

but understandable communication is difficult.

-- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet oftc

[wpferguson]

Why don't we admit that XMP after edit doesn't work and all it's added is problems?

People use it expecting that they can restore the database from it and every time someone uses the XMPs for that, we have a new issue.

The problem is that we want it to work like on import but not be on import. What are we gaining from a functionality standpoint? So far it seems to be disappointed/surprised users. And for what?

• In 2022 I shot 148K+ images. The XMP files took up 640MB, which is equivalent to 16 images from my camera. So space isn't a problem.
• The only other reason I've seen stated is to reduce file system clutter. That's not functionality and it's impossible to measure because it would be user specific.

How much effort, code, complexity, etc are we going to invest trying to make a bad idea work?

[ptilopteri]

Your Weekly Ad Preview is Here!