[Feature] Refresh a collection

[GLLM]

Hi all !

[Problem] I have removed some pictures from a folder which I do use in Darktable. When I use this folder again in DT, there are plenty of unavailable pictures (noted with the white skull on black background thumbnail). Those unavailable thumbnails are slow to appear, and mostly do so when I try to open the pic itself. I'd like to remove all unavailable pictures at once but cannot do so. There is no way to do so, AFAIK. As a matter of fact, folders/collections/roll management is still quite poor w.r.t features.

[Solution] When right-clicking on a folder / film roll / collection, a new menu entry "Refresh collection" could scan the folder, pick up all unavailable pictures and propose to remove them (from darkroom or/and from the disk).

Thanks. GLLM

[ptilopteri]

use: /usr/share/darktable/tools/purge_non_existing_images.sh provided by darktable

[GLLM]

Hi @ptilopteri, @AlicVB ! Thanks for the tip

From a pure UX/UI standpoint, it'd be reaaaally nice to have this integrated somehow in the UI, and to be able to push a parameter for it (a folder for instance), instead of forcing the whole script to run on my 70k pictures collection :) Nice to have would be a live user feeback advising Analyzing xyz / zzz pictures

But I've ran it, and it seems to find non existing pics, but at the end I don't know how many were found & removed ;-) PS : got as many "Error: no such table: masks_history" errors as I have file removed

Thanks GLLM

[phillies]

I second that. I recently moved from Lightroom to darktable and the #1 feature I miss is refreshing my collection. I usually have a folder on my disk where my images are. I copy them manually from several sources and in Lightroom I just click "rescan folder" and get a message how many new images were found and how many were removed. A feature like that would be really useful.

[drbeco]

Darktable would improve immensely with this feature. A simple button to trigger a "reality check" on database. I can't agree more with @GLLM , fom the UX/UI standpoint.

The devs are taking their database too seriously. The users (photographers) are much more interested in the files, they are the real thing. If the database can't easily update itself to mirror the real folder, it is just a useless database.

I've read many blogs and help before comming to find this repo, and in all of them experts tells users to "simple don't delete files outside darktable and you will be fine". This is a lazy solution. Of course being able to delete files from other viewers (faster viewers than darktable, to "clean the folder", is an excelent option).

Take a look at "calibre" (open source for collection of books).

Please, kindly reopen this feature/issue.

[abruenin]

+1 from me. I try to use Darktable as my primary photo editing software, and I always search for that feature. I think, this is perfectly implemented in Digikam or ACDSee, where I can choose to manually refresh the folder or leave it on Auto. This is not only about removing non-existing images. Also new images should be found. And to make it even better, subfolders should be added and removed automatically. So I could add my main directory for new photos. When I add new folders with images, they should just appear on manual or automatic refresh.

[toddmedema]

Following up here as I'd also love to see this. The shell script execution on WIndows doesn't work. (or if it does, it requires a ton of extra setup - Windows command line won't let you execute shell scripts, and trying to execute it on the Git command line tool requires you install sqlite3 which doesn't seem possible, so the only remaining idea I have would be to install full Windows Linux subsystem, which is too much work)

[fulldecent]

A workaround here is to delete your darktable catalog and then reopen the app and reload your (remaining) photos.

If you were using the preference "STORAGE / SIDECAR / SAVE ON EACH EDIT" then all your edits are saved in the sidecar XMP files.

The ratings are NOT saved in the sidecar files and they would be lost with this workaround. (Maybe it would be nice if the ratings were saved in the sidecar, but I digress.)

This workaround doesn't help for lots of people. But it helped for me so I'm sharing it.

[ptilopteri]

ratings are most certainly saved in the accompanying xmp files

[fulldecent]

My experience is that XMPs do not come from rating files. Here is my walkthrough of that.

(Audio is muted in video by default on GitHub)

https://github.com/darktable-org/darktable/assets/382183/94578c64-3a7d-4a09-9c94-5bfc45a10b0e

But if you happen to edit a file then perhaps the rating will go into the XMP at that time.

[fulldecent]

Another workaround here for people is that you can open the darkroom file in DB Browser for SQLite. And you can delete things there. You will VOID your warranty if you do this and don't ask me for any more details about it.

[ptilopteri]

recording the editing of images is one of the reasons for xmp files and rating is an editing action. if you are referring to rating an image in a different application, why would one expect those actions to be recorded by darktable in the xmp file?

[fulldecent]

I don't care about using other applications to do things. I am concerned with rating images inside darktable.

Currently it does not (by default) save my ratings into the side car files. For me that's unfortunate.

For example, separately, I was using Lightroom. I used it only for rating photos and moving them across directories. Apparently no other software does that. Today Lightroom stopped running on my computers (because I block network connections, and requires to phone home). Lightroom also did not save all my ratings out to sidecar files. This means all the time I spent on ratings could be lost. When I'm looking at darktable I am hoping that I wont be screwed again, so I was hoping that my ratings in here would all be saved to sidecar files.

[ptilopteri]

I don't care about using other applications to do things. I am concerned with rating images inside darktable.

Currently it does not (by default) save my ratings into the side car files. For me that's unfortunate.

yes, unfortunate that you do not understand that darktable does save ratings to xmp file when applied within darktable. DARKTABLE SAVES RATINGS TO XMP SIDECAR FILE.

[victoryforce]

Currently it does not (by default) save my ratings into the side car files. For me that's unfortunate.

Darktable does save ratings into sidecar files. You can check this by opening the sidecar as a text file. You'll find a line there that looks like this:

   xmp:Rating="4"

where the number corresponds to the number of stars in the rating.

Regarding your screencast, you obviously have the "write sidecar file for each image" setting set to "after edit". In this mode (it can be called the "lazy" mode), the sidecar for the image is not created immediately after import, but only when some editing of the image by the user has taken place. The screencast shows the expected behavior for this mode. However, this is only a demonstration of when sidecar recording is initiated, but is by no means proof that the sidecar will not contain a rating.

DARKTABLE SAVES RATINGS TO XMP SIDECAR FILE.

@ptilopteri Being calmer and more polite has never hurt anyone. 🙂

[ptilopteri]

DARKTABLE SAVES RATINGS TO XMP SIDECAR FILE.

@ptilopteri Being calmer and more polite has never hurt anyone. 🙂

he didn't get it the first time :)
and I am tooo old and cranky to be much more poleit.

[fulldecent]

I'm pretty sure that I understand what everyone is saying here. But what @ptilopteri says disagrees with what I have shown in my video attachment.

Maybe we're both right.

When I use darkroom it does not make sidecar files. And therefore there is no ratings in the (nonexistant) sidecar files.

When @ptilopteri uses darktable maybe they are using the darkroom module (which I don't use) and that is why they are getting sidecar files.

---

Thank you @victoryforce thank you for explaining. I have identified the problem here. I had specifically chosen ON EDIT with the intention of getting MORE EAGER writes to the file.

The UI currently says "write sidecar file..." and I save files more frequently than I import them so I thought ON EDIT would be MORE EAGER.

But instead this should be named "create sidecar file..." which produces the behavior I want. And which is default.

Here is a PR that fixes it https://github.com/darktable-org/darktable/pull/15145

---

For the avoidance of doubt: once a sidecar file is created (whether ON IMPORT, or ON EDIT then after an edit) any ratings changes will be immediately written to that sidecar file.

[kmilos]

For the avoidance of doubt: once a sidecar file is created (whether ON IMPORT, or ON EDIT then after an edit) any ratings changes will be immediately written to that sidecar file.

Instead of faffing w/ semantics, perhaps there is a new approach to be proposed: write the sidecar on rating (or some other lighttable) change even if it doesn't exist (silently, i.e. treat lighttable changes as ON EDIT, or as a 3rd option)?

[fulldecent]

@kmilos I would like that change. Personally I think that changing a rating is an "edit". (Ignoring that others software like Lightroom treats edits and flagging differently.)

However, still, the current wording is incorrect. The file is not written on edit when you select "write... on edit". Instead the file is created on first edit when "on edit" is selected. And the file is written after every edit (unless NEVER) is checked.

In all cases, the choices are still: don't ever make sidecar files / create sidecar files on first change / always create sidecar files. With default as "always".

[gael-vanderlee]

Any update on the original issue ? I have my pictures on an external hard drive to edit between multiple computers and the desynchronized libraries make it impossible to use.

[wpferguson]

@gael-vanderlee your ESP must be working overtime. I just merged a lua-script, https://github.com/darktable-org/lua-scripts/pull/468, that scans for missing images and/or missing film rolls and then gives you the option to remove the database entries.

If you're on darktable 4.6.x it will appear in your lua-scripts, provided you have check for updates turned on in your preferences.

[gael-vanderlee]

@wpferguson Well I was hoping to get to it but none of my lua scripts work, I have spaces in my windows username so I think it's related to this issue and haven't been able to fix it. When manually updating the lua folder with a git pull, I get an error stating that I need the LUA API version 9.3.0 or later, and I think the current darktable version (4.6.1) only has 9.2.0... Oh well.

[wpferguson]

4.8 comes out next week with API-9.3.0

[gael-vanderlee]

Oh great, thanks for the help !

[gael-vanderlee]

Couldn't get the script to work, had to re import the library. But even that isn't enough, since certain things are not synced beyond the deleted images. Duplicated images for instanced are not detected. Darktable is unsuable for me for now as long as I get database sync issues between devices.