Feature: Support filtering in/out unavailable images

[przemoc]

Is your feature request related to a problem? Please describe.

1. Avoid seeing unavailable images.
2. Display only unavailable images, so that they could be easily selected and removed.

Describe the solution you'd like Additional filter, Availability, which could be enabled, and then filtered by true of false.

Describe alternatives you've considered For 1.: not aware of any. For 2.: there could be a button to remove unavailable images.

Additional context Recently there was change in behavior that previously indexed images that are no longer available are kept in DB. It is reasonable approach in the context of removable storage, but there are also users not using removable media that may reorganize their images outside of DiffusionToolkit, so it would be good to be able to still remove stale content without extra effort (which seems to be necessary now, unless I'm missing some obvious feature).

BTW Thanks for creating DiffusionToolkit!

[RupertAvery]

The thing is that Unavailable images aren't a database "data point", if that makes sense. They only happen if the user deletes the image without Diffusion Toolkit knowing, so I have to do at least a directory scan everywhere to find them. This is pretty fast, but it requires scanning through the entire hard disk (or at least all the subdirectories under each watched folder). I assume it's really fast anyway, and that Windows is just checking the file table if it's there, but I still would rather avoid making unnecesary file scans, so I don't necessarily want a filter for Available images.

I've added an menu item to scan and delete / tag for delete unavailable images.

This build artifact should have these changes, if you don't want to wait for an official build:

https://github.com/RupertAvery/DiffusionToolkit/actions/runs/8773444541/artifacts/1433316835

Latest builds will appear in https://github.com/RupertAvery/DiffusionToolkit/actions. Click on the workflow run and scroll down to the artifacts and download DiffusionToolkit.

[przemoc]

Thank you for letting me know. I downloaded g9269cd0 build that you linked, which allowed me to remove some stale entries.

I noticed one issue, though. I was in Search view, which was showing many Unavailable entries, before I used Tools > Unavailable Files. I chosen Delete immediately option, upon starting progress bar quickly reached ~70%, it seemed to be stuck, but eventually I got information that images were deleted. When I returned to main window, Unavailable entries were still showing and number of results in status bar has not updated. I had to change view to something else and go back to Search view to make it stop showing Unavailable entries and update number of results in status bar.

[danrazor]

hi, what if auto tag is used with first scan of available images and then only available tag is used, just like favorites and tags or albums? would this solve this issue? tag: show available and unavailable? or this is exactly what the new updates in this beta build meant?

[DmytroSokhach]

Hi @RupertAvery. First of all thank you for great tool! Artifacts from your commentary in April are no longer available. 😔 Any chance you would merge the changes and make fresh release? I've recently realized I have bunch of "Unavailable" and just can't use DiffusionToolkit normally, so need to make some cleanup of those from the DB. Since plenty of images are there about 100k+ from multiple folders I'd like to avoid re-adding them from scratch.

[DmytroSokhach]

Hi @RupertAvery is there a chance for new release soon with that fix?

I also try to filter those by Path, because "Unavailable: true" doesn't filter anything yet. But then trying to "Mark all search results for Deletion" and then app crashes with error.

UPD made PR for this error: https://github.com/RupertAvery/DiffusionToolkit/pull/267

[DmytroSokhach]

Oh, Jeez! While debugging I've realized how to resolve that issue above. Root cause: images were moved to another folder (source folder does not exist anymore) Solution

1. Add top level folder (ignore warning message if any)
2. Click Tools -> "Scan for Unavailable images"
   3. Select folder from p.1
   4. Select "Scan and Update" (this will set flag Unavailable = 1)
   5. Click OK
3. After update has finished, can Filter with Unavailable: true
4. Tools -> Search Results -> Remove all query results from the database Profit! 🙌