Doesn't pick up all the files

[f-rank]

Version: 0.8.1 on Windows 10

It doesn't seem to be picking up all the files, I can provide one example file, don't know if it is something specific in the files that makes this happen. Maybe you can figure what it is. Anywhere I place this example file (and many others), add it to the directories to scan, it doesn't show up.

https://ufile.io/fa9ewrla

It doesn't seem related to the number of files in a scanned directory, tried with one single file in a dir and it also didn't pick it up.

[csmit195]

I think I have the same, I've got it set to my outputs folder, and its picking up my older generations. It says at the bottom, 69 images in database, bbut when refreshing metadata, it scans 900+ images, but only 69 are being showed. would really like a usuable tool.

[RupertAvery]

@f-rank it reads the image just fine, using the latest version.

@csmit195 can you upload here sample images that aren't loading, and can you make sure you are on version 0.8.1?

I can try adding logging to the scanning part to see if anything is broken.

[f-rank]

@RupertAvery , I'm on v0.8.1 as stated, and it doesn't see it on this computer. This one and many.

[nicknack23]

Great little app, but it's randomly missing a bunch of files for me too. For example, v0.8.1 only detects one out of 10 sample images for the Hassanblend model (warning NSFW).

Hassanblend: https://civitai.com/models/1173/hassanblend-all-versions

Direct link to sample images: https://mega.nz/file/tDJDGDAY#oxqImbvU5DPj11zQCEUMWrE6wMxwlGLIbiFPGoQVmXA

[RupertAvery]

The latest version now has logging and removes suppression of errors during scanning.

Please try the latest version, try rescan folders, and check the DiffusionToolkit.log file for any errors and post here.

Note that the latest version will scan in all images now (see #41).

This means that even though no metadata was found, the image will still be added, but cannot be searchable via metadata.

I hope that there actually is some sort of error that happens on your end, and the error suppression was hiding it.

That said, I've tried the hassanblend sample images and they all work for me.

This is how metadata scanning works,

1. File types with known metadata fields (PNG, JPG via EXIF) are parsed for the metadata fields
2. a set of heuristics is applied to see what generator created the metadata (since each generator has their own way of doing so)
3. If a match is found, the appropriate method is called to parse the metadata in that format (A1111, InvokeAI, NovelAI or SD)
4. If no match is found, any text files with matching filenames are attempted to be used to read metadata.
5. If no match is found, the NoMetadata flag is set. This is a new feature, previously, a no-match would mean that the image would be ignored.

@f-rank, I would be interested:

• if @nicknack23 's sample images work for you
• if you can provide many more sample images that don't load, maybe I can try to find some pattern as to why they won't load

Thank you

[RupertAvery]

Please try the latest version with better logging and post the stacktrace here (in DiffusionToolkit.log)

https://github.com/RupertAvery/DiffusionToolkit/releases/tag/beta_v0.8.3

[nicknack23]

Wow you're fast! :) Here is the log file with scan errors: DiffusionToolkit.log

EDIT: Just a thought - could this be a Windows path length problem? EDIT 2: Nope. Still the same problem when I moved the sample folder to the root of the C-drive.

[RupertAvery]

I think this is a region problem. My fault probably.

I'm not quite sure, when you drag a PNG into PNGInfo, are the CFG Scales or any numbers with decimals stored as 1.2 or as 1,2?

[f-rank]

• if @nicknack23 's sample images work for you

• if you can provide many more sample images that don't load, maybe I can try to find some pattern as to why they won't load

It only picks up one image from @nicknack23 's example batch

Will provide images later, soon as I'm at the machine with the SD install.

Thanks for looking into it, it's really random.

[RupertAvery]

Can you provide what country/ cultureinfo you are using?

e.g.

• en-US - English US
• de-DE - German
• tr-TR - Turkish

I assume that the error is related to parsing decimal values, but using the tr-TR culture that expects decimals to be commas, I don't get an error, instead I get the number "7.5" parsed as 75.

Quite strange, but at least I have some idea as to what miught be the cause of the problem.

[f-rank]

Can you provide what country/ cultureinfo you are using?

Region pt - Portugal

Regional Format Portuguese (Portugal)

[RupertAvery]

With your information I was able to replicate the issue and devise a fix.

Please run Rebuild Images after updating in case CFG values with decimal values are read incorrectly *e.g. 7.5 becomes 75)

Please see the latest version

[f-rank]

Please see the latest version

You fixed it.

I'll try to provide the log for the other issue/thread when it happens, maybe it will provide the insight you need.

[nicknack23]

Yes, confirmed fixed for me too - thank you and great work! :)