[BUG] Scan folder missing thousands

[Ayelwen]

Describe the bug When setting up Diffusion Toolkit and adding the Outdir which contains the generations, the toolkit only finds 24 out of several thousand. The InvokeAI outdir folder structure has some subfolders for Thumbnails which I put in Exclude as I don't want those and I don't want duplicates.

Version: I was first using 1.2.1 and had the issue, but also tried the 1.2.2 posted in Issues on github and it didn't seem to have any effect.

To Reproduce Generate using latest InvokeAI and nothing shows up.

Expected behavior I expect to be able to view my generations

Source Image

Source Image or Screenshots

Additional context Add any other context about the problem here.

[RupertAvery]

Can you show me your General settings?

Do you have Recursive checked?

Are the missing files extension added in the File Extensions list?

The image you provided loads properly. Does it also load properly in your case? Is it part of the 24 images?

I will need a sample of images that do not load for you. Also, please attach your DiffusionToolkit.log file if it contains any error messages.

[Ayelwen]

General settings:

I do have recursive checked (screenshot). The files are just PNG which is in the list. The image I shared is not part of the 24 it discovers.

[Ayelwen]

I went ahead and did another test as follows (incase the subfolder exclusions were related to the problem)

1. I noticed that Thumbnails are webm files while the generations are PNGs.
2. So I removed webm as a file extension in the tool, and removed all the exclusions.
3. rescanned but still only shows a few files. screenshot the folder as well showing the number and file type

[RupertAvery]

Hello, sorry for the delay. I've been busy with other things.

Can you upload somewhere a set of images, maybe 10-20, that you're sure don't work? I can't seem to replicate your issue with the image you provided.

Also, is there any error listed in diffusion-toolkit.log?

[Ayelwen]

Sorry, I never got back to you. I will get those now.

[Ayelwen]

okay, yes, there are thousdands of errors. its a really long file. An error occurred while reading C:\Users\ayelw\invokeai\outputs\000001.f75fd3bd.3539081754.postprocessed.png: The requested operation requires an element of type 'Array', but the target element has type 'String'.

at System.Text.Json.JsonElement.EnumerateArray() at Diffusion.IO.Metadata.ReadInvokeAIParametersNew(String file, IEnumerable1 directories) at Diffusion.IO.Metadata.ReadFromFile(String file) at Diffusion.IO.Scanner.Scan(IEnumerable1 files)+MoveNext() 5/29/2023 4:08:39 PM: An error occurred while reading C:\Users\ayelw\invokeai\outputs\000005.66a11412.874975903.postprocessed.png: The requested operation requires an element of type 'Array', but the target element has type 'String'.

[Ayelwen]

[Ayelwen]

[Ayelwen]

Anyone know if Avery went on vacation?

[RupertAvery]

I'm in the process of moving to another country, also been a bit busy with work, so apologies if I haven't been able to look into issues. I promise I will do so, when I'm freed up. The move will happen in the next two months, but hopefully I can find some time to squeeze in a few fixes. Thanks for holding up!

[RupertAvery]

Hi, can you attach some of the images that have that specific error?

[Ayelwen]

yes sir! However, there is a slight change now. All of the above is Invoke 2.3.# behavior. Invoke just completed a complete rewrite of their system about a week ago. Invoke 3.0 images your Diffusion Toolkit is finding and adding to the library, but it is not detecting the metadata. I will attach the following images:

1. Direct image that was generated
2. Screenshot from your tool where it found the image, but no metadata
3. Screenshot from Invoke 3 showing the metadata

[Ayelwen]

[RupertAvery]

This looks like a new metadata format for InvokeAI.

I've added support for it.

[RupertAvery]

Let me know if the latest version 1.3 fixes this issue