[Enhancement] Support Fooocus metadata

[Eleksar387]

Describe the bug Metadata in PNG is not read. The png does not even appear in Diffusion Tookit.

Version: 1.5.1

Tool: Latest Fooocus including this pull request that adds metadata. I used Automatic1111 style. https://github.com/lllyasviel/Fooocus/pull/1940

To Reproduce See attached test file. Opening the png in text editor shows metadata. But the image does not even get added to Diffusion Toolkit

Expected behavior Import the images and show metadata

Affected Image(s) Happens with all images done with Fooocus and added metadata

[RupertAvery]

Not strictly a bug. I don't support Fooocus yet since this is a new Fooocus feature. (for me anyway) :)

Thanks for the image.

[Eleksar387]

You are right! Sorry not a bug The author of the pull just says it's 99% compatible with Automatic1111 so I thought this is kind of a standard and it should work. Anyway. Let's see if Fooocus will add this feature.

[RupertAvery]

It looks like this in PNGInfo:

parameters

None

Comment

photograph fantasy landscape painting, highly detailed, masterpiece, 50mm . cinematic 4k epic detailed 4k epic detailed photograph shot on kodak detailed cinematic hbo dark moody, 35mm photo, grainy, vignette, vintage, Kodachrome, Lomography, stained, highly detailed, found footage, fantasy landscape painting, highly detailed, masterpiece, romantic, lush, intricate, cinematic light, clear focus, inspired, sincere, beautiful, thoughtful, color, illumination, pretty, attractive, thought, best, dramatic, contemporary, novel, artistic, sharp, modern, luxurious, amazing detail, creative, cool, iconic, fine, fantastic, awesome, perfect, epic Negative prompt: Brad Pitt, bokeh, depth of field, blurry, cropped, regular face, saturated, contrast, deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime, text, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck, deformed, bad anatomy, disfigured, poorly drawn face, mutated, extra limb, ugly, poorly drawn hands, missing limb, floating limbs, disconnected limbs, disconnected head, malformed hands, long neck, mutated hands and fingers, bad hands, missing fingers, cropped, worst quality, low quality, mutation, poorly drawn, huge calf, bad hands, fused hand, missing hand, disappearing arms, disappearing thigh, disappearing calf, disappearing legs, missing fingers, fused fingers, abnormal eye proportion, Abnormal hands, abnormal legs, abnormal feet, abnormal fingers, drawing, painting, crayon, sketch, graphite, impressionist, noisy, blurry, soft, deformed, ugly, anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, unrealistic, saturated, high contrast, big nose, painting, drawing, sketch, cartoon, anime, manga, render, CG, 3d, watermark, signature, label, bikini Steps: 60, Sampler: dpmpp_2m_sde_gpu, CFG scale: 12.0, Seed: 6168027118089694081, Size: 1024x1024, Model hash: aeb7e9e689, Model: SDXL\juggernautXL_v8Rundiffusion, Lora hashes: "SDXL_FILM_PHOTOGRAPHY_STYLE_BetaV0: bc6db9d8f1", Denoising strength: 1.0, Version: Fooocus v2.1.862, Created By:

So it's there, it's just in a completely different section.

[mashb1t]

@RupertAvery thank you for the hint, much appreciated! In this line of code "Comments" is used but i'll change this to "parameters" (see original code in A1111 here). I assume this would be sufficient. Civitai could read all of the data and the params are the same, so i think this is the last step to make it truly compatible. Would you agree?

@Eleksar387 i'll adjust the code, thanks again for reporting and your analysis!

[RupertAvery]

Can you put an identifier in another tag to identify it as Fooocus? Just in case A1111 or Fooocus diverges metadata implementation, it makes it easier to manage things.

I loop through the PNG Chunk directories, and check the tag names, as you can see, I use the information there to determine how to load. It's fine if you really want to make it compatible with A1111, assuming there is nothing different. It just makes it difficult to adjust if it does change later.

So perhaps add another tag on the same level that identifies it as Fooocus.

                    foreach (var directory in directories)
                    {
                        if (directory.Name == "PNG-tEXt")
                        {
                            foreach (var tag in directory.Tags)
                            {
                                if (tag.Name == "Textual Data")
                                {
                                    if (tag.Description.StartsWith("parameters:"))
                                    {
                                        if (fileParameters == null)
                                        {
                                            var isJson = tag.Description.Substring("parameters: ".Length).Trim().StartsWith("{");
                                            format = isJson ? MetaFormat.RuinedFooocus : MetaFormat.A1111;
                                            fileParameters = isJson ? ReadRuinedFooocusParameters(tag.Description) : ReadA111Parameters(tag.Description);
                                        }
                                    }
                                    else if (tag.Description.StartsWith("Comment:"))
                                    {
                                        if (fileParameters == null)
                                        {
                                            if (directory.Tags.Any(t => t.Description == "Software: NovelAI"))
                                            {
                                                format = MetaFormat.NovelAI;
                                                fileParameters = ReadNovelAIParameters(file, directories);
                                            }
                                            else
                                            {
                                                format = MetaFormat.FooocusMRE;
                                                fileParameters = ReadFooocusMREParameters(tag.Description);
                                            }
                                        }
                                    }
                                    else if (tag.Description == "Software: NovelAI")
                                    {
                                        if (fileParameters == null)
                                        {
                                            format = MetaFormat.NovelAI;
                                            fileParameters = ReadNovelAIParameters(file, directories);
                                        }
                                    }
                                    else if (tag.Description.StartsWith("Dream: "))
                                    {
                                        format = MetaFormat.InvokeAI;
                                        fileParameters = ReadInvokeAIParameters(file, tag.Description);
                                    }
                                    else if (tag.Description.StartsWith("sd-metadata: "))
                                    {
                                        format = MetaFormat.InvokeAINew;
                                        fileParameters = ReadInvokeAIParametersNew(file, tag.Description);
                                    }
                                    else if (tag.Description.StartsWith("invokeai_metadata: "))
                                    {
                                        format = MetaFormat.InvokeAI2;
                                        fileParameters = ReadInvokeAIParameters2(file, tag.Description);
                                    }
                                    else if (tag.Description.StartsWith("prompt: "))
                                    {
                                        if (fileParameters == null)
                                        {
                                            var isJson = tag.Description.Substring("prompt: ".Length).Trim().StartsWith("{");
                                            format = isJson ? MetaFormat.ComfyUI : MetaFormat.EasyDiffusion;
                                            fileParameters = isJson ? ReadComfyUIParameters(file, tag.Description) : ReadEasyDiffusionParameters(file, directories);
                                        }
                                    }
                                    else if (tag.Description.StartsWith("Score:"))
                                    {
                                        decimal.TryParse(tag.Description[6..], NumberStyles.Any, CultureInfo.InvariantCulture, out aestheticScore);
                                    }
                                    else if (tag.Description.StartsWith("aesthetic_score:"))
                                    {
                                        decimal.TryParse(tag.Description[16..], NumberStyles.Any, CultureInfo.InvariantCulture, out aestheticScore);
                                    }

                                }
                            }
                        }
                        else if (directory.Name == "PNG-iTXt")
                        {
                            foreach (var tag in directory.Tags)
                            {
                                if (tag.Name == "Textual Data" && tag.Description.StartsWith("parameters:"))
                                {
                                    format = MetaFormat.A1111;
                                    fileParameters = ReadA111Parameters(tag.Description);
                                }
                            }

                        }
                        else if (directory.Name == "Exif SubIFD")
                        {
                            foreach (var tag in directory.Tags)
                            {
                                if (tag.Name == "User Comment")
                                {
                                    format = MetaFormat.A1111;
                                    fileParameters = ReadA111Parameters(tag.Description);
                                }
                            }
                        }
                    }

[mashb1t]

Will do, thanks for the heads-up. I've adapted the metadata handling from Fooocus-MRE, but there are some small differences as MRE hasn't been updated in a while and Fooocus (core) uses a few different parameter namings. I've made it compatible with ReadRuinedFooocusParameters for testing, will add an additional tag on first level for the future to distinguish.

A1111 handling is... let's say primitive, but it works for all parameters except somehow the model name. https://github.com/RupertAvery/DiffusionToolkit/blob/0693adae498730336ed4b701a1aec7095d49a497/Diffusion.Scanner/Metadata.cs#L1026 https://github.com/mashb1t/Fooocus/blob/feature/add-metadata-to-files/modules/async_worker.py#L872

Can you maybe support here in debugging what the issue could be?

[RupertAvery]

What seems to be the issue?

It reads fine in the latest version

Note that it reads it using A1111 parameters.

For ReadRuinedFooocusParameters, that is in JSON format. Do you store both or just one?

RuinedFooocus uses the same tag as A1111 and also starts with parameters:, but it instead uses JSON, so I just check for the presence of a curly brace. Hopefully no one will start their prompts with a curly brace. :)

[mashb1t]

The model root was (still is) not correctly set for me, did always show "not found", but hash information is present.

For ReadRuinedFooocusParameters, that is in JSON format. Do you store both or just one?

Just one, configurable with radio buttons. There are quite a few parameters, 32 + optional 5x3 (for LoRA) => 47 in total, so it's quite much to map but i'll merge prompts in one attribute with resolved wildcards and style params.

This issue can be closed, i'd open a new one or directly implement mapping in a PR when Fooocus is ready. Thank you for your support!

[Eleksar387]

You guys rock! Thx @mashb1t and @RupertAvery for your efforts! I now see the images in Diffusion Toolkit including the metadata. 🤩

[mashb1t]

Implementation of https://github.com/lllyasviel/Fooocus/pull/1940 has completed, will update you guys again when it's merged.

[mashb1t]

@RupertAvery i've just released the mentioned metadata feature in Fooocus v2.2.0, see meta_parser.py. Your suggestion to add a key on root level has been included and the different schemes ("fooocus" / "a1111") can now be differentiated by the key metadata_scheme.

The Fooocus scheme features an array syntax, while A1111 is the well-known schema with concatenated strings and comma separation.

Example images

Fooocus (can be png with pnginfo, but also jpg and webp with EXIF data): - PNG  - JPG (not really relevant in this repository i assume)  A1111 - PNG 

It would be great if you could (help to) add the new schema to this repository. Thank you very much!

[RupertAvery]

Latest commits have implemented readers for Fooocus metadata

[mashb1t]

@RupertAvery thank you for the implementation, much appreciated! @Eleksar387 please check if this works for you and close the issue if solved. Thanks!