[Bug]: Dynamic prompts/wildcards highly change the results vs firstpass, on the hi-res pass on new hi-res fix.

Panchovix commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits

What happened?

Since https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/ff0e17174f8d93a71fdd5a4a80a4629bbf97f822 dev commit, if you use it at the same time with wildcards/dynamic prompts (https://github.com/adieyal/sd-dynamic-prompts), it can use more than one value inside the {|} brackets.

It is very noticeable on the hires fix pass.

For example, if you use "{green|red|blue|black} eyes", and the choosen token was "green", the result can be a mix of all those eye colours, or a mix of green/red, green/blue, etc

This is more pronounced when using loras/tis with this syntax (for example, "{lora:a1:1|lora:a2:1|lora:a3:1|...}"

The issue get worse when using more than one dyanmic prompt at the same time (aka, random eyes, random hair colours, random hair style, etc)

The issue doesn't happen if the commit is reverted to https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/3885f8a63e0954ac96e6681aa8b33281f10337a2

And the issue doesn't happen when not using dynamic prompts/wildcards.

No visible error in the console

Steps to reproduce the problem

Generate any prompt with an extense wildcard/dynamic prompt (example if you want to test): {blue-gray eyes|orange eyes|light blue eyes|dark red eyes|blind eyes|silver eyes|light brown eyes|light green eyes|dark brown eyes|retro eyes|heart eyes|dark purple eyes|amber eyes|light pink eyes|hazel eyes|yellow eyes|blue eyes|dark green eyes|light red eyes|red eyes|diamond eyes|brown eyes|closed eyes|cat eyes|dark blue eyes|green-hazel eyes|dark pink eyes|purple eyes|white eyes|demon eyes|squinted eyes|gray eyes|pink eyes|light purple eyes|magenta eyes|sparkling eyes|gold eyes|black eyes|green eyes|heterochromia}
Select/enable hi-res fix
Generate the image and see if the eye colour result is the desired one, and not a mix of the others colours.

What should have happened?

The result should be only the selected by the wildcard/dynamic prompt, and not a mix of the possible values inside the {} brackets.

Commit where the problem happens

https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/ff0e17174f8d93a71fdd5a4a80a4629bbf97f822

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

Command Line Arguments

--xformers --listen --api --enable-insecure-extension-access --opt-channelslast --administrator

List of extensions

a1111-sd-webui-lycoris sd_webui_SAG sd-dynamic-prompts sd-webui-controlnet stable-diffusion-webui-wd14-tagger sd-webui-additional-networks

Console logs

Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]
Version: v1.2.1-297-g39ec4f06
Commit hash: 39ec4f06ffb2c26e1298b2c5d80874dc3fd693ac
Installing requirements

Launching Web UI with arguments: --xformers --listen --api --enable-insecure-extension-access --opt-channelslast --administrator
[AddNet] Updating model hashes...
100%|██████████████████████████████████████████████████████████████████████████| 4385/4385 [00:00<00:00, 114855.39it/s]
[AddNet] Updating model hashes...
100%|██████████████████████████████████████████████████████████████████████████| 4385/4385 [00:00<00:00, 116944.78it/s]
ControlNet v1.1.180
ControlNet v1.1.180
Loading weights [fc7a94c300] from G:\Stable difussion\stable-diffusion-webui\models\Stable-diffusion\30\a_m\foxy_meinah_mp4v3_hhmv2_xxxmix-half.safetensors
Creating model from config: G:\Stable difussion\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: G:\Stable difussion\stable-diffusion-webui\models\VAE\vae-ft-ema-560000-ema-pruned.vae.pt
Applying xformers cross attention optimization.
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 25.6s (import torch: 1.4s, import gradio: 0.9s, import ldm: 0.4s, other imports: 1.9s, list SD models: 0.7s, load scripts: 14.4s, create ui: 1.3s, gradio launch: 4.5s).
Textual inversion embeddings loaded(2): EasyNegative, verybadimagenegative_v1.3
Model loaded in 8.1s (load weights from disk: 0.8s, create model: 0.7s, apply weights to model: 0.7s, apply channels_last: 0.3s, apply half(): 0.4s, load VAE: 0.1s, move model to device: 1.1s, load textual inversion embeddings: 3.8s).
100%|██████████████████████████████████████████████████████████████████████████████████| 80/80 [00:08<00:00,  9.47it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:05<00:00,  1.93it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 80/80 [00:06<00:00, 12.59it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:05<00:00,  1.92it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████| 180/180 [00:38<00:00,  4.67it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 80/80 [00:06<00:00, 12.09it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:05<00:00,  1.92it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 80/80 [00:06<00:00, 12.59it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:05<00:00,  1.92it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████| 180/180 [00:32<00:00,  5.54it/s]

Additional information

No response

Panchovix commented 1 year ago

To add, I've also tried adding the same exact positive and negative prompt, into the prompt for hires fix, and the issue persist

Panchovix commented 1 year ago

I think I have kinda found the issue, not sure how to fix it.

When on modules/processing.py, changing the lines

        if self.hr_prompt == '':
            self.hr_prompt = self.prompt

        if self.hr_negative_prompt == '':
            self.hr_negative_prompt = self.negative_prompt

to

        if self.hr_prompt == '':
            self.hr_prompt = self.all_prompts

        if self.hr_negative_prompt == '':
            self.hr_negative_prompt = self.all_negative_prompts

on lines 1077-1081, you can see a print of the hi-res fix prompt on the webui.

And it seems to be doing a pass of the values or {value1|values2} directly on the hires pass, instead of using the value that should be selected on the firstpass.

Without changing those lines, you can add a print and it will have the same issue on the console.

Firstpass: best quality,sfw,1girl,solo,grey hair, multiple braids, very long hair, cat eyes,hair ornament, necklace, tiara, hair between eyes, leggings, shawl, looking down, space elevator, closed mouth, portrait,

Hi-res fix pass: ["best quality,sfw,1girl,solo,__haircolor__, __hairstyles__, __hairlength__, __eyecolour__, __attire_jewelry_and_accessories_head_and_face__, __accessories__, __attire_headwear__, __bangs__, __legwear__, __neckwear__, __lookingdirections__, {__locations_outdoors_natural__|__locations_buildings__}, {__face_emotions__|__face__}, portrait,"]

Basically any prompt inside of processing.py is using the default prompt without wildcards/dynamic prompts with __ or {} syntax

Panchovix commented 1 year ago

Okay, found the total culprit and not sure why it happens.

On modules/processing.py, line 1126 to 1135

    def parse_extra_network_prompts(self):
        res = super().parse_extra_network_prompts()

        if self.enable_hr:
            self.hr_prompts = self.all_hr_prompts[self.iteration * self.batch_size:(self.iteration + 1) * self.batch_size]
            self.hr_negative_prompts = self.all_hr_negative_prompts[self.iteration * self.batch_size:(self.iteration + 1) * self.batch_size]

            self.hr_prompts, self.hr_extra_network_data = extra_networks.parse_prompts(self.hr_prompts)

        return res

Seems that the first 2 lines after if self.enable_hr, or, specially the self.hr_prompts = self.all_hr_prompts[self.iteration * self.batch_size:(self.iteration + 1) * self.batch_size] lines makes the issue surface.

If you comment this line, or well all the function, it works as it was before https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/ff0e17174f8d93a71fdd5a4a80a4629bbf97f822 commit, but then hi-res negative prompt doesn't work by itself. It needs to have any prompt on the positive hires prompt. Now, doing that, makes the issue come back (getting tokens outside the {} or wildcard values), but it is that code that's doing the issue.

Alyndiar commented 1 year ago

From what I can see, this issue is still not solved. I still get highly different results on hires vs firstpass, nothing like I got with 1.2.1.

Alyndiar commented 1 year ago

As a suggestion: Once Dynamic Prompt/wildcards are evaluated on the first pass, if hr_prompt is empty (same thing for hr_negative_prompt) then the evaluated prompt with all cases/select/wildcard resolved should be copied to the hr_prompt. Of course, if the hr prompt is specified, this does not apply..

Panchovix commented 1 year ago

Closing this as PR https://github.com/adieyal/sd-dynamic-prompts/pull/495 and PR https://github.com/adieyal/sd-dynamic-prompts/pull/498 fixed it.

@Alyndiar updating your dynamic prompts extension should fix the issue.

brimston3 commented 1 year ago

As a suggestion: Once Dynamic Prompt/wildcards are evaluated on the first pass, if hr_prompt is empty (same thing for hr_negative_prompt) then the evaluated prompt with all cases/select/wildcard resolved should be copied to the hr_prompt. Of course, if the hr prompt is specified, this does not apply..

To any future extension implementers who come across this suggestion, you actually have to detect if hr_prompt == prompt and hr_negative_prompt == negative_prompt. This is because those fields are automatically populated prior to the call to the extension's Script.process().

AUTOMATIC1111 / stable-diffusion-webui