Unexpected behaviour in comlex prompt & hires

[miasik]

My example prompt is:

[A female model : [<'Cameron'><' '><'Diaz'>|<'Drew'><' '><'Barrymore'>|<'Lucy'><' '><'Liu'>] : 0.25]

After rendering I see the face that looks like the expected mix, so the extension works as expected, but without "EmbeddingMerge" in the image info:

[A female model : [ <'EM_1'> <'EM_2'> | <'EM_3'> <'EM_4'> | <'EM_5'> <'EM_6'> ] : 0.25] Negative prompt: sad, monotone, low quality, low resolution, [mutated | extra | missed | broken] fingers Steps: 30, Sampler: DPM++ 3M SDE Karras, CFG scale: 7, Seed: 3564665820, Size: 512x768, Model hash: ce49fd5253, Model: YACAM-SR-16111.fp16, RNG: CPU, CDT: "1,1,0,0,0,0,0,0,0,0,1,-1,1,0", Eta: 0.5, Version: v1.7.0

Hires.fix changes the face so much, see below. The face looks like a1111 painted the checkpoint's default face over, so I'm not sure if the extension works correctly here. After hires.fix I see a separate string for hires.fix in the image info:

[A female model : [ <'EM_1'> <'EM_2'> | <'EM_3'> <'EM_4'> | <'EM_5'> <'EM_6'> ] : 0.25] Negative prompt: sad, monotone, low quality, low resolution, [mutated | extra | missed | broken] fingers Steps: 30, Sampler: DPM++ 3M SDE Karras, CFG scale: 7, Seed: 3564665820, Size: 512x768, Model hash: ce49fd5253, Model: YACAM-SR-16111.fp16, Denoising strength: 0.4, RNG: CPU, Hires prompt: "[A female model : [<'Cameron'><' '><'Diaz'>|<'Drew'><' '><'Barrymore'>|<'Lucy'><' '><'Liu'>] : 0.25]", Hires upscale: 2, Hires steps: 15, Hires upscaler: 4x_NMKD-Siax_200k, CDT: "1,1,0,0,0,0,0,0,0,0,1,-1,1,0", Eta: 0.5, Version: v1.7.0

Conversation my prompt to a simple one changes the image(obviously) but doesn't have any influence on the extension [A female model | <'Cameron'><' '><'Diaz'>|<'Drew'><' '><'Barrymore'>|<'Lucy'><' '><'Liu'>]

[aleksusklim]

1. Why are you using <'Cameron'><' '><'Diaz'> and not just <'Cameron Diaz'>? (I'm just curious, that shouldn't cause the bug)
2. Why are you using "Alternating Words" syntax [ | ] in negative prompt too? (Again, I'm curious: the less steps you have, the less profit you will get of this, because the result would be biased/skewed rather than averaged equally)
3. You are using "Prompt editing" [ X : Y : num] with Alternating Words inside. Are you sure WebUI is not bugging on its own when abused like that? Can you confirm that the result uses all the words properly as expected when invoked without embeddings?

Can you reproduce the bug with highres but without alternating syntax or prompt editing (e.g. without using square brackets)?

We need to understand, is there a bug, where is it if yes, and whose exactly. Maybe processing of highres prompt became different…

Oh, by the way: you can use EmbeddingMerge to pre-create embeddings as normal ones. Will your bug still manifest in the same prompt but without using <''> runtime syntax in favor of simple words to reference embeddings?

[miasik]

1. The idea of using the extension is screening my mixes from civiai automoderator. It's sooo aggressive.
2. I've been in an experiment right now. I decided to minimize repeating the same words in my negatives. As for now I can't say it became worse, so I'm going to leave it as it is now.
3. I'm not sure if I got the question correctly but I'm sure that this technique is great and works correctly. I'm writing this because I use it almost always. I see such the behavior the first time and only with the extension. All my other images works correctly as before.

[miasik]

A simplified prompt: A female model <'Cameron'><' '><'Diaz'> <'Drew'><' '><'Barrymore'> <'Lucy'><' '><'Liu'> Get an error here in the webui log but webui continue working: *** Error running process: E:\SD\automatic1111\extensions\stable-diffusion-webui-embedding-merge\scripts\embedding_merge.py Traceback (most recent call last): File "E:\SD\automatic1111\modules\scripts.py", line 718, in process script.process(p, *script_args) File "E:\SD\automatic1111\extensions\stable-diffusion-webui-embedding-merge\scripts\embedding_merge.py", line 1336, in process getattr(_webui_embedding_merge_,'embedding_merge_extension')(p) File "E:\SD\automatic1111\extensions\stable-diffusion-webui-embedding-merge\scripts\embedding_merge.py", line 1255, in embedding_merge_extension if embed[0]=='<': IndexError: string index out of range Image info: A female model <'EM_1'> <'EM_2'> <'EM_3'> <'EM_4'> <'EM_5'> <'EM_6'> Negative prompt: sad, monotone, low quality, low resolution, [mutated | extra | missed | broken] fingers Steps: 30, Sampler: DPM++ 3M SDE Karras, CFG scale: 7, Seed: 3564665820, Size: 512x768, Model hash: ce49fd5253, Model: YACAM-SR-16111.fp16, RNG: CPU, CDT: "1,1,0,0,0,0,0,0,0,0,1,-1,1,0", Eta: 0.5, Version: v1.7.0 The generated image looks OK:

[miasik]

Hires.fix: The same error in the log. Image info: A female model <'EM_1'> <'EM_2'> <'EM_3'> <'EM_4'> <'EM_5'> <'EM_6'> Negative prompt: sad, monotone, low quality, low resolution, [mutated | extra | missed | broken] fingers Steps: 30, Sampler: DPM++ 3M SDE Karras, CFG scale: 7, Seed: 3564665820, Size: 512x768, Model hash: ce49fd5253, Model: YACAM-SR-16111.fp16, Denoising strength: 0.4, RNG: CPU, Hires prompt: A female model <'Cameron'><' '><'Diaz'> <'Drew'><' '><'Barrymore'> <'Lucy'><' '><'Liu'>, Hires upscale: 2, Hires steps: 15, Hires upscaler: 4x_NMKD-Siax_200k, CDT: "1,1,0,0,0,0,0,0,0,0,1,-1,1,0", Eta: 0.5, Version: v1.7.0 And the upscaled image looks more generalized:

[aleksusklim]

Can you not use <' '>, I'm not even sure how that should work because the space is NOT tokenized as a stand-alone token! (Use https://github.com/AUTOMATIC1111/stable-diffusion-webui-tokenizer to see exactly how words are split; also my EM tab should show you the same)

[aleksusklim]

For me,

Cameron Diaz
Cameron  Diaz
<'Cameron'><'Diaz'>
<'Cameron'> <'Diaz'>
<'Cameron'><''><'Diaz'>
<'Cameron'><' '><'Diaz'>
<'Cameron'> <' '> <'Diaz'>

– Are tokenizing the same way, giving the same raw numbers in vectors.

[miasik]

Removing <' '> removes the error from the log and returns recipe to image info:

A female model <'EM_1'> <'EM_2'> <'EM_3'> <'EM_4'> <'EM_5'> <'EM_6'> Negative prompt: sad, monotone, low quality, low resolution, [mutated | extra | missed | broken] fingers Steps: 30, Sampler: DPM++ 3M SDE Karras, CFG scale: 7, Seed: 3564665820, Size: 512x768, Model hash: ce49fd5253, Model: YACAM-SR-16111.fp16, RNG: CPU, EmbeddingMerge: "<'EM_1'>=<'Cameron'>, <'EM_2'>=<'Diaz'>, <'EM_3'>=<'Drew'>, <'EM_4'>=<'Barrymore'>, <'EM_5'>=<'Lucy'>, <'EM_6'>=<'Liu'>", CDT: "1,1,0,0,0,0,0,0,0,0,1,-1,1,0", Eta: 0.5, Version: v1.7.0

For Hires.fix still the same — a generalized face and a separate string for Hires prompt:

A female model <'EM_1'> <'EM_2'> <'EM_3'> <'EM_4'> <'EM_5'> <'EM_6'> Negative prompt: sad, monotone, low quality, low resolution, [mutated | extra | missed | broken] fingers Steps: 30, Sampler: DPM++ 3M SDE Karras, CFG scale: 7, Seed: 3564665820, Size: 512x768, Model hash: ce49fd5253, Model: YACAM-SR-16111.fp16, Denoising strength: 0.4, RNG: CPU, EmbeddingMerge: "<'EM_1'>=<'Cameron'>, <'EM_2'>=<'Diaz'>, <'EM_3'>=<'Drew'>, <'EM_4'>=<'Barrymore'>, <'EM_5'>=<'Lucy'>, <'EM_6'>=<'Liu'>", Hires prompt: A female model <'Cameron'> <'Diaz'> <'Drew'> <'Barrymore'> <'Lucy'> <'Liu'>, Hires upscale: 2, Hires steps: 15, Hires upscaler: 4x_NMKD-Siax_200k, CDT: "1,1,0,0,0,0,0,0,0,0,1,-1,1,0", Eta: 0.5, Version: v1.7.0

[aleksusklim]

Now try pre-creating embeddings and use those instead of runtime syntax.

To do this, enter each of your words separately into EM tab, and fill the box titled "Type here a name for your new embedding …", for example em_diaz when you parse the string Diaz

Then use that instead of <'Diaz'> At that point you can even disable EmbeddingMerge extension, since created physical files would be accessible to WebUI as normal embeddings.

[miasik]

It works for the first render, hires and both image infos. The hires image looks like the expected mix.

Those are static EM files but I need to work with online generation in prompts.

[aleksusklim]

If that is so, then the problem is in my extension indeed. I will take a look and reply when I'll got something or if I would need a cleaner test case.

[aleksusklim]

@miasik, I fixed both of errors (<''> and hr_prompt), but did not push the production yet.

I've faced another strange issue: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/13830#issuecomment-1853709799

Can you test something for me on the current version?

1. Disable all other third-party extensions.
2. Use simple prompt like <'dog'>
3. Fixed seed, no hires.fix, no negative prompt.
4. Generate; then change the prompt to <'cat'> and generate again.

Is the image is changing for you? If not, that's a big problem! Then, follow the method I linked: comment-out

for cache in caches:
    if cache[0] is not None and cached_params == cache[0]:
        return cache[1]

in modules/processing.py

Then repeat the experiment.

[miasik]

OMG! It had to be my second report here :-D I faced into the issue almost immediately after I installed the extension. Here's my yesterday grid with confirmation: I'm doing the correction and its test right now.

[miasik]

Not the best crocodile I've ever seen but the commenting made the magic!

[aleksusklim]

Good!

I wonder why it wasn't seen much earlier than now… I'll try to fix this and push the new version.

(I also had an ongoing SDXL support edition, but there are too much of new code added, so I will postpone that for now)

[miasik]

Well. I'm pretty much used to this sort of thing. I start using an extension and find bugs that were not detected before. Looking forward to seeing the corrected bug.

[aleksusklim]

I'm pushing right now, please try the new version!

[miasik]

Looks like the bug was eliminated! Thank you!

[miasik]

I was happy too early. Enabling Adetailer breaks the extension again. Even the first step works, but doesn't decrypt the image info: EmbeddingMerge: "<'EM_1'>=<'EM_1'>, <'EM_2'>=<'EM_2'>, <'EM_3'>=<'EM_3'>, <'EM_4'>=<'EM_4'>",

[aleksusklim]

Is it possible to change the order of which extensions are called by WebUI? I heard something about this (for example, it was one of the early purposes of https://github.com/vladmandic/automatic)

Is there any extension to set the order of others? Maybe there is an Issue in the AUTOMATIC1111 repo?

P.S. Maybe try renaming their folders!?

[miasik]

I switched to uddetailer, looks like it works correctly for both passes https://github.com/wkpark/uddetailer

[aleksusklim]

Adetailer was probably mentioned in https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/13830#issuecomment-1792892322 too…

[miasik]

Adetailer was probably mentioned in AUTOMATIC1111/stable-diffusion-webui#13830 (comment) too…

Yes, but it was mentioned as a healer, but it breaks working now.

[aleksusklim]

Have you tried renaming the folder in extensions? What if that was just because the alphabetical order of invocations?

[miasik]

Have you tried renaming the folder in extensions? What if that was just because the alphabetical order of invocations?

I wouldn't play with a1111 if it's working. At least it looks like that ;-)

[aleksusklim]

Also I think when installing an extension by URL, there is a textbox to set the folder name instead of using the default one.

[aleksusklim]

I fixed the bug with ADetailer! The order or naming did not matter because that extension is forcefully calling script sequences manually: https://github.com/Bing-su/adetailer/blob/ebd69df305c20fd880a0b39333eb115369b7de76/scripts/!adetailer.py#L771

Now my decision is cease to work if EmbeddingMerge is somehow already present in the current generation info at the beginning. Looks good.

[miasik]

Could you please open "Discussion" section here?

[aleksusklim]

I don't mind having as much Issues as needed. I will close only fixed bugs; free talk could stay.

[miasik]

I'd like to ask some questions related to the extension but they're offtopic for the issue. Using "Discussion" section is much better as for me

[aleksusklim]

Just create an Issue titled "some questions related to the extension", etc. As I said, I'm perfectly fine!

[aleksusklim]

By the way, the dedicated Discussion page is linked at top of the main readme file of my extension. Discussion: https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/7659

[miasik]

By the way, the dedicated Discussion page That isn't the dedicated page ;-) The dedicated Discussion page for the extension can be turned on in settings for the repository.

[miasik]

@miasik, I fixed both of errors (<''> and hr_prompt), but did not push the production yet.

I've faced another strange issue: AUTOMATIC1111/stable-diffusion-webui#13830 (comment)

I'm not sure if it's fixed in the a1111 v1.8 Could you check it on your side, please?