Any plans to work on SDXL?

[duskfallcrew]

I realize that SDXL is extremely new, and i'm aware you probably don't have a ton of time, but this is like my GO-To plugin, and making TI's out of prompts doesn't replace trained ones - but it enhances things in ways that sometimes trained TI's to me feel that would be more static if you get what i mean?

I'm happy to wait until the post apocalypse happens for this to update :P

[aleksusklim]

Do you have any ideas how UI should look like, and what features should be available for SDXL?

You see, SDXL has two CLIP models, so it would have two independent embedding spaces (one from SD1 and another from SD2, actually).

How to show this in the table? How to make calculations at runtime? How to target only the specific CLIP? (From the user's point of view)

We cannot target both CLIPs simultaneously, because a see a plenty of cases when the user would want to change only one CLIP – at least to test them both separately (what each CLIP adds to the prompt).

But how to express that in concise syntax? I'm open to discussions.

How to actually train TI embedding from SDXL ? How .pt file of embedding for SDXL looks like? Can anybody show an example?

[aleksusklim]

Also, ComfyUI allows to set different prompts for these two CLIPs, but WebUI does not.

If WebUI would have an option to separate prompts, we could use different embeddings on they own. Up to the point when they would overlap at token level. But with unified prompt, different embeddings cannot overlap (since this should technically be the same embedding with two named tensors in it. Or am I wrong?)

Does Automatic1111 have plans to separate SDXL prompts? (I'm not talking about Refiner, that's another can of worms…)

[duskfallcrew]

I don't use refiner, I don't however know if TI's are getting easy trained for SDXL yet. However this is stuff i don't quite understand yet - but in my quest to help because I ADORE THIS EXTENSION - i'll try and do some research, i don't know how to program but at least i can see if i can find solutions!

On Fri, Sep 15, 2023 at 5:45 PM Kly_Men_COmpany @.***> wrote:

Also, ComfyUI https://github.com/comfyanonymous/ComfyUI allows to set different prompts for these two CLIPs, but WebUI does not.

If WebUI would have an option to separate prompts, we could use different embeddings on they own. Up to the point when they would overlap at token level. But with unified prompt, different embeddings cannot overlap (since this should technically be the same embedding with two named tensors in it. Or am I wrong?)

Does Automatic1111 have plans to separate SDXL prompts? (I'm not talking about Refiner, that's another can of worms…)

— Reply to this email directly, view it on GitHub https://github.com/klimaleksus/stable-diffusion-webui-embedding-merge/issues/7#issuecomment-1720710147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOBTJ66EFZBMK3KRAF7326LX2PTP7ANCNFSM6AAAAAA4XHK5HA . You are receiving this because you authored the thread.Message ID: <klimaleksus/stable-diffusion-webui-embedding-merge/issues/7/1720710147@ github.com>

-- Kieran Somerville Artist / Indie Music / Designer +64 22 678 8131 @.*** https://www.kieransomerville.com "Do not print this email, a tree will eat you for breakfast."

[duskfallcrew]

Also, ComfyUI allows to set different prompts for these two CLIPs, but WebUI does not.

If WebUI would have an option to separate prompts, we could use different embeddings on they own. Up to the point when they would overlap at token level. But with unified prompt, different embeddings cannot overlap (since this should technically be the same embedding with two named tensors in it. Or am I wrong?)

Does Automatic1111 have plans to separate SDXL prompts? (I'm not talking about Refiner, that's another can of worms…)

Just updating here - Most people by now don't bother with the refiner.. MOST I don't know how to program (i duct tape lol) -- but i noticed something funny.. when i accidentally leave the TI's in from doing 1.5 testing - and then i go in and swap to SDXL...

Sometimes -- just sometimes. Especially positive prompters? They work lol.

I'll see what i can poke around at for research to guide ahead on this, like i said i don't program but i'm happy to poke around :)

[aleksusklim]

Can you find any good SDXL trained embedding? Looks like training is already possible: https://github.com/kohya-ss/sd-scripts/blob/main/README.md#format-of-textual-inversion-embeddings-for-sdxl

I want to be sure WebUI correctly reads SDXL embeddings, so I need one.

[duskfallcrew]

Yea this one here: https://civitai.com/models/119032/unaestheticxl-or-negative-ti It's a trained one.

Most of what i'd love to use it for is positive, so lemme find you some positive embeds for SDXL https://civitai.com/models/139230/crystalitixl?modelVersionId=158700 https://civitai.com/models/118758/simplepositivexl?modelVersionId=182974 https://civitai.com/models/195224/arcpastel?modelVersionId=257061

Hope this helps :)

[duskfallcrew]

FUUU hold on LOL I had a comment but i closed the tab - TWO MINTUES HERE! LOL

[duskfallcrew]

https://civitai.com/models/118758/simplepositivexl?modelVersionId=182974 https://civitai.com/models/195224/arcpastel?modelVersionId=257061 https://civitai.com/models/139230/crystalitixl?modelVersionId=158700 https://civitai.com/models/119032/unaestheticxl-or-negative-ti?modelVersionId=245812 https://civitai.com/models/144327/negative-embedding-for-sdxl-based-anime-models?modelVersionId=195614

I MEAN THERES TON at civit for SDXL ALREADY! (Well not a ton but a fair whack)

https://civitai.com/models/155817/dynodawn?modelVersionId=174820

Those are more the pos end i didn't really care to send too many negs, as negatives do affect how a model looks MORE SO than SD 1.5

... SD 1.5 negatives are WAY MORE important. I'm looking to do more my lazy text concepts for SDXL :)

[duskfallcrew]

OOH I DID COMMENT OOPS LMAO GUESS WHAT you got extra!!

[chromesun]

Hi aleksusklim, I’ve just stumbled across your EM extension. Since SDXL came out I haven’t touched SD1x, but your readme is making me think I need to re-visit it just to try out EM! Of course I’d rather have an EMXL extension :-)

Anything that could make the TI process faster and/or more understandable would be ++good for me. My focus is on making TIs rather than LoRAs. I’m using kohya-ss to make SDXL TIs. So far it only allows use of CLIP1, and since I use a1111 webui that’s all I need for now. Each TI takes about 1hr15m per 1000 steps to cook in koyha on my PC - frustrating.

If you want some SDXL TIs for examination/testing, I’ve published a few on CivitAI. I see duskfallcrew mentioned one of them - CrystalTIXL.

https://civitai.com/models/139230/crystalitixl https://civitai.com/models/154898/marblingtixl https://civitai.com/models/243600/dark-world-dreams

They are all style TIs. I have not tried to make a TI for negative prompts. Yet. The marblingtixl TI works well with most checkpoints... and combines well with most of the LoRAs I’ve tried. I’m using a1111 v1.7.0. The TI uses 8 tokens.

If you haven’t seen it already, this might be of interest - SDXL should be able to use SD1x embeddings?: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/14287

[aleksusklim]

Can anyone confirm or confute that clip_g of SDXL is the same as in SD2 ? I confirmed that clip_l is strictly the same as in SD1, but why I have different results for OpenCLIP of SDXL? Was it fine-tuned officially, or what?

This question means, "can we use embeddings from SD2 for SDXL?" (while the answer for SD1 is true, CLIP was not modified in SDXL compared to SD1)

[aleksusklim]

Basic SDXL support is live on sdxl branch (main README explains how to enable it), please try!

Everything should work. Nothing for SD1/SD2 should break. If anything gives different results for SD1 model on my sdxl branch – it is a bug that must be fixed; let me know if you encounter something like that.

When SDXL model is loaded and you inspect vectors in EM tab – you will see two tables for two CLIPs.

As for merge expressions: two sets of vectors are processed jointly. Currently you cannot target only L or G vectors yet; only them both at once.

[duskfallcrew]

Tested it today, and as always LOL it's not the extension that's weird it's SDXL itself - so yes 100% it works, just trying to figure out how to make certain wordsets FIT in a prompt on anime models is always fun XD