[Feature]: Pipeline-Specific VAE (and/or other?)

[midcoastal]

Feature description

I have a large collection of models, and bounce back and forth from SD1.5/2 and SDXL quite a bit. This results in the selected VAE being wrong often (and Automatic doesn't seem to be working in my setup [which consists of each model having it's own sub-directory, including the VAEs, which may break Automatic selection of the correct VAE, which probably looks in the same dir as the model, and not sibling directories, or directories at all, which is not a setup I can sustain at this time, each and every model must have it's own dir, for reasons]). Regardless, I don't uee Auto, and would like to specif the VAE, but would like to specify it for SDXL and standard SD separately.

Doable?

Version Platform Description

No response

[vladmandic]

thats not how automatic works. simplest way is to name your vae identical to model name and place it in vae folder.

[midcoastal]

thats not how automatic works. simplest way is to name your vae identical to model name and place it in vae folder.

Um... Whelp, ok, so I "don't use automatic," and I'll modify that to "never," as I have...

Extra networks: page='model' items=4541 subdirs=4877 tab=txt2img dirs=['models/Stable-diffusion', 'models/Diffusers', '/home/user/Development/webui/models/Stable-diffusion'] time=51.22s

(BTW, prior optimizations you did are top notch, danke)

So, per-pipeline?

[vladmandic]

i don't want to introduce per-pipeline vae. if there is some way we can enhance auto to fit your use case, i would consider that.

[midcoastal]

i don't want to introduce per-pipeline vae. if there is some way we can enhance auto to fit your use case, i would consider that.

So, on one hand I do understand. On the other: We have the ability to select None, Automatic, and a specific VAE. Since VAEs are, by their nature, Pipeline-specific, and Settings can be locked while there is a front-end to change the Model (and thus Pipeline), it would stand to reason that each potential pipeline would have it's own selection.

The hand that understands hopes that "I don't want to introduce" means "I don't have the time or desire to implement at this moment," and is open to community contributions should they come? Yeah?

[vladmandic]

there are quite a few different pipelines that may have its own vae, so how many different text field settings we would have? 1? 5? 10? i want to remain on single and improve it in any way possible without introducing 9 others.

[midcoastal]

there are quite a few different pipelines that may have its own vae, so how many different text field settings we would have? 1? 5? 10? i want to remain on single and improve it in any way possible without introducing 9 others.

Noted. My (initial) thought was: at model collection load (either base models, refiners, or vaes) there must be some indicator of pipeline type that we get ro have access to. The "number" of setting dropdowns wouldn't be fixed, but would be added at that time in the format of something like pipeline_vae_{pipeline}. So we don't have a dropdown for any pipelines we will never trigger.

Unfortunately, with a single dropdown, there isn't really a better way.

THAT SAID:

Another option could possibly be a list like the Quick Settings list. VAE priority order. Load X if it is compatible, or Y, or Automatic, or Z, or None.

That would allow, say: Automatic, Default for SD1.5/2, XL, others, in that order. That would also account for the possibility of some new Pipeline that somehow is able to use another Pipeline's VAEs (it's the wild west out here, who knows), as the system will use the first that is compatible.

Potential?

[midcoastal]

The "per pipeline" thing does still have tiny little legs, though. For instance, I would like to use different precision settings for XL and non-XL (VRAM limit), and I like one sampler better for SDXL than non-XL. That kind of thing.

This could also be handled like Styles maybe? Meh. Just spewing now.

[vladmandic]

styles can apply to gen settings, but not server settings as that would open pandora's box. i understand what you'd want, but in that case configs really start diverging a lot and why not have separate configs and choose which to use using webui --config filename.json when starting a server.

[midcoastal]

styles can apply to gen settings, but not server settings as that would open pandora's box. i understand what you'd want, but in that case configs really start diverging a lot and why not have separate configs and choose which to use using webui --config filename.json when starting a server.

Because starting servers takes a while for me, and if I start in a configuration that "supports" one pipeline, I can still load models from another pipeline and bugger the works and make it a moot point. It would be faster to just go in to config and change the VAE, or to make the VAE a quick setting, which is also fine. However, ti still stands that that makes several things to switch when switching pipeline type, and there isn't obvious feedback when a pipeline is switched (some models indicate as such in the name, but not all).

It also still stands that this is a setting that is a "global" that implies it is a set once, and yet, it is variable based on the pipeline, which can be changed on a whim in Extra Networks. Maybe VAEs should be in Extra Networks? That seems too much (particularly if you are following the model of name the VAE the same as the model).

[midcoastal]

As an aside: I am surprised an error is not raised in "Setting model VAE" when using incompatible VAE/Model combinations. Are there not functional conflicts? Or is it an enigmatic issue of "just not optimized for?"

In the end: you can backburner this if you would like. I will look in to it if I get a chance.

[vladmandic]

actually, no - pretty much all latent models work in same latent space: 64x64x4, so there are no glaring incompatibilities that can be looked at. but then mappings of that latent space can be different between models. and expected output is also same - its pixel space. but just for example, sd15 would use RGB and sdxl uses BGR.

[midcoastal]

actually, no - pretty much all latent models work in same latent space: 64x64x4, so there are no glaring incompatibilities that can be looked at. but then mappings of that latent space can be different between models. and expected output is also same - its pixel space. but just for example, sd15 would use RGB and sdxl uses BGR.

Well then that stinks, because it would really have to be Pipeline-specific... :-( I'll have a think on it.