search

AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
141.5k stars 26.74k forks source link

Support loading models created with diffusers #1266

Closed errnoh closed 2 years ago

errnoh commented 2 years ago

Is your feature request related to a problem? Please describe.

stable-diffusion-webui has been putting the most effort to the user experience of using stable diffusion and is widely adopted. diffusers has a lot of utility and also currently requires the lowest amount of VRAM for things like Dreambooth. Would be nice to lower the gap between these two solutions.

Describe the solution you'd like

I see two good options. Either allowing an easy way to load diffusers models or provide a script to convert diffusers models to ckpt files.

Describe alternatives you've considered

Script to convert between models could be done by either party, diffusers already provides some conversion scripts but mostly towards diffusers format and not for going back to ckpt.

Additional context

There's a similar issue on diffusers side at https://github.com/huggingface/diffusers/issues/468

trufty commented 2 years ago

+1 to having a way to convert diffusers models to .ckpt format. Though I don't expect anyone here to tackle that task.

Having to support two different types of model formats in this code base would be no fun

trufty commented 2 years ago

Looks like we may get a notebook to translate models eventually https://github.com/huggingface/diffusers/issues/672

AmericanPresidentJimmyCarter commented 2 years ago

Been looking for a script to port diffusers weights --> stable-diffusion weights for a while.

errnoh commented 2 years ago

There's now a PR about such a script at https://github.com/huggingface/diffusers/pull/701, if it works as intended I'd consider this feature request solved.

errnoh commented 2 years ago

Marking this as closed since the commit above was merged, which allows converting models into format stable-diffusion-webui accepts.

TeKett commented 4 months ago

Want to reopen this. Since it would be good to be able to load diffusers natively in A1111, while there are alternative programs that do allow diffusers like InvokeAI, it would be good to get all the other things about A1111. Kohya since a couple months back allows to train both diffusers and safetensors. The issue with safetensors is that it lacks the config and has to be loaded from somewhere else, so its incompatible with models that don't run the default clip tokenizer since its using the same config for all models. Now for 99% of people who "just use" this is a non-issue. While you could go and edit that config, its not loaded in plain text, so its a bit tricky. Diffusers are AIO, so it does not have this problem.

I'm not sure why this was closed with a "just convert it", as opposed to just implemented. While you can convert a diffusers into a safetensor, unless that diffusers is "vanilla" you wont be able to load it, since the config does not match the encoder/tokenizer due to how A1111 supplements the config.