search

AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
131.71k stars 25.28k forks source link

[Feature Request]: Please add support for the COSXL model by Stability AI #15598

Open CCpt5 opened 4 weeks ago

CCpt5 commented 4 weeks ago

Is there an existing issue for this?

What would your feature do ?

Stability AI recently recent an alternate version of the SDXL base model called COSXL. This mode can be found here:

https://huggingface.co/stabilityai/cosxl

Description: Cos Stable Diffusion XL 1.0 and Cos Stable Diffusion XL 1.0 Edit

Cos Stable Diffusion XL 1.0 Base is tuned to use a Cosine-Continuous EDM VPred schedule. The most notable feature of this schedule change is its capacity to produce the full color range from pitch black to pure white, alongside more subtle improvements to the model's rate-of-change to images across each step.

It's also possible to merge the COSXL model with normal SDXL models as outlined at the bottom of this ComfyUI page: https://comfyanonymous.github.io/ComfyUI_examples/model_merging/

As results there are more than just 2 models that would benefit from support for changes related to COSXL.

A PR to support COSXL in the Forge fork of A1111 was submitted today, although it doesn't appear the file modified is one used by the original code here. Link: https://github.com/lllyasviel/stable-diffusion-webui-forge/pull/686

Proposed workflow

  1. Load COSXL / COSXL Edit / or a model merged with COSXL
  2. Proper loading of model and VAE (if automatic/needed) 3.Proper generation (currently solid red or garbage colors are what are generated).

Additional information

Thanks for your efforts on this tool. It has been and still is amazing.

mcmonkey4eva commented 4 weeks ago

Can be detected via the edm_vpred keys. CosXL should work as-is with just the specific Cos-V-EDM schedule applied, Edit should be able to work via the existing ip2p support. You can detect the edit model via modelspec.architecture metadata key set to stable-diffusion-xl-v1-edit (per ModelSpec) alongside the edm_vpred keys signalling the preferred schedule.