search

AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
135.55k stars 25.88k forks source link

Use refiner with SD1.5 models #13063

Open df2df opened 10 months ago

df2df commented 10 months ago

Is there an existing issue for this?

What happened?

Set VAE to Auto and sometimes it would work correctly but on a 2nd try after it would generate fried images similar to incorrect VAE selection which I think is what's happening. Was testing it while using Ultimate upscale and sometimes 1 tile would be fried while others were good. Subsequent tries led to all tiles being scrambled and continues to get worse.

May be related to this issue which is something to do with XL models somehow polluting the memory even after unloading and loading new checkpoints.

Settings: no caching of VAE, 2 checkpoints cached with 1 one on device selected

Steps to reproduce the problem

  1. Select SD1.5 model, enable refiner in tab and select XL base refiner
  2. Set to Auto VAE option
  3. Generate normally or with Ultimate upscale

What should have happened?

Uncorrupted images

Sysinfo

Linux Ubuntu 22.04 6800XT 32GB ram

What browsers do you use to access the UI ?

No response

Console logs

No errors

Additional information

No response

w-e-w commented 10 months ago

you're not supposed to be able to mix SD1.5 with XL refine the latent space of SD1.5 and SDXL are not the same

w-e-w commented 10 months ago

someone seems to be interested in making this possible but it's not working natively it's not bug so I reopened and labeled this as enhancement

BlandiZ commented 8 months ago

in original Fooocus latest release from Ilia, Fooocus_win64_2-1-754 based on CUDA 12 and Pytorch2.1, there is a possibility to use SDXL as base and SD1.5 as refiner so the opposite should be possible as well (technically)? not in fooocus thou but in 1111 or ComfyUI?

logicmoon commented 5 months ago

In ComfyUI this is also possible through workflows or extensions (see links). It would be great to have this feature in A1111. Until supported, we will have to manually use img2img I guess, but thank you for looking into this.

w-e-w commented 5 months ago

I knew about SD-Latent-Interposer https://github.com/w-e-w/sd-webui-hires-fix-tweaks/issues/3 but implementing it in webview is not a straightforward

SmirkingKitsune commented 2 months ago

I think what you are describing is what the sd-webui-refiner extension does. It seems to have a large VRAM requirement, so it is not suitable for low VRAM systems (<12 GB VRAM), but since you have an RX 6800XT (16 GB VRAM), it should work for you.