search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
26.24k stars 5.41k forks source link

VAE not work,when use StableDiffusionPipeline.from_single_file #4194

Closed zhaoyun0071 closed 1 year ago

zhaoyun0071 commented 1 year ago

Describe the bug

pipe = StableDiffusionPipeline.from_single_file("xx.safetensors",
                                                torch_dtype=torch.float16, load_safety_checker=False, controlnet=False,vae=AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse",torch_dtype=torch.float16).to("cuda")
                                                ).to("cuda")

**above code ,vae not work!! pictures use same seed with or without vae have no difference

but bellow work ,hahhaaaaa**

pipe = StableDiffusionPipeline.from_single_file("xx.safetensors",
                                                torch_dtype=torch.float16, load_safety_checker=False, controlnet=False
                                                ).to("cuda")
pipe.vae=AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse",torch_dtype=torch.float16).to("cuda")

Reproduction

above code

Logs

No response

System Info

diffusers-0.19.0.dev0 6b1abba18dce2eb169b6363ea5f626e7cd87cf21

Who can help?

@patrickvonplaten, @sayakpaul, and @williamberman

patrickvonplaten commented 1 year ago

I have no idea what this issue is supposed to mean

zhaoyun0071 commented 1 year ago

I have no idea what this issue is supposed to mean

pipe = StableDiffusionPipeline.from_single_file("xx.safetensors", torch_dtype=torch.float16, load_safety_checker=False, controlnet=False,vae=AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse",torch_dtype=torch.float16).to("cuda") ).to("cuda") vae in this pipe not load。

sayakpaul commented 1 year ago

Could we have a fully reproducible and self-contained code snippet please?

sayakpaul commented 1 year ago

@zhaoyun0071 I can do the following:

from diffusers import StableDiffusionXLPipeline, AutoencoderKL
import torch

pipe = StableDiffusionXLPipeline.from_single_file(
    "https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/blob/main/sd_xl_base_0.9.safetensors",
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16"
)

vae_2 = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe.vae = vae_2

Isn't this what you're looking for?

zhaoyun0071 commented 1 year ago

@zhaoyun0071 I can do the following:

from diffusers import StableDiffusionXLPipeline, AutoencoderKL
import torch

pipe = StableDiffusionXLPipeline.from_single_file(
    "https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/blob/main/sd_xl_base_0.9.safetensors",
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16"
)

vae_2 = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe.vae = vae_2

Isn't this what you're looking for?

yes this is right

but when vae used as a para of from_single can't work

sayakpaul commented 1 year ago

Could you please provide a reproducible snippet preferably in the format which I followed?

dai-ichiro commented 1 year ago 
wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5_pruned.safetensors
wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5.vae.pt

Method 0 (default VAE)

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    load_safety_checker=False
    )
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method0_result.png")

Method 1

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt"),
    load_safety_checker=False
    )
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method1_result.png")

Method 2

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    load_safety_checker=False
    )
pipe.vae = vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt")
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method2_result.png")

Method 0 and Method 1 have the same results because vae not works. I think method 2 is the right way. But perhaps some people prefer method 1 to method 2.

zhaoyun0071 commented 1 year ago 
wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5_pruned.safetensors
wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5.vae.pt

Method 0 (default VAE)

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    load_safety_checker=False
    )
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method0_result.png")

Method 1

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt"),
    load_safety_checker=False
    )
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method1_result.png")

Method 2

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    load_safety_checker=False
    )
pipe.vae = vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt")
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method2_result.png")

Method 0 and Method 1 have the same results because vae not works. I think method 2 is the right way. But perhaps some people prefer method 1 to method 2.

That's what I mean, why can't developers understand

sayakpaul commented 1 year ago 
from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    load_safety_checker=False
    )
pipe.vae = vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt")
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method2_result.png")

Should be the recommended method as IIUC while using from_single_file() the provided keyword arguments (like vae) are overridden by what's constructed as a part of unpacking the single file checkpoint. Not sure if this needs fixing.

Ccing @patrickvonplaten for further details if I am missing any.

patrickvonplaten commented 1 year ago

@dai-ichiro @zhaoyun0071 thanks for clarifying - I now understand the problem I think. The problem was that passing the vae to from_single_file had no effect. Can you check if https://github.com/huggingface/diffusers/pull/4242 works?

dai-ichiro commented 1 year ago 
pip install torch==2.0.1+cu118 --index-url https://download.pytorch.org/whl/cu118
pip install git+https://github.com/huggingface/diffusers.git@allow_vae_to_be_loaded_from_single_file
pip install accelerate transformers safetensors omegaconf pytorch-lightning
wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5_pruned.safetensors
wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5.vae.pt
from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt"),
    load_safety_checker=False
    )
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method1_result.png")

It works fine. Thanks you.

sayakpaul commented 1 year ago

Closing the issue then.

zhaoyun0071 commented 1 year ago

@dai-ichiro @zhaoyun0071 thanks for clarifying - I now understand the problem I think. The problem was that passing the vae to from_single_file had no effect. Can you check if https://github.com/huggingface/diffusers/pull/4242 works?

great work

andysingal commented 1 year ago

@zhaoyun0071 SDXL 1.0 includes base and refiners... i kept the base vae as default and added the vae in the refiners. I hope that helps