huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
26.47k stars 5.46k forks source link

[SDXL0.9] Strange pixel in generated img. #4035

Closed TinyTigerPan closed 1 year ago

TinyTigerPan commented 1 year ago

Hi, thanks for your work on SDXL. When I use diffuser to inference SDXL, some pixels of the generated image appear strange colors, like the following image.

image

Here is the code I used. I download the SDXL ckpt to load.

from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("/path/to/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
pipe.to("cuda")

prompt = "Disney boy Low saturation Pixar Super details, double ponytails, anime waifu, Serafleur Art Style, divine cinematic edge lighting,soft focus,bokeh,chiaroscuro 8k,best quality ultra-detail 3d,c4d.blender,OCrenderer.cinematic lighting, ultra HD 3D rendering"
print(pipe.unet)
print(pipe.vae)
images = pipe(prompt=prompt, output_type="latent").images

pipe = DiffusionPipeline.from_pretrained("/path/to/stable-diffusion-xl-refiner-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
pipe.to("cuda")
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)

images = pipe(prompt=prompt, image=image).images
images[0].save('test_1024x1024_refiner.png')

and wheter I used refiner or not, the problem still remain.

sayakpaul commented 1 year ago

Hi.

Does this happen without torch.compile()?

Also is this behaviour fairly consistency across different number of inference steps and initial seeds?

TinyTigerPan commented 1 year ago

Thanks for your reply. I have tried different num_inference_steps, and try not use torch.compile(), but the problem still remain. I found when I save image as jpg format, the problem will not appear, because of the compression. I also try to upgrade pillow(PIL), but it seems not the problem of saving picture.

sayakpaul commented 1 year ago

Thanks!

Does this happen for particular types of prompts?

TinyTigerPan commented 1 year ago

I have tried different prompts, all have this problem. The strange pixel only have sepecific color (pink and cyan), and in random position, seems like some special coding or watermark.

n00mkrad commented 1 year ago

This is the "invisible" watermark, which is not invisible.

Disable it using:

class NoWatermark:
    def apply_watermark(self, img):
        return img

...
pipe.watermark = NoWatermark()

...
refiner.watermark = NoWatermark()
TinyTigerPan commented 1 year ago

Thanks a lot, it works. Maybe there are some bugs to make the watermark visible.

sayakpaul commented 1 year ago

Possibly duplicate then: https://github.com/huggingface/diffusers/issues/4014. Let's continue the discussion on that thread. So, closing this one.

AmericanPresidentJimmyCarter commented 1 year ago

This happens even without the watermarker for me. It doesn't seem to happen with Euler scheduler, but does seem to happen with DPMSolverMultistepScheduler. Using VAE tiling and bf16 for unet, VAE, and CLIP.

patrickvonplaten commented 1 year ago

@AmericanPresidentJimmyCarter could you open a new issue with a repro?