[SDXL0.9] Strange pixel in generated img.

[TinyTigerPan]

Hi, thanks for your work on SDXL. When I use diffuser to inference SDXL, some pixels of the generated image appear strange colors, like the following image.

Here is the code I used. I download the SDXL ckpt to load.

from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("/path/to/stable-diffusion-xl-base-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
pipe.to("cuda")

prompt = "Disney boy Low saturation Pixar Super details, double ponytails, anime waifu, Serafleur Art Style, divine cinematic edge lighting,soft focus,bokeh,chiaroscuro 8k,best quality ultra-detail 3d,c4d.blender,OCrenderer.cinematic lighting, ultra HD 3D rendering"
print(pipe.unet)
print(pipe.vae)
images = pipe(prompt=prompt, output_type="latent").images

pipe = DiffusionPipeline.from_pretrained("/path/to/stable-diffusion-xl-refiner-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
pipe.to("cuda")
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)

images = pipe(prompt=prompt, image=image).images
images[0].save('test_1024x1024_refiner.png')

and wheter I used refiner or not, the problem still remain.

[sayakpaul]

Hi.

Does this happen without torch.compile()?

Also is this behaviour fairly consistency across different number of inference steps and initial seeds?

[TinyTigerPan]

Thanks for your reply. I have tried different num_inference_steps, and try not use torch.compile(), but the problem still remain. I found when I save image as jpg format, the problem will not appear, because of the compression. I also try to upgrade pillow(PIL), but it seems not the problem of saving picture.

[sayakpaul]

Thanks!

Does this happen for particular types of prompts?

[TinyTigerPan]

I have tried different prompts, all have this problem. The strange pixel only have sepecific color (pink and cyan), and in random position, seems like some special coding or watermark.

[n00mkrad]

This is the "invisible" watermark, which is not invisible.

Disable it using:

class NoWatermark:
    def apply_watermark(self, img):
        return img

...
pipe.watermark = NoWatermark()

...
refiner.watermark = NoWatermark()

[TinyTigerPan]

Thanks a lot, it works. Maybe there are some bugs to make the watermark visible.

[sayakpaul]

Possibly duplicate then: https://github.com/huggingface/diffusers/issues/4014. Let's continue the discussion on that thread. So, closing this one.

[AmericanPresidentJimmyCarter]

This happens even without the watermarker for me. It doesn't seem to happen with Euler scheduler, but does seem to happen with DPMSolverMultistepScheduler. Using VAE tiling and bf16 for unet, VAE, and CLIP.

[patrickvonplaten]

@AmericanPresidentJimmyCarter could you open a new issue with a repro?