huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
26.05k stars 5.36k forks source link

FluxInpaintPipeline overrides pixels outside the mask #9867

Open Clement-Lelievre opened 12 hours ago

Clement-Lelievre commented 12 hours ago

Describe the bug

When inpainting (with diffusers==0.31.0 and torch==2.4.1) using FluxInpaintPipeline, I get some pixels outside the mask (and pretty far away from the mask border) that are overrided whereas the mask at their indices was black. With either flux schnell or dev.

Is this expected?

I could paste the output image on the input image using the mask to get rid of this, but I'd like to check first if this is a bug or not.

Here is my input image and mask:

vampire_input_rgb vampire_mask

Here is the kind of diff I get:

diff

Reproduction

I am using a small, square (224*224) image (see above), and a non-rectangular mask that has values only in {0, 255}.

from diffusers import FluxInpaintPipeline

pipe = FluxInpaintPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16, device='cuda')
prompt = "skull"
image= # above, convert it to RGB
mask = # above,  convert it to RGB
image = pipe(prompt=prompt, image=source, mask_image=mask, height=224, width=224, num_inference_steps=2, guidance_scale=3.5, strength=0.76).images[0]  # the latter params are probably not necessary to reproduce the issue
image.save("flux_inpainting.png")

# now visualize the pixel diff between the output you get and the input image, you should see non-zero pixels outside the mask, particularly on the right part

Logs

No response

System Info

Who can help?

@asomoza @sayakpaul

sayakpaul commented 12 hours ago

PyTorch 3.2.0? That sounds like invalid.

Clement-Lelievre commented 11 hours ago

PyTorch 3.2.0? That sounds like invalid.

torch==2.4.1

yiyixuxu commented 5 hours ago

cc @asomoza