search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
26.33k stars 5.42k forks source link

SD3 inpaint pipeline can not use EulerA scheduler. bug: The `scale_model_input` function should be called before `step` to ensure correct denoising. #9084

Open huangjun12 opened 3 months ago

huangjun12 commented 3 months ago

Describe the bug

SD3 inpaint pipeline can not use EulerA scheduler.

Reproduction

pipe = StableDiffusion3InpaintPipeline.from_pretrained(xxx)
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)

# then run the pipe
  out_images = pipe(
      prompt=caption, 
      image=input_image, 
      mask_image=input_mask, ).images

Logs

The `scale_model_input` function should be called before `step` to ensure correct denoising.  See `StableDiffusionPipeline` for a usage example.

System Info

torch: 2.0.0 + cu117 diffusers: commit_id: 41b705f42db6ce43bbcb41bd2cefd479d8b73774

Who can help?

No response

a-r-r-o-w commented 3 months ago

cc @yiyixuxu There seems to be a line missing after preparing latent_model_input:

https://github.com/huggingface/diffusers/blob/b7058d142c162bc33f1f47f7909e0913ead3d248/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py#L873

latent_model_input = self.scheduler.scale_model_input(latent_model_input, t)

Is this intentionally not present, or was it forgotten? If forgotten, would be awesome if you could open a PR @huangjun12!

yiyixuxu commented 3 months ago

our flow match pipelines are meant to work with flow match schedulers, though

huangjun12 commented 3 months ago

@yiyixuxu Will SD3_inpaint_pipeline support the EulerA scheduler? If Yes, when will it be supported?

huangjun12 commented 3 months ago

cc @yiyixuxu There seems to be a line missing after preparing latent_model_input:

https://github.com/huggingface/diffusers/blob/b7058d142c162bc33f1f47f7909e0913ead3d248/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py#L873

latent_model_input = self.scheduler.scale_model_input(latent_model_input, t)

Is this intentionally not present, or was it forgotten? If forgotten, would be awesome if you could open a PR @huangjun12!

I tried this, but it doesn't seem to work.

sauravp commented 3 months ago

@huangjun12 I want to reproduce this. What pre-trained model are you using?

pipe = StableDiffusion3InpaintPipeline.from_pretrained(xxx)

huangjun12 commented 3 months ago

@huangjun12 I want to reproduce this. What pre-trained model are you using?

pipe = StableDiffusion3InpaintPipeline.from_pretrained(xxx)

I trained this myself

github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.