search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
25.41k stars 5.27k forks source link

Allow configuring `shift=` for SD3 dynamically #8501

Open isidentical opened 3 months ago

isidentical commented 3 months ago

Is your feature request related to a problem? Please describe. Allow passing shift= per inference call (like timesteps) on the pipeline, for flow matching scheduler, or allow set_shift() etc. on the scheduler. This seems to be the key to getting good results with SD3 https://x.com/bfitzgerald242/status/1801018438120341911

asomoza commented 3 months ago

Hi, you can do it like this:

from diffusers import  FlowMatchEulerDiscreteScheduler

pipe.scheduler = FlowMatchEulerDiscreteScheduler.from_config(pipe.scheduler.config, shift=3.0)
isidentical commented 3 months ago

yep! but the same format is applicable for timesteps and was wondering if we can get around without re-instating the scheduler again and again?

asomoza commented 3 months ago

Not for the moment, but I can see the potential in adding it as an argument if people change it a lot for each inference.

In my experience it didn't fix the anatomy problems and sometimes it made the quality worse but I tested it with the T5, still need to test it without it and do some more generations.

Can you share some examples where changing the shift helped with the generation? That would help a lot.

bghira commented 3 months ago

the anatomy problems are from DPO safety alignment. it will require training to fix.

github-actions[bot] commented 3 weeks ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.