search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
26.08k stars 5.37k forks source link

Support for control lora #4679

Open adhikjoshi opened 1 year ago

adhikjoshi commented 1 year ago

There's new control lora release,

https://huggingface.co/stabilityai/control-lora?s=09

How can we use it with diffusers?

sayakpaul commented 1 year ago

You cannot yet.

haofanwang commented 1 year ago

Any timeline for supporting it?

MaxTran96 commented 1 year ago

any timeline for supporting this?

yanchaoguo commented 1 year ago

+1

patrickvonplaten commented 1 year ago

Working on it!

patrickvonplaten commented 1 year ago

@sayakpaul do you have bandwitdh for it? Otherwise I can look into picking it up :-)

HighCWu commented 1 year ago

I implemented a control-lora training code based on diffusers' controlnet. The idea is basically the same as stabilityai's. I found diffusers clear the lora layer of attention layers when xformers is enabled, so I did a little hack. At present, I have only tested the training and inference on sd v1.5. In theory, it can be compatible with sdxl's control lora by making some modifies. Here is my repo HighCWu/control-lora-v2.

patrickvonplaten commented 1 year ago

It's being worked on: https://github.com/huggingface/diffusers/pull/4899

HighCWu commented 1 year ago

It's being worked on: #4899

Great work.

yanchaoguo commented 1 year ago

It's being worked on: #4899

Please make it merge into the main branch as soon as possible

sayakpaul commented 1 year ago

@HighCWu thanks for bringing your implementation to our attention. From what I can see from here, we're doing exactly the same in https://github.com/huggingface/diffusers/pull/4899. If you have any comments there let me know :)

yanchaoguo commented 1 year ago

It's being worked on: #4899

No effect! @sayakpaul @patrickvonplaten @HighCWu please，when this problem can be solved: 【Results don't seem to be following the conditioning image at all. I will play with the hyperparameters a bit to see how the results get affected】

MaxTran96 commented 1 year ago

Any eta on when this will be done?

patrickvonplaten commented 12 months ago

@sayakpaul do you want to give this another try or open it for the community?

sayakpaul commented 11 months ago

@patrickvonplaten https://github.com/huggingface/diffusers/issues/5800

github-actions[bot] commented 10 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

alexblattner commented 3 weeks ago

Bump

alexblattner commented 4 days ago

bump