search

lllyasviel / ControlNet-v1-1-nightly

Nightly release of ControlNet 1.1
4.48k stars 364 forks source link

LoRA + ControlNet? #42

Closed andreemic closed 1 year ago

andreemic commented 1 year ago

Hey! Do you think it's possible to adapt ControlNet onto a LoRA network? I think it could be really powerful for real-world tasks where a very specific output is desired. So potentially anyone could take a LoRA model from https://civitai.com and append a ControlNet to it.

Let me know if you have any pointers and I'll take it from there.

Thanks!

alitan commented 1 year ago

What do you mean? You can do that already (if i am understanding you correctly). Just add LoRA into your prompts while using ControlNet

andreemic commented 1 year ago

How would you do that? LoRA models are separate small models that are trained separately from SD models and need to be attached at runtime, so as far as I understand you need to write a bit of connector code.

And I'm not even sure if it makes sense conceptually.

lllyasviel commented 1 year ago

This is why some people in research community are clearly lagging behind industry. People have been using LoRAs with ControlNets and Multi-ControlNets for months in industry-oriented A1111 and results are usually many times better than what all reseach papers and old-fashined research labs can achieve.