search

lllyasviel / ControlNet

Let us control diffusion models!
Apache License 2.0
29.77k stars 2.69k forks source link

How to use controlNet with pretrained LoRA? #205

Open zengqixun12 opened 1 year ago

zengqixun12 commented 1 year ago

I want to combine the pretrained LORA_model and pretrained controlnet_mdoel like "[control_sd15_depth.pth]",and export new model.

How to realize this function?

JamesSand commented 1 year ago

I have the same problem with you, is there any solution?

zengqixun12 commented 1 year ago

I have the same problem with you, is there any solution?

I've solved this problem, and I combined Lora + Train + ControlNet + change basemodel, export the final result and the model without Stable-Diffusion-WebUI (only use the command line)

zengqixun12 commented 1 year ago

I have the same problem with you, is there any solution?

I will release the code in few days ;)

JamesSand commented 1 year ago

I have the same problem with you, is there any solution?

I will release the code in few days ;)

thanks a lot. I am also trying to solve this issue. May I ask should we apply lora on the trainable copy of controlnet?

zengqixun12 commented 1 year ago

Not exactly, Lora is only effective on the main network and is independent from ControlNet

SnowdenLee commented 1 year ago

Hi @zengqixun12, may I ask do you still plan to release the code? Thank you :)

gwyong commented 11 months ago

@zengqixun12 Hello, do you still have plan to release the code? Thanks,

jiachen0212 commented 2 months ago

I have the same problem with you, is there any solution?

I will release the code in few days ;)

thanks a lot. I am also trying to solve this issue. May I ask should we apply lora on the trainable copy of controlnet?

You can try: first merge the SD model of base and the parameters of lora, similar to the merger of con parameters and bn parameters. Then train your own data according to the document of train.md. I followed this plan and it worked.