search

HighCWu / control-lora-v2

ControlLoRA Version 2: A Lightweight Neural Network To Control Stable Diffusion Spatial Information Version 2
Apache License 2.0
98 stars 6 forks source link

Is there a way to use a pre-trained controlnet? #2

Open alexanderswerdlow opened 9 months ago

alexanderswerdlow commented 9 months ago

Instead of loading from the base U-Net, can we perform LoRA on an already trained controlnet?

HighCWu commented 9 months ago

I think the diffusers library itself already contains this support. You only need to use diffusers to load the controlnet model, and then train the lora of controlnet by referring to the method of training unet's lora in diffusers.

venkateshtata commented 3 months ago

@HighCWu if I am right the implementation in diffusers library is plugging the LoRA to UNet of the StableDiffusionControlNetPipeline and not the ControlNet's attention blocks. Is there any way we can use the pre-trained ControlNet that understands say "Canny" or "HED" conditions and plug the LoRA on top of one such pre-trained model ?

HighCWu commented 3 months ago

@HighCWu if I am right the implementation in diffusers library is plugging the LoRA to UNet of the StableDiffusionControlNetPipeline and not the ControlNet's attention blocks. Is there any way we can use the pre-trained ControlNet that understands say "Canny" or "HED" conditions and plug the LoRA on top of one such pre-trained model ?

Now the official t2i lora sample is unet.add_adapter. I think, for controlnet, it should be done with controlnet.add_adapter

venkateshtata commented 3 months ago

It's throwing AttributeError: 'ControlNetModel' object has no attribute 'add_adapter' error.

HighCWu commented 3 months ago

It's throwing AttributeError: 'ControlNetModel' object has no attribute 'add_adapter' error.

@venkateshtata check my new added file models/controlnet.py, import it. it may work. I wrote some sample code in it, you can check the source code