ControlLoRA Version 3 is a neural network structure extended from ControlNet to control diffusion models by adding extra conditions.
Inspired by control-lora (StabilityAI), ControlLoRA, control-lora-v2 and script train_controlnet.py from diffusers, control-lora-v3 does not add new features, but provides a PEFT implement of ControlLoRA.
To train ControlLoRA, you should have image-conditioning_image-text datasets. Of course you can hardly train on LAION-5B dataset in direct like Stable Diffusion. Here are some:
Stable Diffusion v1-5 is the base model.
Stable Diffusion v1-4, Stable Diffusion v2-1, Stable Diffusion XL need to be vertified.
You can train either ControlNet or ControlLoRA using script train_control_lora.py.
By observation, training 50000 steps with batch size of 4 is the balance between image quality, control ability and time.
accelerate launch train_control_lora.py \
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
--output_dir="controlnet-model" \
--dataset_name="fusing/fill50k" \
--resolution=512 \
--learning_rate=1e-5 \
--train_batch_size=4 \
--max_train_steps=100000 \
--tracker_project_name="controlnet" \
--checkpointing_steps=5000 \
--validation_steps=5000 \
--report_to wandb
To train ControlLoRA, add --use_lora
in start command to activate it.
accelerate launch train_control_lora.py \
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
--output_dir="control-lora-model" \
--dataset_name="fusing/fill50k" \
--resolution=512 \
--learning_rate=1e-4 \
--train_batch_size=4 \
--max_train_steps=100000 \
--tracker_project_name="control-lora" \
--checkpointing_steps=5000 \
--validation_steps=5000 \
--report_to wandb \
--use_lora \
--lora_r=32 \
--lora_bias="all"
You can also train ControlLoRA / ControlNet with your own dataset.
accelerate launch train_control_lora.py \
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
--output_dir="control-lora-model" \
--conditioning_image_column="hint" \
--image_column="jpg" \
--caption_column="txt" \
--resolution=512 \
--learning_rate=1e-4 \
--train_batch_size=4 \
--num_train_epochs=3 \
--max_train_steps=100000 \
--tracker_project_name="control-lora" \
--checkpointing_steps=5000 \
--validation_steps=5000 \
--report_to wandb \
--use_lora \
--lora_r=32 \
--lora_bias="all" \
--custom_dataset="custom_datasets.tutorial.MyDataset"
If you want to merge ControlLoRA to ControlNet, use merge_lora.py script.
python merge_lora.py --base_model runwayml/stable-diffusion-v1-5 --control_lora /path/to/control-lora --output_dir /path/to/save/ControlNet
Now you can convert ControlLoRA weight from HuggingFace diffusers type to Stable Diffusion type. The converted model can be used in AUTOMATIC1111's Stable Diffusion web UI and ComfyUI.
PS: ControlLoRA should set --lora_bias="all"
in training script.
python convert_diffusers.py --adapter_model /path/to/adapter/model --output_model /path/to/output/model
Original image:
Output:
@software{lavinal7122024controllorav3,
author = {lavinal712},
month = {5},
title = {{ControlLoRA Version 3: A Lightweight Neural Network To Control Stable Diffusion Spatial Information Version 3}},
url = {https://github.com/lavinal712/control-lora-v3},
version = {1.0.0},
year = {2024}
}