Great job.
my code backbone:threestudio-dreamwaltz
I want to try civitai model, as you mainly mentioned make personalized avatar by load lora weight, but I found it doesn't work,
# guidance/multi_controlnet_guidance.py
# when I download or train a lora weight, my calling method refers to diffusers method as shown below
self.pipe = StableDiffusionControlNetPipeline.from_pretrained(
self.cfg.pretrained_model_name_or_path,
**pipe_kwargs,
).to(self.device)
# after
self.pipe.load_lora_weights(self.cfg.lora_weights_path",weight_name="pytorch_lora_weights.safetensors")
To verify whether lora is effective, you can test it through the following code
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import numpy as np
import torch
import cv2
from PIL import Image
# download an image
image = load_image(
"https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
)
image = np.array(image)
# get canny image
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)
# load control net and stable diffusion v1-5
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)
pipe.load_lora_weights("pretrained_models/s15_girl_character_lora_v3_c1500",weight_name="pytorch_lora_weights.safetensors")
# speed up diffusion process with faster scheduler and memory optimization
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# remove following line if xformers is not installed
pipe.enable_xformers_memory_efficient_attention()
pipe.enable_model_cpu_offload()
# generate image
generator = torch.manual_seed(0)
image = pipe(
"a zoomed out DSLR photo of sks female anime character", num_inference_steps=200, generator=generator, image=canny_image
).images[0]
image.save("infer_test_lora.png")
Great job. my code backbone:threestudio-dreamwaltz I want to try civitai model, as you mainly mentioned make personalized avatar by load lora weight, but I found it doesn't work,
To verify whether lora is effective, you can test it through the following code
The result is as follows, it is valid
pytorch_lora_weights.zip
By the way, I found it works when I didn't use lora finetune sd, but full finetune sd。