[bug] Train a personalized SD1.5 model by DreamBooth Lora. Execute stage 1, but there is no personalized effect

[xdobetter]

Great job. my code backbone：threestudio-dreamwaltz I want to try civitai model, as you mainly mentioned make personalized avatar by load lora weight, but I found it doesn't work,

# guidance/multi_controlnet_guidance.py
# when I download or train a lora weight, my calling method refers to diffusers method as shown below
self.pipe = StableDiffusionControlNetPipeline.from_pretrained(
            self.cfg.pretrained_model_name_or_path,
            **pipe_kwargs,
        ).to(self.device)

# after
self.pipe.load_lora_weights(self.cfg.lora_weights_path",weight_name="pytorch_lora_weights.safetensors")

To verify whether lora is effective, you can test it through the following code

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import numpy as np
import torch

import cv2
from PIL import Image

# download an image
image = load_image(
    "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
)
image = np.array(image)

# get canny image
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

# load control net and stable diffusion v1-5
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)
pipe.load_lora_weights("pretrained_models/s15_girl_character_lora_v3_c1500",weight_name="pytorch_lora_weights.safetensors")
# speed up diffusion process with faster scheduler and memory optimization
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# remove following line if xformers is not installed
pipe.enable_xformers_memory_efficient_attention()

pipe.enable_model_cpu_offload()

# generate image
generator = torch.manual_seed(0)
image = pipe(
    "a zoomed out DSLR photo of sks female anime character", num_inference_steps=200, generator=generator, image=canny_image
).images[0]
image.save("infer_test_lora.png")

The result is as follows, it is valid

pytorch_lora_weights.zip

By the way, I found it works when I didn't use lora finetune sd, but full finetune sd。