search

tencent-ailab / IP-Adapter

The image prompt adapter is designed to enable a pretrained text-to-image diffusion model to generate images with image prompt.
Apache License 2.0
5.08k stars 331 forks source link

ip-adapter-plus-face_sdxl cannot work with controlnet? #339

Closed tanghengjian closed 5 months ago

tanghengjian commented 5 months ago 
base_model_path = 'models/RealVisXL_V4.0_Lightning.safetensors'
image_encoder_path = "models/h94/IP-Adapter/models/image_encoder"
ip_ckpt_face = "models/h94/IP-Adapter/sdxl_models/ip-adapter-plus-face_sdxl_vit-h.bin" # a experimental version
device = "cuda"

# load controlnet
#controlnet_model_path = "lllyasviel/control_v11f1p_sd15_depth"
controlnet_depth_path = f'diffusers/controlnet-depth-sdxl-1.0-small'
controlnet_openpose_path = f'thibaud/controlnet-openpose-sdxl-1.0'

controlnet = ControlNetModel.from_pretrained(controlnet_depth_path, torch_dtype=torch.float16)
# load SD pipeline
pipe = StableDiffusionXLControlNetPipeline.from_single_file(
    base_model_path,
    controlnet=controlnet,
    use_safetensors=True,
    torch_dtype=torch.float16,
    add_watermarker=False,
)

# load ip-adapter

ip_model = IPAdapterPlusXL(pipe, image_encoder_path, ip_ckpt_face, device)
ip_model = IPAdapterPlusXL(pipe, image_encoder_path, ip_ckpt_face, device)

File "/IP-Adapter/ip_adapter/ip_adapter.py", line 84, in init self.load_ip_adapter() File "IP-Adapter/ip_adapter/ip_adapter.py", line 139, in load_ip_adapter self.image_proj_model.load_state_dict(state_dict["image_proj"]) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Resampler: size mismatch for latents: copying a param with shape torch.Size([1, 16, 1280]) from checkpoint, the shape in current model is torch.Size([1, 4, 1280]).

tanghengjian commented 5 months ago

IPAdapterPlusXL(pipe, image_encoder_path, ip_ckpt_face, device, num_tokens=16）