[Bug]: - Githubissues

hugo4711 commented 5 months ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

When using ip_adapter_face_id or ip_adapter_face_id_plus with ip-adapter-plus-face_sdxl_vit_h I get error messages as posted below

2024-06-07 13:14:38,508 - roop - INFO - Restore face with CodeFormer
Total progress: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:33<00:00,  1.39s/it]
2024-06-07 13:15:13,122 - ControlNet - INFO - unit_separate = False, style_align = False██████████████████████████████████████████████████| 24/24 [00:33<00:00,  1.15s/it]
2024-06-07 13:15:13,122 - ControlNet - INFO - Loading model from cache: ip-adapter-plus-face_sdxl_vit-h [368cf551]
2024-06-07 13:15:13,125 - ControlNet - INFO - Using preprocessor: ip-adapter_face_id
2024-06-07 13:15:13,125 - ControlNet - INFO - preprocessor resolution = 1024
*** Error running process: D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\Program Files\stable-diffusion-webui\modules\scripts.py", line 825, in process
        script.process(p, *script_args)
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1222, in process
        self.controlnet_hack(p)
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1207, in controlnet_hack
        self.controlnet_main_entry(p)
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1162, in controlnet_main_entry
        param.control_model.hook(
      File "D:\Program Files\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\ipadapter\plugable_ipadapter.py", line 148, in hook
        *[self.ipadapter.get_image_emb(o) for o in preprocessor_outputs]
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\ipadapter\plugable_ipadapter.py", line 148, in <listcomp>
        *[self.ipadapter.get_image_emb(o) for o in preprocessor_outputs]
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\ipadapter\ipadapter_model.py", line 347, in get_image_emb
        return self._get_image_embeds(preprocessor_output)
      File "D:\Program Files\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\ipadapter\ipadapter_model.py", line 170, in _get_image_embeds
        clip_vision_output["hidden_states"][-2].to(
    IndexError: too many indices for tensor of dimension 2

2024-06-07 13:15:57,250 - roop - INFO - Restore face with CodeFormer
Total progress: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:41<00:00,  1.73s/it]
2024-06-07 13:17:12,712 - ControlNet - INFO - unit_separate = False, style_align = False██████████████████████████████████████████████████| 24/24 [00:41<00:00,  1.24s/it]
2024-06-07 13:17:12,713 - ControlNet - INFO - Loading model from cache: ip-adapter-plus-face_sdxl_vit-h [368cf551]
2024-06-07 13:17:12,715 - ControlNet - INFO - Using preprocessor: ip-adapter_face_id_plus
2024-06-07 13:17:12,715 - ControlNet - INFO - preprocessor resolution = 512
*** Error running process: D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\Program Files\stable-diffusion-webui\modules\scripts.py", line 825, in process
        script.process(p, *script_args)
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1222, in process
        self.controlnet_hack(p)
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1207, in controlnet_hack
        self.controlnet_main_entry(p)
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1162, in controlnet_main_entry
        param.control_model.hook(
      File "D:\Program Files\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\ipadapter\plugable_ipadapter.py", line 148, in hook
        *[self.ipadapter.get_image_emb(o) for o in preprocessor_outputs]
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\ipadapter\plugable_ipadapter.py", line 148, in <listcomp>
        *[self.ipadapter.get_image_emb(o) for o in preprocessor_outputs]
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\ipadapter\ipadapter_model.py", line 347, in get_image_emb
        return self._get_image_embeds(preprocessor_output)
      File "D:\Program Files\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\Program Files\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\ipadapter\ipadapter_model.py", line 170, in _get_image_embeds
        clip_vision_output["hidden_states"][-2].to(
    TypeError: 'FaceIdPlusInput' object is not subscriptable

Steps to reproduce the problem

Go to Controlnet > Select Preprocessor and Model
Upload an face image
Generate
Console Log shows the messages above

What should have happened?

Face should be generated / replaced in the generated image

Commit where the problem happens

webui: c91dbe5c3b9377854239310311715b060de82ff1 controlnet:

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--xformers

List of enabled extensions

Console logs

See above

Additional information

No response

huchenlei commented 5 months ago

I don't think we have "ip-adapter-plus-face_sdxl_vit_h" preprocessor. It seems like you are using wrong preprocessor for your ipadapter model. Just select "ipadapter-auto" preprocessor and it will automatically match the correct preprocessor for you.

hugo4711 commented 5 months ago

Ok, I downloaded it from https://huggingface.co/h94/IP-Adapter/blob/main/sdxl_models/ip-adapter-plus-face_sdxl_vit-h.safetensors

See: https://github.com/tencent-ailab/IP-Adapter/issues/59

Is that not compatible with the sd-webui-controlnet extension in automatic 1111?

mostafablender commented 4 months ago

this is what i get and i dont know what is wrong here too : 2024-07-09 03:06:03,292 - ControlNet - INFO - unit_separate = False, style_align = False██████████████████████████████████████████████████████████████████████████████████████████| 20/20 [06:49<00:00, 21.86s/it] 2024-07-09 03:06:03,294 - ControlNet - INFO - Loading model from cache: ip-adapter-faceid-plusv2_sd15 [6e14fc1a] 2024-07-09 03:06:03,299 - ControlNet - INFO - Using preprocessor: ip-adapter-auto 2024-07-09 03:06:03,299 - ControlNet - INFO - preprocessor resolution = 512 2024-07-09 03:06:03,301 - ControlNet - INFO - ip-adapter-auto => ip-adapter_face_id_plus 2024-07-09 03:06:05,321 - ControlNet - INFO - ControlNet Hooked - Time = 2

what is ControlNet - INFO - unit_separate = False, style_align = False for ?

Mikubill / sd-webui-controlnet

[Bug]: #2949