search

Mikubill / sd-webui-controlnet

WebUI extension for ControlNet
GNU General Public License v3.0
17.04k stars 1.96k forks source link

[Bug]: RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x1024 and 1280x8192) when using SDXL IP-Adapter on M2 Mac #2342

Closed orfeaus closed 10 months ago

orfeaus commented 10 months ago

Is there an existing issue for this?

What happened?

Whenever I try to use the controlnet cdxl IP-Adatper I see an error in my console. This doesn't happen with SDXL depth or canny models. The error is as follows:

2023-12-19 16:59:54,536 - ControlNet - INFO - unit_separate = False, style_align = False                                                                        | 10/40 [00:23<01:15,  2.53s/it]
2023-12-19 16:59:54,538 - ControlNet - INFO - Loading model from cache: ip-adapter_xl [4209e9f7]
2023-12-19 16:59:54,544 - ControlNet - INFO - Loading preprocessor: ip-adapter_clip_sd15
2023-12-19 16:59:54,544 - ControlNet - INFO - preprocessor resolution = 512
*** Error running before_process_batch: /Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
    Traceback (most recent call last):
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/modules/scripts.py", line 627, in before_process_batch
        script.before_process_batch(p, *script_args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1063, in before_process_batch
        self.controlnet_hack(p)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1042, in controlnet_hack
        self.controlnet_main_entry(p)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1020, in controlnet_main_entry
        param.control_model.hook(
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlmodel_ipadapter.py", line 367, in hook
        self.image_emb, self.uncond_image_emb = self.ipadapter.get_image_embeds(clip_vision_output)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlmodel_ipadapter.py", line 230, in get_image_embeds
        image_prompt_embeds = self.image_proj_model(clip_image_embeds)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
        return forward_call(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlmodel_ipadapter.py", line 38, in forward
        clip_extra_context_tokens = self.proj(embeds).reshape(-1, self.clip_extra_context_tokens,
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
        return forward_call(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions-builtin/Lora/networks.py", line 429, in network_Linear_forward
        return originals.Linear_forward(self, input)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/stable-diffusion-webui-composable-lora/composable_lora.py", line 150, in lora_Linear_forward
        return lora_forward(self, input, torch.nn.Linear_forward_before_lora(self, input))
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
        return F.linear(input, self.weight, self.bias)
    RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x1024 and 1280x8192)

Steps to reproduce the problem

  1. M2 Mac running macOS Ventura Version 13.5 (22G74)
  2. Go to text2image and enter a prompt.
  3. Add an image to controlnet.
  4. Choose the IP-Adapter XL model.
  5. Try to generate an image.

What should have happened?

Should have been able to use the IP-Adapter.

Commit where the problem happens

webui: v1.6.1 (4afaaf8) controlnet: v1.1.423

What browsers do you use to access the UI ?

Apple Safari

Command Line Arguments

I don't think so (but it's possible).

List of enabled extensions

Screenshot 2023-12-19 at 17 12 16

Console logs

2023-12-19 16:59:54,536 - ControlNet - INFO - unit_separate = False, style_align = False                                                                        | 10/40 [00:23<01:15,  2.53s/it]
2023-12-19 16:59:54,538 - ControlNet - INFO - Loading model from cache: ip-adapter_xl [4209e9f7]
2023-12-19 16:59:54,544 - ControlNet - INFO - Loading preprocessor: ip-adapter_clip_sd15
2023-12-19 16:59:54,544 - ControlNet - INFO - preprocessor resolution = 512
*** Error running before_process_batch: /Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
    Traceback (most recent call last):
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/modules/scripts.py", line 627, in before_process_batch
        script.before_process_batch(p, *script_args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1063, in before_process_batch
        self.controlnet_hack(p)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1042, in controlnet_hack
        self.controlnet_main_entry(p)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1020, in controlnet_main_entry
        param.control_model.hook(
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlmodel_ipadapter.py", line 367, in hook
        self.image_emb, self.uncond_image_emb = self.ipadapter.get_image_embeds(clip_vision_output)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlmodel_ipadapter.py", line 230, in get_image_embeds
        image_prompt_embeds = self.image_proj_model(clip_image_embeds)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
        return forward_call(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlmodel_ipadapter.py", line 38, in forward
        clip_extra_context_tokens = self.proj(embeds).reshape(-1, self.clip_extra_context_tokens,
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
        return forward_call(*args, **kwargs)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions-builtin/Lora/networks.py", line 429, in network_Linear_forward
        return originals.Linear_forward(self, input)
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/extensions/stable-diffusion-webui-composable-lora/composable_lora.py", line 150, in lora_Linear_forward
        return lora_forward(self, input, torch.nn.Linear_forward_before_lora(self, input))
      File "/Users/ME/Documents/stable-diffusion/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
        return F.linear(input, self.weight, self.bias)
    RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x1024 and 1280x8192)


### Additional information

_No response_
huchenlei commented 10 months ago

The log shows you are trying to call ip-adapter_clip_sd15 preprocessor on sdxl model.