[Bug]: IP-Adapter doesnt work anymore for Auto1111 DirectML (AMD on Windows)

CS1o commented 7 months ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

Im using the Latest Version of the DirectML Fork of Automatic1111 (without onnx or olive) on Windows 10. Webui Version: [1.7.0] • python: 3.10.11 • torch: 2.0.0+cpu • xformers: N/A • gradio: 3.41.2

After an ControlNet Update last week or before (could be since version 1.1.431), the IP-Adapter models dont work anymore.

Using Preprocessor: ip-adapter_clip_sd15 and model: ip-adapter-full-face_sd15 or any other model

When pressing Generate, I now get the following error posted down below:

The IP-Adapter worked without problems before, for months. I informed the Dev of the DirectML Fork who suggested that i could try --use-cpu controlnet (which uses the CPU for all ControlNet tasks), by adding it in the webui-user.bat.

With that option the IP-Adapter works. BUT now Openpose and all the other ControlNet models dont work when using CPU.

Stated this new issue with error code there as i think that would be a directml issue: https://github.com/lshqqytiger/stable-diffusion-webui-directml/issues/362#issuecomment-1907023903

The ControlNet Update shouldnt have changed how IP-Adapter was working. Everything worked fine for months. Now you have to decide if you want to use IP-Adapter OR all the other controlnet models.

Steps to reproduce the problem

Use the DirectML Fork of Automatic1111 (without using --onnx or olive)
Download the IP-Adapter Models for ControlNet
Select the Preprocessor Clip and any IP-Adapter model
Get the error from above.

What should have happened?

IP-Adapter should have applied the face to the generated image. There shouldnt be the error from below in the cmd.

Commit where the problem happens

webui: Version: [1.7.0] • python: 3.10.11 • torch: 2.0.0+cpu • xformers: N/A • gradio: 3.41.2 controlnet: v1.1.436 and v1.1.435 and v1.1.434 and maybe startet since v1.1.431

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--use-directml --update-all-extensions --medvram --opt-sub-quad-attention --opt-split-attention --no-half --upcast-sampling

List of enabled extensions

a1111-sd-webui-tagcomplete adetailer clip-interrogator-ext multidiffusion-upscaler-for-automatic1111 openpose-editor sd-webui-controlnet sd-webui-cutoff sd-webui-freeu sd-webui-llul sd-webui-openpose-editor sd-webui-reactor stable-diffusion-webui-Prompt_Generator stable-diffusion-webui-rembg test_my_prompt ultimate-upscale-for-automatic1111

Console logs

2024-01-23 23:09:54,952 - ControlNet - INFO - unit_separate = False, style_align = False
2024-01-23 23:09:55,127 - ControlNet - INFO - Loading model: ip-adapter-full-face_sd15 [852b9843]
2024-01-23 23:09:55,133 - ControlNet - INFO - Loaded state_dict from [D:\Programme\AI-Zeug\stable-diffusion-webui-directml\models\ControlNet\ip-adapter-full-face_sd15.safetensors]
2024-01-23 23:09:55,229 - ControlNet - INFO - ControlNet model ip-adapter-full-face_sd15 [852b9843] loaded.
2024-01-23 23:09:55,231 - ControlNet - INFO - Using preprocessor: ip-adapter_clip_sd15
2024-01-23 23:09:55,231 - ControlNet - INFO - preprocessor resolution = 512
*** Error running process: D:\Programme\AI-Zeug\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\modules\scripts.py", line 718, in process
        script.process(p, *script_args)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1117, in process
        self.controlnet_hack(p)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1102, in controlnet_hack
        self.controlnet_main_entry(p)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\controlnet.py", line 923, in controlnet_main_entry
        detected_map, is_image = self.preprocessor[unit.module](
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\utils.py", line 80, in decorated_func
        return cached_func(*args, **kwargs)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\utils.py", line 64, in cached_func
        return func(*args, **kwargs)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\global_state.py", line 37, in unified_preprocessor
        return preprocessor_modules[preprocessor_name](*args, **kwargs)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\processor.py", line 389, in clip
        clip_encoder[config] = ClipVisionDetector(config, low_vram)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\annotator\clipvision\__init__.py", line 115, in __init__
        sd = torch.load(file_path, map_location=self.device)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\modules\safe.py", line 108, in load
        return load_with_extra(filename, *args, extra_handler=global_extra_handler, **kwargs)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\modules\safe.py", line 156, in load_with_extra
        return unsafe_torch_load(filename, *args, **kwargs)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 809, in load
        return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 1172, in _load
        result = unpickler.load()
      File "C:\Users\webyo\AppData\Local\Programs\Python\Python310\lib\pickle.py", line 1213, in load
        dispatch[key[0]](self)
      File "C:\Users\webyo\AppData\Local\Programs\Python\Python310\lib\pickle.py", line 1254, in load_binpersid
        self.append(self.persistent_load(pid))
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 1142, in persistent_load
        typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 1116, in load_tensor
        wrap_storage=restore_location(storage, location),
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 1086, in restore_location
        return default_restore_location(storage, str(map_location))
      File "D:\Programme\AI-Zeug\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 220, in default_restore_location
        raise RuntimeError("don't know how to restore data location of "
    RuntimeError: don't know how to restore data location of torch.storage.UntypedStorage (tagged with privateuseone:0)

Additional information

No response

huchenlei commented 7 months ago

Please follow this instruction to run CLIP preprocessor on CPU. https://github.com/Mikubill/sd-webui-controlnet/pull/2495

SunGreen777 commented 7 months ago

Please follow this instruction to run CLIP preprocessor on CPU. #2495

It helped, thank you very much!

CS1o commented 7 months ago

Thanks alot, the ControlNet setting "Load CLIP preprocessor model on CPU" fixed it :)

Mikubill / sd-webui-controlnet