search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.7k stars 423 forks source link

[Issue]: Unable to use custom ControlNet models #3036

Closed kairosecrans closed 7 months ago

kairosecrans commented 7 months ago

Issue Description

I'm having some problems using custom ControlNet models (on Windows). I put them in the /models/control/controlnet folder and they appear in the list, but they won't load. It seems like part of the path is being doubled and using the wrong slash.

When I tweak that path code to fix it for my system, SD1.5 models load properly and are usable, but SDXL-based ones break.

For reference, I'm attempting to use monster-labs/control_v1p_sd15_qrcode_monster and monster-labs/control_v1p_sdxl_qrcode_monster.

Version Platform Description

Windows 10, AMD RX 6800 via ZLUDA, accessed using Brave

Starting SD.Next Logger: file="C:\ai\automatic\sdnext.log" level=INFO size=91726 mode=append Python 3.10.12 on Windows Version: app=sd.next updated=2024-04-06 hash=e783b098 branch=master url=https://github.com/vladmandic/automatic.git/tree/master Platform: arch=AMD64 cpu=Intel64 Family 6 Model 167 Stepping 1, GenuineIntel system=Windows release=Windows-10-10.0.19045-SP0 python=3.10.12 AMD ROCm toolkit detected ZLUDA support: experimental Using ZLUDA in C:\ai\zluda Deleted files: ['modules/control/units/controlnet.py', '"webui-user (autolaunch).cmd"', '"webui-user - Copy.cmd"', 'webui-user.cmd'] Modified files: ['modules/control/units/controlnet.py', 'webui-user.cmd'] Extensions: disabled=[] Extensions: enabled=['Lora', 'sd-extension-chainner', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg'] extensions-builtin Extensions: enabled=[] extensions

Relevant log output

16:13:02-187908 DEBUG    Control ControlNet model loading: id="sdxl_qrcode_monster"
                         path="models\control\controlnet\models\control\controlnet\sdxl_qrcode_monster.safetensors"
16:13:02-190881 ERROR    Control ControlNet model load failed: id="sdxl_qrcode_monster" error=list index out of range
16:13:02-192874 ERROR    Control ControlNet load: IndexError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ C:\ai\automatic\modules\control\units\controlnet.py:161 in load                                                     │
│                                                                                                                     │
│   160 │   │   │   if model_path.endswith('.safetensors'):                                                           │
│ > 161 │   │   │   │   self.load_safetensors(model_path)                                                             │
│   162 │   │   │   else:                                                                                             │
│                                                                                                                     │
│ C:\ai\automatic\modules\control\units\controlnet.py:125 in load_safetensors                                         │
│                                                                                                                     │
│   124 │   │   │   parts = model_path.split('/')                                                                     │
│ > 125 │   │   │   repo_id = f'{parts[0]}/{parts[1]}'                                                                │
│   126 │   │   │   filename = os.path.splitext('/'.join(parts[2:]))[0]                                               │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
IndexError: list index out of range

-- AND --

14:50:56-973175 DEBUG    Control ControlNet model loading: id="control_v1p_sd15_qrcode_monster"
                         path="models\control\controlnet\models\control\controlnet\control_v1p_sd15_qrcode_monster.safe
                         tensors"
14:50:58-652007 DEBUG    Control ControlNet model loaded: id="control_v1p_sd15_qrcode_monster"
                         path="models\control\controlnet\models\control\controlnet\control_v1p_sd15_qrcode_monster.safe
                         tensors" time=1.68
14:51:01-124795 DEBUG    Control ControlNet model loading: id="control_v1p_sdxl_qrcode_monster"
                         path="models\control\controlnet\models\control\controlnet\control_v1p_sdxl_qrcode_monster.safe
                         tensors"
14:51:01-483903 ERROR    Control ControlNet model load failed: id="control_v1p_sdxl_qrcode_monster"
                         error='middle_block_out.0.weight'
14:51:01-487812 ERROR    Control ControlNet load: KeyError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ C:\ai\automatic\modules\control\units\controlnet.py:162 in load                                                     │
│                                                                                                                     │
│   161 │   │   │   if model_path.endswith('.safetensors'):                                                           │
│ > 162 │   │   │   │   self.load_safetensors(model_path)                                                             │
│   163 │   │   │   else:                                                                                             │
│                                                                                                                     │
│ C:\ai\automatic\modules\control\units\controlnet.py:145 in load_safetensors                                         │
│                                                                                                                     │
│   144 │   │   │   self.load_config['original_config_file '] = config_path                                           │
│ > 145 │   │   self.model = ControlNetModel.from_single_file(model_path, **self.load_config)                         │
│   146                                                                                                               │
│                                                                                                                     │
│ C:\ai\automatic\venv\lib\site-packages\huggingface_hub\utils\_validators.py:118 in _inner_fn                        │
│                                                                                                                     │
│   117 │   │                                                                                                         │
│ > 118 │   │   return fn(*args, **kwargs)                                                                            │
│   119                                                                                                               │
│                                                                                                                     │
│ C:\ai\automatic\venv\lib\site-packages\diffusers\loaders\controlnet.py:124 in from_single_file                      │
│                                                                                                                     │
│   123 │   │                                                                                                         │
│ > 124 │   │   component = create_diffusers_controlnet_model_from_ldm(                                               │
│   125 │   │   │   class_name,                                                                                       │
│                                                                                                                     │
│ C:\ai\automatic\venv\lib\site-packages\diffusers\loaders\single_file_utils.py:973 in create_diffusers_controlnet_mo │
│                                                                                                                     │
│    972 │                                                                                                            │
│ >  973 │   diffusers_format_controlnet_checkpoint = convert_controlnet_checkpoint(checkpoint, diffusers_config)     │
│    974                                                                                                              │
│                                                                                                                     │
│ C:\ai\automatic\venv\lib\site-packages\diffusers\loaders\single_file_utils.py:937 in convert_controlnet_checkpoint  │
│                                                                                                                     │
│    936 │   # mid block                                                                                              │
│ >  937 │   new_checkpoint["controlnet_mid_block.weight"] = controlnet_state_dict.pop("middle_block_out.0.weight")   │
│    938 │   new_checkpoint["controlnet_mid_block.bias"] = controlnet_state_dict.pop("middle_block_out.0.bias")       │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
KeyError: 'middle_block_out.0.weight'

Backend

Diffusers

Branch

Master

Model

SD-XL

Acknowledgements

vladmandic commented 7 months ago

path duplication was a mistake, fixed. for custom controlnets, issue was that sdxl model was only released in diffusers format which requires custom config.json, so it needs to be handled differently.

anyhow, i've added support for that in latest dev branch. from changelog: