search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.56k stars 409 forks source link

[Issue]: Error(s) in loading state_dict for LatentDiffusion: size mismatch #2592

Closed Tigwin closed 10 months ago

Tigwin commented 10 months ago

Issue Description

I'm likely doing something wrong, but I installed automatic, it works fine with the 1.5 model. I then downloaded the sdxl base and refiner, dropped them into modles/stable-diffusion/ and reloaded the model list.

When I try to render with sdxl base (I didn't see where to select refiner), it always errors:

10:07:43-055159 INFO Select: model="sd_xl_base_1.0" Loading model: C:\automatic\models\Stable-diffusion\sd_xl_base_1.0.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/6.9 GB -:--:-- 10:07:43-207220 INFO Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad fp16=True bf16=False 10:07:49-586595 INFO LDM: LatentDiffusion: mode=eps 10:07:49-587595 INFO LDM: DiffusionWrapper params=859.52M 10:07:49-589681 WARNING Model detected as SD-XL base model, but attempting to load using backend=original: model=C:\automatic\models\Stable-diffusion\sd_xl_base_1.0.safetensors size=6617 MB 10:07:49-589681 INFO Autodetect: model="Stable Diffusion XL" class=StableDiffusionXLPipeline file="C:\automatic\models\Stable-diffusion\sd_xl_base_1.0.safetensors" size=6617MB 10:07:49-735536 ERROR Error loading model weights: C:\automatic\models\Stable-diffusion\sd_xl_base_1.0.safetensors 10:07:49-737643 ERROR Error(s) in loading state_dict for LatentDiffusion: size mismatch for model.diffusion_model.input_blocks.4.1.proj_in.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([640, 640, 1, 1]). 10:07:50-325290 WARNING Model not loaded

Version Platform Description

Windows 11 Nvidia 3080 AMD 7950X 64gb ram

===============

$ ./webui.bat Using VENV: C:\automatic\venv 09:31:58-320180 INFO Starting SD.Next 09:31:58-322186 INFO Logger: file="C:\automatic\sdnext.log" level=INFO size=32523 mode=append 09:31:58-323192 INFO Python 3.10.11 on Windows 09:31:58-406020 INFO Version: app=sd.next updated=2023-12-04 hash=6cd4abba url=https://github.com/vladmandic/automatic.git/tree/de v 09:31:58-684980 INFO Latest published version: 93f35ccf885b7ad0e85878d58421507aecf14f95 2023-12-04T18:31:52Z 09:31:58-691691 INFO Platform: arch=AMD64 cpu=AMD64 Family 25 Model 97 Stepping 2, AuthenticAMD system=Windows release=Windows-10-10.0.22631-SP0 python=3.10.11 09:31:58-695115 INFO nVidia CUDA toolkit detected: nvidia-smi present 09:31:58-722244 WARNING Modified files: ['webui.sh'] 09:31:58-743926 INFO Startup: standard 09:31:58-743926 INFO Verifying requirements 09:31:58-752054 INFO Verifying packages 09:31:58-753080 INFO Installing package: git+https://github.com/openai/CLIP.git 09:32:02-797483 INFO Installing package: git+https://github.com/patrickvonplaten/invisible-water mark.git@remove_onnxruntime_depedency 09:32:07-890960 INFO Verifying submodules 09:32:46-221632 INFO Extension installed packages: sd-webui-agent-scheduler ['greenlet==3.0.1', 'SQLAlchemy==2.0.23'] 09:33:00-785084 INFO Extension installed packages: sd-webui-controlnet ['pywin32==306', 'lxml==4.9.3', 'opencv-contrib-python==4.8.1.78', 'reportlab==4.0.7', 'pycparser==2.21', 'portalocker==2.8.2', 'cffi==1.16.0', 'svglib==1.5.1', 'tinycss2==1.2.1', 'mediapipe==0.10.8', 'tabulate==0.9.0', 'cssselect2==0.7.0', 'webencodings==0.5.1', 'sounddevice==0.4.6', 'iopath==0.1.9', 'yacs==0.1.8', 'fvcore==0.1.5.post20221221'] 09:33:02-541592 INFO Extension installed packages: stable-diffusion-webui-images-browser ['Send2Trash==1.8.2'] 09:33:06-412447 INFO Extension installed packages: stable-diffusion-webui-rembg ['rembg==2.0.52', 'pooch==1.8.0', 'PyMatting==1.1.12'] 09:33:06-413488 INFO Extensions enabled: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg'] 09:33:06-414499 INFO Verifying requirements 09:33:06-428788 INFO Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0} 09:33:06-430844 INFO Command line args: [] 09:33:09-493471 INFO Load packages: torch=2.1.1+cu121 diffusers=0.24.0 gradio=3.43.2 09:33:10-016115 INFO Engine: backend=Backend.ORIGINAL compute=cuda mode=no_grad device=cuda cross-optimization="Scaled-Dot-Product" 09:33:10-065184 INFO Device: device=NVIDIA GeForce RTX 3080 n=1 arch=sm_90 cap=(8, 6) cuda=12.1 cudnn=8801 driver=546.17 09:34:30-893110 INFO Create: folder="models\Stable-diffusion" 09:34:30-896145 INFO Create: folder="models\Diffusers" 09:34:30-897145 INFO Create: folder="models\VAE" 09:34:30-898181 INFO Create: folder="models\Lora" 09:34:30-899181 INFO Create: folder="models\embeddings" 09:34:30-900193 INFO Create: folder="models\hypernetworks" 09:34:30-900193 INFO Create: folder="outputs\text" 09:34:30-901198 INFO Create: folder="outputs\image" 09:34:30-902196 INFO Create: folder="outputs\extras" 09:34:30-903196 INFO Create: folder="outputs\grids" 09:34:30-903196 INFO Create: folder="outputs\save" 09:34:30-905701 INFO Available VAEs: path="models\VAE" items=0 09:34:30-907192 INFO Disabled extensions: [] 09:34:30-908193 INFO Available models: path="models\Stable-diffusion" items=0 time=0.00 Download the default model? (y/N) y 09:35:16-785897 INFO Downloading: url="https://huggingface.co/runwayml/stable-diffusion-v 1-5/resolve/main/v1-5-pruned-emaonly.safetensors" file=C:\automatic\models\Stable-diffusion\v1-5-pruned-e maonly.safetensors Downloading ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:01:35 09:37:06-427732 INFO Extension: script='extensions-builtin\sd-webui-agent-scheduler\scr ipts\task_scheduler.py' Using sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_schedu ler.sqlite3 09:37:06-896896 INFO Extension: script='extensions-builtin\sd-webui-controlnet\scripts\ controlnet.py' Warning: ControlNet failed to load SGM - will use LDM instead. 09:37:06-897898 INFO Extension: script='extensions-builtin\sd-webui-controlnet\scripts\ controlnet.py' ControlNet preprocessor location: C:\automatic\extensions-builtin\sd-webui-controlnet\ann otator\downloads 09:37:06-904995 INFO Extension: script='extensions-builtin\sd-webui-controlnet\scripts\ hook.py' Warning: ControlNet failed to load SGM - will use LDM instead. 09:37:06-959328 INFO Extension: script='extensions-builtin\stable-diffusion-webui-image s-browser\scripts\image_browser.py' Image Browser: Creating database 09:37:06-961390 INFO Extension: script='extensions-builtin\stable-diffusion-webui-image s-browser\scripts\image_browser.py' Image Browser: Database created 09:37:45-763486 INFO Extensions time: 43.38 { Lora=0.81 sd-extension-chainner=0.12 sd-webui-agent-scheduler=3.06 sd-webui-controlnet=0.48 stable-diffusion-webui-rembg=38.80 } 09:37:45-860978 INFO Upscaler create: folder="models\chaiNNer" 09:37:45-864066 INFO Upscaler create: folder="models\RealESRGAN" 09:37:45-890540 INFO Load UI theme: name="black-teal" style=Auto base=sdnext.css 09:37:50-017225 INFO Local URL: http://127.0.0.1:7860/ 09:37:50-019519 INFO Initializing middleware 09:37:50-140849 INFO [AgentScheduler] Task queue is empty 09:37:50-142056 INFO [AgentScheduler] Registering APIs 09:37:50-249932 INFO Model metadata saved: file="metadata.json" items=1 time=0.00 09:37:50-252069 INFO Startup time: 283.82 { torch=2.36 gradio=0.67 libraries=81.40 extensions=43.38 models=142.12 face-restore=9.36 upscalers=0.10 ui-img2img=0.05 ui-settings=3.03 ui-extensions=0.64 ui-defaults=0.05 launch=0.22 api=0.06 app-started=0.17 } 09:41:38-036686 INFO MOTD: N/A 09:41:40-559198 INFO Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36

Relevant log output

No response

Backend

Original

Branch

Master

Model

SD-XL

Acknowledgements

vladmandic commented 10 months ago

10:07:49-589681 WARNING Model detected as SD-XL base model, but attempting to load using backend=original

before creating an issue - check docs, wiki, search issues - anything, its mentioned pretty much everywhere.

Tigwin commented 10 months ago

10:07:49-589681 WARNING Model detected as SD-XL base model, but attempting to load using backend=original

before creating an issue - check docs, wiki, search issues - anything, its mentioned pretty much everywhere.

@vladmandic

Yep, I did search. It's not in the install docs. If you search for " WARNING Model not loaded", there aren't matches. Look for yourself: https://github.com/vladmandic/automatic/issues?q=+WARNING++Model+not+loaded

This really isn't user friendly if you need a bunch of extra hidden directions to simply load SDXL which should be standard.

vladmandic commented 10 months ago

big text in readme is not enough? https://github.com/vladmandic/automatic#backend-support

Tigwin commented 10 months ago

big text in readme is not enough? https://github.com/vladmandic/automatic#backend-support

@vladmandic thanks for replying.

I'd like to think I can get this working, as I'm a software dev myself. And I've used A1111, Comfy, and a few others in the past. I like the options in your version and want to get it working but no matter what I do, it doesn't seem to work.

I went over the page again and I see this:

For all other model types, use backend Diffusers and use built in Model downloader or select model from Networks -> Models -> Reference list in which case it will be auto-downloaded and loaded

Originally I went to System > Models & Networks, and it had nothing shown there.

Later I found under Text > Networks > Model > Diffusers, but it was empty. I went to Reference and was able to download SDXL Turbo.

Then at the top of the page, I hit reload icon, then selected Diffusers\stabilityai\sdxl-turbo

Now it returns:

'StableDiffusionXLPipeline' object has no attribute 'alphas_cumprod'

So I went into settings, changed refiner to none, user interface to use sdxl-turbo. Apply settings.

Didn't work, so I changed execution backend from diffusers to original.

I get:

Error: model not loaded

This guy seems to have the same error I had:

https://www.reddit.com/r/StableDiffusion/comments/16tagbu/vlad_sdnext_sdxl_directml/

He says to change VAE model, but under Settings > Execution & Models > image options, it shows either "Automatic" or "None"

Somewhere around here I got this error:

Another guy in that thread said:

All of these settings did not help me... until I also changed the "Cross-attention optimization method " back from scaled-dot-product to sub-quadratic. I also needed to completely shutdown the server with Strg-C, a reload thru the UI did not work once I toggled the optimization again.

It didn't work, so I did a restart.

Still doesn't work, this is the last error that I have gotten:

12:21:49-219149 INFO Load UI theme: name="invoked" style=Auto base=sdnext.css 12:21:54-115454 INFO Local URL: http://127.0.0.1:7860/ 12:21:54-121456 INFO Initializing middleware 12:21:54-320370 INFO [AgentScheduler] Task queue is empty 12:21:54-477359 INFO [AgentScheduler] Registering APIs 12:21:54-975682 INFO Startup time: 337.33 { samplers=329.92 extensions=1.57 extra-networks=0.05 ui-extra-networks=0.15 ui-txt2img=0.21 ui-img2img=0.39 ui-train=0.10 ui-models=0.09 ui-interrogate=0.07 ui-settings=0.36 ui-extensions=2.55 ui-defaults=0.29 launch=0.71 app-started=0.82 } 12:21:55-956688 INFO MOTD: N/A 12:22:04-607909 INFO Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 12:22:54-915767 INFO MOTD: N/A 12:23:04-830335 INFO Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 12:23:10-283936 INFO GPU high memory utilization: 100% {'ram': {'used': 1.3, 'total': 63.74}, 'gpu': {'used': 10.0, 'total': 10.0}, 'retries': 0, 'oom': 0} 12:23:13-199827 ERROR Running script process: extensions-builtin\sd-webui-controlnet\scripts\controlnet.py: AttributeError ╭───────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────╮ │ C:\automatic\modules\scripts.py:518 in process │ │ │ │ 517 │ │ │ │ args = p.per_script_args.get(script.title(), p.script_args[script.args_f │ │ ❱ 518 │ │ │ │ script.process(p, *args, *kwargs) │ │ 519 │ │ │ except Exception as e: │ │ │ │ C:\automatic\extensions-builtin\sd-webui-controlnet\scripts\controlnet.py:993 in process │ │ │ │ 992 │ │ if not Script.process_has_sdxl_refiner(p): │ │ ❱ 993 │ │ │ self.controlnet_hack(p) │ │ 994 │ │ return │ │ │ │ C:\automatic\extensions-builtin\sd-webui-controlnet\scripts\controlnet.py:982 in controlnet_hack │ │ │ │ 981 │ │ t = time.time() │ │ ❱ 982 │ │ self.controlnet_main_entry(p) │ │ 983 │ │ if len(self.enabled_units) > 0: │ │ │ │ C:\automatic\extensions-builtin\sd-webui-controlnet\scripts\controlnet.py:653 in controlnet_main_entry │ │ │ │ 652 │ │ sd_ldm = p.sd_model │ │ ❱ 653 │ │ unet = sd_ldm.model.diffusion_model │ │ 654 │ │ self.noise_modifier = None │ │ │ │ C:\automatic\venv\lib\site-packages\diffusers\configuration_utils.py:138 in getattr │ │ │ │ 137 │ │ │ │ ❱ 138 │ │ raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'" │ │ 139 │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ AttributeError: 'StableDiffusionXLPipeline' object has no attribute 'model' 12:23:13-704239 INFO GPU high memory utilization: 100% {'ram': {'used': 1.3, 'total': 63.74}, 'gpu': {'used': 10.0, 'total': 10.0}, 'retries': 0, 'oom': 0} 12:23:17-012697 ERROR Exception: 'StableDiffusionXLPipeline' object has no attribute 'alphas_cumprod' 12:23:17-014697 ERROR Arguments: args=('task(u2jv2eh4fyszo4g)', 'cat', '', [], 20, None, None, True, False, False, 1, 1, 6, 6, 0.7, 1, -1.0, -1.0, 0, 0, 0, 512, 512, False, 0.5, 2, 'None', False, 20, 0, 0, 5, 0.8, '', '', False, 4, 0.95, False, 1, 1, False, 0.6, 1, [], 0, False, False, 'positive', 'comma', 0, False, False, '', 0, '', [], 0, '', [], 0, '', [], False, True, False, False, False, False, 0, None, None, False, None, None, False, None, None, False, 50, 'None', 16, 'None', 1, False, 'None', 2, True, 1, 0, 'none', 0.5, None, UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', save_detected_map=True), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', save_detected_map=True), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', save_detected_map=True)) kwargs={} 12:23:17-068632 ERROR gradio call: AttributeError ╭───────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────╮ │ C:\automatic\modules\call_queue.py:31 in f │ │ │ │ 30 │ │ │ try: │ │ ❱ 31 │ │ │ │ res = func(args, **kwargs) │ │ 32 │ │ │ │ progress.record_results(id_task, res) │ │ │ │ C:\automatic\modules\txt2img.py:69 in txt2img │ │ │ │ 68 │ if processed is None: │ │ ❱ 69 │ │ processed = processing.process_images(p) │ │ 70 │ p.close() │ │ │ │ C:\automatic\modules\processing.py:733 in process_images │ │ │ │ 732 │ │ │ with context_hypertile_vae(p), context_hypertile_unet(p): │ │ ❱ 733 │ │ │ │ res = process_images_inner(p) │ │ 734 │ │ │ │ C:\automatic\extensions-builtin\sd-webui-controlnet\scripts\batch_hijack.py:42 in processing_process_images_hijack │ │ │ │ 41 │ │ │ # we are not in batch mode, fallback to original function │ │ ❱ 42 │ │ │ return getattr(processing, 'controlnet_original_process_images_inner')(p, │ │ 43 │ │ │ │ C:\automatic\modules\processing.py:885 in process_images_inner │ │ │ │ 884 │ │ │ │ from modules.processing_diffusers import process_diffusers │ │ ❱ 885 │ │ │ │ x_samples_ddim = process_diffusers(p, p.seeds, p.prompts, p.negative_pro │ │ 886 │ │ │ else: │ │ │ │ ... 2 frames hidden ... │ │ │ │ C:\automatic\modules\sd_samplers.py:59 in create_sampler │ │ │ │ 58 │ elif shared.backend == shared.Backend.DIFFUSERS: │ │ ❱ 59 │ │ sampler = config.constructor(model) │ │ 60 │ │ if not hasattr(model, 'scheduler_config'): │ │ │ │ C:\automatic\modules\sd_samplers_compvis.py:11 in │ │ │ │ 10 samplers_data_compvis = [ │ │ ❱ 11 │ sd_samplers_common.SamplerData('UniPC', lambda model: VanillaStableDiffusionSampler( │ │ 12 │ sd_samplers_common.SamplerData('DDIM', lambda model: VanillaStableDiffusionSampler(l │ │ │ │ C:\automatic\modules\sd_samplers_compvis.py:19 in init │ │ │ │ 18 │ def init(self, constructor, sd_model): │ │ ❱ 19 │ │ self.sampler = constructor(sd_model) │ │ 20 │ │ self.is_ddim = hasattr(self.sampler, 'p_sample_ddim') │ │ │ │ C:\automatic\modules\unipc\sampler.py:16 in init │ │ │ │ 15 │ │ self.after_sample = None │ │ ❱ 16 │ │ self.register_buffer('alphas_cumprod', to_torch(model.alphas_cumprod)) │ │ 17 │ │ │ │ C:\automatic\venv\lib\site-packages\diffusers\configuration_utils.py:138 in getattr │ │ │ │ 137 │ │ │ │ ❱ 138 │ │ raise AttributeError(f"'{type(self).name__}' object has no attribute '{name}'" │ │ 139 │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ AttributeError: 'StableDiffusionXLPipeline' object has no attribute 'alphas_cumprod' 12:23:17-477861 INFO GPU high memory utilization: 100% {'ram': {'used': 1.3, 'total': 63.74}, 'gpu': {'used': 10.0, 'total': 10.0}, 'retries': 0, 'oom': 0}

vladmandic commented 10 months ago

image

set that, resetart server and do not change anything else. if you did, you probably want to reset configuration since i have no idea what else you've changed while clicking around.

and step-by-step guide is wiki and its linked in readme i've posted.

i'm sorry, but there are thousands of users that don't have issue with those intstructions.

Tigwin commented 10 months ago

@vladmandic I had that set, so I started over. Deleted automatic/, fresh git checkout.

I couldn't download sdxl via the client because there was nothing in that section. This was fixed by restarting.

Downloaded SDXL turbo, then I set diffuser backend, and it didn't work. So I restarted again.

Now it's rendering. Just needed a few restarts and not to change much of the defaults.

thanks