search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.33k stars 381 forks source link

[Issue]: Model not loaded with diffusers backend #2553

Closed Arctomachine closed 8 months ago

Arctomachine commented 8 months ago

Issue Description

After some change in settings, probably enabling hypertile and freeu, main model is now not loaded anymore when running diffusers backend.

Version Platform Description

From system info tab image

Relevant log output

--- LAUNCH ---

Using VENV: C:\sd-next\venv
20:56:20-136770 DEBUG    Logger: file=C:\sd-next\sdnext.log level=10 size=0 mode=create
20:56:20-144775 INFO     Starting SD.Next
20:56:20-147769 INFO     Python 3.11.3 on Windows
20:56:20-479770 INFO     Version: app=sd.next updated=2023-11-23 hash=88aabcb3
                         url=https://github.com/vladmandic/automatic/tree/master
20:56:20-956779 INFO     Latest published version: 36562509f0c2bceee386ccd1ea14ada51e89e2fa
                         2023-11-23T19:38:46Z
20:56:20-992543 INFO     Platform: arch=AMD64 cpu=AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD
                         system=Windows release=Windows-10-10.0.19044-SP0 python=3.11.3
20:56:20-999547 DEBUG    Setting environment tuning
20:56:21-002543 DEBUG    Cache folder: C:\Users\Alexander\.cache\huggingface\hub
20:56:21-004541 DEBUG    Torch overrides: cuda=False rocm=False ipex=False diml=False openvino=False
20:56:21-007544 DEBUG    Torch allowed: cuda=True rocm=True ipex=True diml=True openvino=True
20:56:21-014553 INFO     nVidia CUDA toolkit detected: nvidia-smi present
20:56:23-293168 WARNING  Modified files: ['scripts/detect_extension.py']
20:56:23-382332 DEBUG    Repository update time: Thu Nov 23 18:45:40 2023
20:56:23-386328 INFO     Startup: standard
20:56:23-387692 INFO     Verifying requirements
20:56:23-409801 INFO     Verifying packages
20:56:23-412804 INFO     Verifying submodules
20:56:32-574115 DEBUG    Submodule: extensions-builtin/sd-extension-chainner / main
20:56:32-753975 DEBUG    Submodule: extensions-builtin/sd-extension-system-info / main
20:56:32-937242 DEBUG    Submodule: extensions-builtin/sd-webui-agent-scheduler / main
20:56:33-111246 DEBUG    Submodule: extensions-builtin/sd-webui-controlnet / main
20:56:33-303242 DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-images-browser / main
20:56:33-476572 DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-rembg / master
20:56:33-653572 DEBUG    Submodule: modules/k-diffusion / master
20:56:33-825574 DEBUG    Submodule: modules/lora / main
20:56:34-009569 DEBUG    Submodule: wiki / master
20:56:34-104573 DEBUG    Register paths
20:56:34-326571 DEBUG    Installed packages: 221
20:56:34-328569 DEBUG    Extensions all: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sd-webui-controlnet',
                         'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg']
20:56:34-769574 DEBUG    Running extension installer:
                         C:\sd-next\extensions-builtin\sd-extension-system-info\install.py
20:56:35-791314 DEBUG    Running extension installer:
                         C:\sd-next\extensions-builtin\sd-webui-agent-scheduler\install.py
20:56:36-851892 DEBUG    Running extension installer:
                         C:\sd-next\extensions-builtin\sd-webui-controlnet\install.py
20:56:37-864632 DEBUG    Running extension installer:
                         C:\sd-next\extensions-builtin\stable-diffusion-webui-images-browser\install.py
20:56:38-874247 DEBUG    Running extension installer:
                         C:\sd-next\extensions-builtin\stable-diffusion-webui-rembg\install.py
20:56:39-910617 DEBUG    Extensions all: ['adetailer', 'canvas-zoom', 'sd-webui-ar',
                         'sd-webui-regional-prompter']
20:56:39-914621 DEBUG    Running extension installer: C:\sd-next\extensions\adetailer\install.py
20:56:40-960841 DEBUG    Running extension installer: C:\sd-next\extensions\canvas-zoom\install.py
20:56:45-185019 INFO     Extensions enabled: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sd-webui-controlnet',
                         'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg',
                         'adetailer', 'canvas-zoom', 'sd-webui-ar', 'sd-webui-regional-prompter']
20:56:45-192023 INFO     Verifying requirements
20:56:45-213021 DEBUG    Setup complete without errors: 1700848605
20:56:45-222020 INFO     Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0}
20:56:45-224019 DEBUG    Starting module: <module 'webui' from 'C:\\sd-next\\webui.py'>
20:56:45-227020 INFO     Command line args: ['--lowvram', '--debug'] lowvram=True debug=True
20:56:48-765982 INFO     Load packages: torch=2.1.0+cu121 diffusers=0.23.1 gradio=3.43.2
20:56:49-519979 DEBUG    Read: file="config.json" json=33 bytes=1573
20:56:49-524981 INFO     Engine: backend=Backend.DIFFUSERS compute=cuda mode=no_grad device=cuda
                         cross-optimization="xFormers"
20:56:49-628970 INFO     Device: device=NVIDIA GeForce GTX 960 n=1 arch=sm_90 cap=(5, 2) cuda=12.1 cudnn=8801
                         driver=536.23
20:56:53-896324 DEBUG    Entering start sequence
20:56:53-905318 DEBUG    Initializing
20:56:53-910315 INFO     Available VAEs: path="models\VAE" items=1
20:56:53-914316 INFO     Disabling uncompatible extensions: backend=Backend.DIFFUSERS ['sd-webui-controlnet',
                         'multidiffusion-upscaler-for-automatic1111', 'a1111-sd-webui-lycoris']
20:56:53-918316 DEBUG    Scanning diffusers cache: models\Diffusers models\Diffusers items=0 time=0.00
20:56:53-922317 DEBUG    Read: file="cache.json" json=2 bytes=860
20:56:53-926320 DEBUG    Read: file="metadata.json" json=6 bytes=444
20:56:53-930319 INFO     Available models: path="models\Stable-diffusion" items=1 time=0.01
20:56:54-186316 DEBUG    Load extensions
20:56:56-667315 INFO     Extension:
                         script='extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py' Using
                         sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_scheduler.sqlite3
20:56:58-684543 INFO     Extension: script='extensions\adetailer\scripts\!adetailer.py' [-] ADetailer
                         initialized. version: 23.11.1, num models: 9
20:56:58-780540 INFO     Extensions time: 4.59 { Lora=1.81 sd-extension-chainner=0.10
                         sd-extension-system-info=0.05 sd-webui-agent-scheduler=0.50
                         stable-diffusion-webui-images-browser=0.14 stable-diffusion-webui-rembg=0.83
                         adetailer=1.05 sd-webui-regional-prompter=0.09 }
20:56:58-928557 DEBUG    Read: file="html/upscalers.json" json=4 bytes=2672
20:56:58-932556 DEBUG    Read: file="extensions-builtin\sd-extension-chainner\models.json" json=24 bytes=2719
20:56:58-937556 DEBUG    chaiNNer models: path="models\chaiNNer" defined=24 discovered=0 downloaded=0
20:56:58-944556 DEBUG    Load upscalers: total=52 downloaded=0 user=0 ['None', 'Lanczos', 'Nearest',
                         'ChaiNNer', 'ESRGAN', 'LDSR', 'RealESRGAN', 'SCUNet', 'SD', 'SwinIR']
20:56:58-972557 DEBUG    Load styles: folder="models\styles" items=288
20:56:58-977556 DEBUG    Creating UI
20:56:59-131080 INFO     Load UI theme: name="black-teal" style=Auto base=sdnext.css
20:56:59-189049 DEBUG    Read: file="html\reference.json" json=13 bytes=7962
20:56:59-225047 DEBUG    Extra networks: page='model' items=14 subfolders=3 tab=txt2img
                         folders=['models\\Stable-diffusion', 'models\\Diffusers', 'models\\Reference',
                         'C:\\sd-next\\models\\Stable-diffusion'] list=0.02 desc=0.00 info=0.00
20:56:59-260050 DEBUG    Extra networks: page='style' items=288 subfolders=2 tab=txt2img
                         folders=['models\\styles', 'html'] list=0.02 desc=0.00 info=0.00
20:56:59-267047 DEBUG    Extra networks: page='embedding' items=0 subfolders=1 tab=txt2img
                         folders=['models\\embeddings'] list=0.00 desc=0.00 info=0.00
20:56:59-273047 DEBUG    Extra networks: page='hypernetwork' items=0 subfolders=1 tab=txt2img
                         folders=['models\\hypernetworks'] list=0.00 desc=0.00 info=0.00
20:56:59-280048 DEBUG    Extra networks: page='vae' items=1 subfolders=1 tab=txt2img folders=['models\\VAE']
                         list=0.00 desc=0.00 info=0.00
20:56:59-287048 DEBUG    Extra networks: page='lora' items=2 subfolders=1 tab=txt2img
                         folders=['models\\Lora', 'models\\LyCORIS'] list=0.00 desc=0.00 info=0.00
20:56:59-558048 DEBUG    Read: file="ui-config.json" json=0 bytes=2
20:56:59-611048 DEBUG    Themes: builtin=6 default=5 external=55
20:57:00-345048 DEBUG    Script: 0.41 ui_tabs
                         C:\sd-next\extensions-builtin\stable-diffusion-webui-images-browser\scripts\image_br
                         owser.py
20:57:00-353048 DEBUG    Read: file="C:\sd-next\html\extensions.json" json=321 bytes=187710
20:57:02-177996 DEBUG    Extension list: processed=305 installed=11 enabled=10 disabled=1 visible=305
                         hidden=0
20:57:02-726747 INFO     Local URL: http://127.0.0.1:7860/
20:57:02-728746 DEBUG    Gradio functions: registered=1758
20:57:02-730751 INFO     Initializing middleware
20:57:02-736747 DEBUG    Creating API
20:57:03-021746 INFO     [AgentScheduler] Task queue is empty
20:57:03-023746 INFO     [AgentScheduler] Registering APIs
20:57:03-042746 DEBUG    Script: 0.11 app_started
                         C:\sd-next\extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py
20:57:03-181746 DEBUG    Scripts setup: ['X/Y/Z Grid:0.007', 'ADetailer:0.026', 'Aspect Ratio picker:0.012',
                         'Regional Prompter:0.014']
20:57:03-186748 DEBUG    Model metadata: file="metadata.json" no changes
20:57:03-188746 DEBUG    Model auto load disabled
20:57:03-193751 DEBUG    Save: file="config.json" json=33 bytes=1516
20:57:03-197750 INFO     Startup time: 17.96 { torch=2.46 gradio=1.01 diffusers=0.07 libraries=5.13
                         extensions=4.59 face-restore=0.25 upscalers=0.17 ui-extra-networks=0.31
                         ui-txt2img=0.07 ui-img2img=0.10 ui-settings=0.37 ui-extensions=2.30 ui-defaults=0.23
                         launch=0.29 api=0.12 app-started=0.33 }

--- OPENED UI ---

20:58:16-389058 INFO     MOTD: N/A
20:58:20-482848 DEBUG    Themes: builtin=6 default=5 external=55
20:58:20-830852 INFO     Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0;
                         Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36

--- CLICKED ON LCM LORA ---

20:59:36-189269 INFO     Select: model="revAnimated_v121 [f57b21e57b]"
20:59:36-193269 DEBUG    Load model weights: existing=False
                         target=C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors info=None
Loading model: C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors ----------- 0.0/2.0 GB -:--:--
20:59:36-286266 DEBUG    Desired Torch parameters: dtype=FP16 no-half=False no-half-vae=False upscast=False
20:59:36-289269 INFO     Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16
                         unet=torch.float16 context=no_grad fp16=True bf16=False
20:59:36-295267 INFO     Diffusers: model="Stable Diffusion" class=StableDiffusionPipeline
                         file="C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors" size=1874MB
20:59:37-308267 ERROR    Diffusers failed loading:
                         model=C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors
                         pipeline=Stable Diffusion/NoneType 'encoder.conv_in.weight'

--- REMOVED LORA FROM PROMPT ---

21:00:00-381023 DEBUG    Server: alive=True jobs=0 requests=53 uptime=191 memory=4.32/31.95
                         backend=Backend.DIFFUSERS state=idle
21:00:31-388148 INFO     Select: model="revAnimated_v121 [f57b21e57b]"
21:00:31-394145 DEBUG    Load model weights: existing=False
                         target=C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors info=None
Loading model: C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors ----------- 0.0/2.0 GB -:--:--
21:00:31-669165 DEBUG    Desired Torch parameters: dtype=FP16 no-half=False no-half-vae=False upscast=False
21:00:31-672164 INFO     Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16
                         unet=torch.float16 context=no_grad fp16=True bf16=False
21:00:31-677167 INFO     Diffusers: model="Stable Diffusion" class=StableDiffusionPipeline
                         file="C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors" size=1874MB
21:00:32-589693 ERROR    Diffusers failed loading:
                         model=C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors
                         pipeline=Stable Diffusion/NoneType 'encoder.conv_in.weight'

--- GENERATE IMAGE ---

21:02:02-343825 DEBUG    txt2img:
                         id_task=task(4s3ww60m3trznco)|prompt=|negative_prompt=|prompt_styles=[]|steps=20|sam
                         pler_index=0|latent_index=0|full_quality=True|restore_faces=False|tiling=False|n_ite
                         r=1|batch_size=1|cfg_scale=6|clip_skip=1|seed=-1.0|subseed=-1.0|subseed_strength=0|s
                         eed_resize_from_h=0|seed_resize_from_w=0||height=512|width=512|enable_hr=False|denoi
                         sing_strength=0.5|hr_scale=2|hr_upscaler=None|hr_force=False|hr_second_pass_steps=20
                         |hr_resize_x=0|hr_resize_y=0|image_cfg_scale=6|diffusers_guidance_rescale=0.7|refine
                         r_steps=5|refiner_start=0.8|refiner_prompt=|refiner_negative=|override_settings_text
                         s=[]
21:02:02-363823 INFO     Select: model="revAnimated_v121 [f57b21e57b]"
21:02:02-366823 DEBUG    Load model weights: existing=False
                         target=C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors info=None
Loading model: C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors ----------- 0.0/2.0 GB -:--:--
21:02:02-614822 DEBUG    Desired Torch parameters: dtype=FP16 no-half=False no-half-vae=False upscast=False
21:02:02-617821 INFO     Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16
                         unet=torch.float16 context=no_grad fp16=True bf16=False
21:02:02-622821 INFO     Diffusers: model="Stable Diffusion" class=StableDiffusionPipeline
                         file="C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors" size=1874MB
21:02:03-525824 ERROR    Diffusers failed loading:
                         model=C:\sd-next\models\Stable-diffusion\revAnimated_v121.safetensors
                         pipeline=Stable Diffusion/NoneType 'encoder.conv_in.weight'
21:02:03-676820 WARNING  Model not loaded

Backend

Diffusers

Branch

Master

Model

SD 1.5

Acknowledgements

vladmandic commented 8 months ago

can you reproduce this with any other model besides revAnimated_v121.safetensors?

Arctomachine commented 8 months ago

No, downloaded 2 different models and they load successfully. Tried to download this model again in case it got corrupted, but it errors still. So issue is with this specific model

vladmandic commented 8 months ago

that model does not have a valid vae - its missing keys. i don't want to support partial models, that is a neverending rathole. you may want to contact author and tell him to publish corrected version.