search

AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
143.59k stars 27.03k forks source link

[Bug]: In ver1.10.1, loading is slow when there are a large number of models. #16399

Open NaokiSato102 opened 3 months ago

NaokiSato102 commented 3 months ago

Checklist

What happened?

After updating from the previous version 1.9.x to version 1.10.1, loading of various tabs became slower. It seems data may not be loading properly, as even in the prompt input field, it indicates that existing LoRa models are not found. When trying to disable extensions, the list doesn't load. Attempting to update the UI is not possible as this is already the latest major update with a version tag. There don't seem to be any existing reports of this issue. Tried backing up and deleting config.json and ui-config.json before starting, but it wouldn't launch.

Steps to reproduce the problem

Place a large number of models in various folders. This includes LoRas, Checkpoints, TIs, etc. In my environment, there are many LoRas. The "--lora-dir" argument is used to specify the loading location. Since it was working normally before, I don't think the problem lies there. If left running, it seems to improve at some point. This might be due to the completion of some kind of database construction. Clearing the UI from the browser resets it to the initial state.

What should have happened?

The UI should load within a reasonable time frame, allowing browsing of various tabs, extension lists, etc. For LoRas that exist, the prompt should not show errors about their absence, while showing errors only for truly non-existent ones or those not yet updated. (Ideally, updates would take effect immediately and errors would only show for truly non-existent items, but that might be a separate issue.)

What browsers do you use to access the UI ?

Mozilla Firefox

Sysinfo

{
  "date": "Sun Aug 18 02:02:04 2024",
  "timestamp": "02:02:09",
  "uptime": "Sun Aug 18 01:58:36 2024",
  "version": {
    "app": "stable-diffusion-webui.git",
    "updated": "2024-07-27",
    "hash": "82a973c0",
    "url": "https://github.com/AUTOMATIC1111/stable-diffusion-webui.git/tree/master"
  },
  "torch": "2.1.0+cu121 autocast  half",
  "gpu": {
    "device": "NVIDIA GeForce RTX 3060 (1) (sm_90) (8, 6)",
    "cuda": "12.1",
    "cudnn": 8801,
    "driver": "546.01"
  },
  "state": {
    "started": "Sun Aug 18 02:02:09 2024",
    "step": "0 / 0",
    "jobs": "0 / 0",
    "flags": "",
    "job": "",
    "text-info": ""
  },
  "memory": {
    "ram": {
      "free": 63.17,
      "used": 0.69,
      "total": 63.86
    },
    "gpu": {
      "free": 3.13,
      "used": 8.87,
      "total": 12
    },
    "gpu-active": {
      "current": 7.04,
      "peak": 7.04
    },
    "gpu-allocated": {
      "current": 7.04,
      "peak": 7.04
    },
    "gpu-reserved": {
      "current": 7.3,
      "peak": 7.3
    },
    "gpu-inactive": {
      "current": 0.26,
      "peak": 0.28
    },
    "events": {
      "retries": 0,
      "oom": 0
    },
    "utilization": 0
  },
  "optimizations": [
    "none"
  ],
  "libs": {
    "xformers": "0.0.20",
    "diffusers": "0.16.1",
    "transformers": "4.30.2"
  },
  "repos": {
    "Stable Diffusion": "[cf1d67a] 2023-03-25",
    "Stable Diffusion XL": "[45c443b] 2023-07-26",
    "BLIP": "[48211a1] 2022-06-07",
    "k_diffusion": "[ab527a9] 2023-08-12"
  },
  "device": {
    "active": "cuda",
    "dtype": "torch.float16",
    "vae": "torch.float32",
    "unet": "torch.float16"
  },
  "model": {
    "configured": {
      "base": "Pony_その他／汎\\はっさくXL(変態)(Ikena)_v1_3 目改良 [hassakuXLHentai_v13BetterEyesVersion].safetensors [bbad3bea96]",
      "refiner": "",
      "vae": "sdxl_vae MadeByollin氏改.safetensors"
    },
    "loaded": {
      "base": "D:\\Programs_D\\_MachineLearning\\StableDiffusionModels\\Online\\CheckPoint\\Pony_その他／汎\\はっさくXL(変態)(Ikena)_v1_3 目改良 [hassakuXLHentai_v13BetterEyesVersion].safetensors",
      "refiner": "",
      "vae": "D:\\Programs_D\\_MachineLearning\\StableDiffusionModels\\Online\\vae\\SDXL\\sdxl_vae MadeByollin氏改.safetensors"
    }
  },
  "schedulers": [
    "DDIM",
    "DDIM CFG++",
    "DPM adaptive",
    "DPM fast",
    "DPM++ 2M",
    "DPM++ 2M SDE",
    "DPM++ 2M SDE Heun",
    "DPM++ 2S a",
    "DPM++ 3M SDE",
    "DPM++ SDE",
    "DPM2",
    "DPM2 a",
    "Euler",
    "Euler a",
    "Heun",
    "LCM",
    "LMS",
    "PLMS",
    "Restart",
    "UniPC"
  ],
  "extensions": [
    "Generate-TransparentIMG (enabled)",
    "LDSR (enabled builtin)",
    "Lora (enabled builtin)",
    "PBRemTools (enabled)",
    "ScuNET (enabled builtin)",
    "Stable-Diffusion-WebUI-TensorRT (disabled)",
    "Stable-Diffusion-Webui-Civitai-Helper (enabled)",
    "SwinIR (enabled builtin)",
    "a1111-sd-webui-tagcomplete (enabled)",
    "adetailer (enabled)",
    "canvas-zoom (enabled)",
    "canvas-zoom-and-pan (enabled builtin)",
    "extra-options-section (enabled builtin)",
    "hypertile (enabled builtin)",
    "mobile (enabled builtin)",
    "novelai-2-local-prompt (enabled)",
    "openpose-editor (enabled)",
    "postprocessing-for-training (enabled builtin)",
    "prompt-bracket-checker (enabled builtin)",
    "prompt_translator (disabled)",
    "sd-colab-commands-browser (disabled)",
    "sd-dynamic-prompts (enabled)",
    "sd-extension-system-info (enabled)",
    "sd-webui-animatediff (enabled)",
    "sd-webui-ar (enabled)",
    "sd-webui-bilingual-localization (enabled)",
    "sd-webui-cardmaster (enabled)",
    "sd-webui-controlnet (enabled)",
    "sd-webui-custom-autolaunch (enabled)",
    "sd-webui-cutoff (disabled)",
    "sd-webui-deepdanbooru-object-recognition (enabled)",
    "sd-webui-deepdanbooru-tag2folder (enabled)",
    "sd-webui-enable-checker (enabled)",
    "sd-webui-freeu (enabled)",
    "sd-webui-lora-block-weight (enabled)",
    "sd-webui-lycoris-sorter (enabled)",
    "sd-webui-model-converter (enabled)",
    "sd-webui-negpip (enabled)",
    "sd-webui-prompt-history (enabled)",
    "sd-webui-regional-prompter (enabled)",
    "sd-webui-txt-img-to-3d-model (enabled)",
    "sd-webui-weight-helper (enabled)",
    "sd_extension-prompt_formatter (enabled)",
    "sd_lama_cleaner (enabled)",
    "sdweb-merge-block-weighted-gui (enabled)",
    "sdwebui-close-confirmation-dialogue (enabled)",
    "soft-inpainting (enabled builtin)",
    "stable-diffusion-webui-localization-ja_JP (enabled)",
    "stable-diffusion-webui-pixelization (enabled)",
    "stable-diffusion-webui-wd14-tagger (enabled)"
  ],
  "platform": {
    "arch": "AMD64",
    "cpu": "Intel64 Family 6 Model 167 Stepping 1, GenuineIntel",
    "system": "Windows",
    "release": "Windows-10-10.0.19045-SP0",
    "python": "3.10.9"
  },
  "crossattention": "Automatic",
  "backend": "",
  "pipeline": ""
}

Console logs

D:\Programs_D\_MachineLearning\StableDiffusionLauncher>cd ..\StableDiffusion_0301

D:\Programs_D\_MachineLearning\StableDiffusion_0301>call .\run.bat

D:\Programs_D\_MachineLearning\StableDiffusion_0301>set MODELS_DIR=D:\Programs_D\_MachineLearning\StableDiffusionModels\Online

D:\Programs_D\_MachineLearning\StableDiffusion_0301>powershell .\sub.ps1        --opt-sdp-attention --opt-sdp-no-mem-attention  --no-half-vae   --deepdanbooru  --skip-version-check    --disable-nan-check     --port 50012    --autolaunch   --controlnet-dir                 "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\controlnet"        --codeformer-models-path        "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\Codeformer"        --ckpt-dir     "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\CheckPoint" --vae-dir                                      "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\vae"        --embeddings-dir                        "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\embeddings"        --esrgan-models-path            "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\ESRGAN"    --gfpgan-models-path            "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\GFPGAN"    --hypernetwork-dir                      "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\hypernetworks"     --lora-dir                                      "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\Lora"      --lyco-dir                                      "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\LyCORIS"   --realesrgan-models-path        "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\RealESRGAN"        --swinir-models-path            "D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\SwinIR"    --xformers
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: v1.10.1
Commit hash: 82a973c04367123ae98bd9abdf80d9eda9b910e2
Installing requirements
loading WD14-tagger reqs from D:\Programs_D\_MachineLearning\StableDiffusion_0301\stable-diffusion-webui\extensions\stable-diffusion-webui-wd14-tagger\requirements.txt
Checking WD14-tagger requirements.
Launching Web UI with arguments: --opt-sdp-attention --opt-sdp-no-mem-attention --no-half-vae --deepdanbooru --skip-version-check --disable-nan-check --port 50012 --autolaunch --controlnet-dir 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\controlnet' --codeformer-models-path 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\Codeformer' --ckpt-dir 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\CheckPoint' --vae-dir 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\vae' --embeddings-dir 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\embeddings' --esrgan-models-path 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\ESRGAN' --gfpgan-models-path 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\GFPGAN' --hypernetwork-dir 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\hypernetworks' --lora-dir 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\Lora' --lyco-dir 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\LyCORIS' --realesrgan-models-path 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\RealESRGAN' --swinir-models-path 'D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\SwinIR' --xformers
2024-08-18 02:23:09.917055: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-08-18 02:23:10.619814: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
CHv1.8.10: Get Custom Model Folder
Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu.
[-] ADetailer initialized. version: 24.8.0, num models: 10
dirname:  D:\Programs_D\_MachineLearning\StableDiffusion_0301\stable-diffusion-webui\localizations
localizations:  {'ja_JP': 'D:\\Programs_D\\_MachineLearning\\StableDiffusion_0301\\stable-diffusion-webui\\extensions\\stable-diffusion-webui-localization-ja_JP\\localizations\\ja_JP.json', 'ks_JP': 'D:\\Programs_D\\_MachineLearning\\StableDiffusion_0301\\stable-diffusion-webui\\extensions\\stable-diffusion-webui-localization-ja_JP\\localizations\\ks_JP.json'}
ControlNet preprocessor location: D:\Programs_D\_MachineLearning\StableDiffusion_0301\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2024-08-18 02:23:21,702 - ControlNet - INFO - ControlNet v1.1.455
[sd-webui-freeu] Controlnet support: *enabled*
dirname:  D:\Programs_D\_MachineLearning\StableDiffusion_0301\stable-diffusion-webui\localizations
localizations:  {'ja_JP': 'D:\\Programs_D\\_MachineLearning\\StableDiffusion_0301\\stable-diffusion-webui\\extensions\\stable-diffusion-webui-localization-ja_JP\\localizations\\ja_JP.json', 'ks_JP': 'D:\\Programs_D\\_MachineLearning\\StableDiffusion_0301\\stable-diffusion-webui\\extensions\\stable-diffusion-webui-localization-ja_JP\\localizations\\ks_JP.json'}
== WD14 tagger /gpu:0, uname_result(system='Windows', node='FREEDOM', release='10', version='10.0.19045', machine='AMD64') ==
Loading weights [bbad3bea96] from D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\CheckPoint\Pony_その他／汎\はっさくXL(変態)(Ikena)_v1_3 目改良 [hassakuXLHentai_v13BetterEyesVersion].safetensors
CHv1.8.10: Set Proxy:
Creating model from config: D:\Programs_D\_MachineLearning\StableDiffusion_0301\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Loading VAE weights specified in settings: D:\Programs_D\_MachineLearning\StableDiffusionModels\Online\vae\SDXL\sdxl_vae MadeByollin氏改.safetensors
Applying attention optimization: xformers... done.
Model loaded in 4.5s (load weights from disk: 0.3s, create model: 0.4s, apply weights to model: 3.0s, load VAE: 0.2s, load textual inversion embeddings: 0.3s, calculate empty prompt: 0.2s).
2024-08-18 02:23:31,554 - ControlNet - INFO - ControlNet UI callback registered.
start ..\python/Scripts/lama-cleaner --model=lama --device=cud --port=7870
Running on local URL:  http://xxx.xxx.xxx.xxx:xxxxx

To create a public link, set `share=True` in `launch()`.
Startup time: 44.2s (prepare environment: 17.5s, import torch: 6.1s, import gradio: 1.3s, setup paths: 0.5s, initialize shared: 0.2s, other imports: 0.5s, list SD models: 0.3s, load scripts: 10.2s, scripts before_ui_callback: 0.1s, create ui: 7.2s, gradio launch: 0.3s).

Additional information

No response

NaokiSato102 commented 3 months ago

I am posting using Claude's Japanese translation.

w-e-w commented 3 months ago

When trying to disable extensions, the list doesn't load.

first disable extensions using the command ling arg --disable-extra-extensions https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Command-Line-Arguments-and-Settings see if the issue persists after

NaokiSato102 commented 3 months ago

Oh my, is that so? My apologies for the oversight. After trying this and disabling all extensions,
the loading finished in about 2 minutes, which is a reasonable amount of time.

w-e-w commented 3 months ago

the loading finished in about 2 minutes, which is a reasonable amount of time.

2 minutes that's still feels very slow

what do you actually mean by "load time"

for server load time you can see details from the Startup profile

image

how many lora do have? happens if you temporarily remove all your lora?

modify --lora-dir to point to a directory that doesn't contain lora

what kind of storage are you using?

if the issue is due to the amount of Loras then this could help

NaokiSato102 commented 3 months ago

"load time" refers to the time it takes for the tab contents to become viewable. It seems that 2 minutes was the worst-case scenario; generally, it became viewable in about 30 seconds.

The Startup profile is as follows: image

There are approximately 2,500 LoRa models. Even when trying --lora-dir "C:\Converted" (an empty folder), there was no change.

The storage is HDD. (I'd like to switch to SSD, but the cost...)

It seems that rather than the number of LoRas, some extension might be causing the slowdown. Binary search would be a hassle...

metapea commented 2 months ago

For me after i get a high amount loras (over 1k) in reForge:

I think it be a problem with webui handing a large amount of Lora in general since Pre UI update for the lora tab (when it had one button for sorting and not three), using --ui-debug-mode would give an error to the lora tab, as it turned off when you first start, making the sd-civitai-browser-plus usable without the high ram uses, until you turn on SD (by using the two refresh buttons on top and press generate or go to the lora tab).

Chengym2023 commented 2 months ago

the same proplem, i think it happen when i frequent switching checkpoint,then sdwebui crazy read my other hdd(no the local of app),if i Use checkpoint all the time there is no proplem happen.