[Issue]: Long wait times before generation and display of Extra Networks after every launch, due to large amounts of Loras and Checkpoints

freecoderwaifu commented 8 months ago

Checklist

[X] The issue exists after disabling all extensions
[X] The issue exists on a clean installation of webui
[ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
[X] The issue exists in the current version of the webui
[X] The issue has not been reported before recently
[ ] The issue has been reported before but has not been fixed yet

What happened?

When having a ridiculous amount of Loras (pic related), the web UI takes a very long time before generations can start getting processed, due to it loading all the Extra Network files. This means even after pressing the Generate button, it takes way too long, about 5 to 10 min, before the generation finally starts getting processed and rendered.

There's no substantial and noticeably high HDD activity or logs while this is happening, it might just be the browser or Gradio caching or loading every thumbnail and data. This is with just the Lora and checkpoint .safetensors and their corresponding thumbnail, no additional .json or .info files made by Civitai Helper.

The Extra Networks UI understandably displays a loading icon while this is happening, and once the thumbnails are displayed, generations and overall UI usage is normal and as expected.

However, this happens on every single relaunch of the UI. This is different to the understandably longer load on the very first .bat launch after a fresh reboot. Maybe this will make it easier to understand:

.bat launch load times when having too many Loras First .bat launch on fresh PC (re)boot: About 5 minutes max. Every subsequent launch: 1 minute max.

Load times for Extra Networks tabs and wait time until generations can get processed, when having too many Loras: 5 to 10 minutes on every load and reload.

I've noticed at least on Librewolf, the web UI doesn't store any cache files in the browser's cache. Maybe having a way to save the Extra Networks thumbnails in the browser's persistent cache could alleviate the issue, but unsure if that's possible or if it would help.

Steps to reproduce the problem

1) Launch and relaunch web UI 2) Press Generate 3) Generation doesn't process until all Extra Networks are loaded in their corresponding tab 4) Wait about 5 to 10 minutes, potentially more as amount of models increases 5) Extra Networks tabs are finally populated with their thumbnails and files 6) Generation finally processes

What should have happened?

1) Launch web UI 2) Can Generate while Extra Networks are loaded in the background 3) Possible browser cache system to load Extra Networks faster

What browsers do you use to access the UI ?

Mozilla Firefox, Other

Sysinfo

sysinfo-2024-01-02-17-40.json

Console logs

No additional console logs out of the ordinary

venv "E:\variousAI\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: v1.7.0-266-ge4dcdcc9
Commit hash: e4dcdcc9554d7ff56993f5019eb90fe4ddf1e2e7
loading WD14-tagger reqs from E:\variousAI\stable-diffusion-webui\extensions\stable-diffusion-webui-wd14-tagger\requirements.txt
Checking WD14-tagger requirements.
Launching Web UI with arguments: --no-download-sd-model --xformers --skip-python-version-check
2024-01-02 10:46:01.684968: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
WARNING:tensorflow:From E:\variousAI\stable-diffusion-webui\venv\lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.
Civitai Helper: Get Custom Model Folder
Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu.
ControlNet preprocessor location: E:\variousAI\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2024-01-02 10:49:38,233 - ControlNet - INFO - ControlNet v1.1.426
2024-01-02 10:49:39,837 - ControlNet - INFO - ControlNet v1.1.426
[sd-webui-freeu] Controlnet support: *enabled*
== WD14 tagger /gpu:0, uname_result(system='Windows', node='vvv', release='10', version='10.0.22631', machine='AMD64') ==
Loading weights [05a9bb9e14] from 
2024-01-02 10:49:40,354 - AnimateDiff - INFO - Injecting LCM to UI.
2024-01-02 10:50:00,277 - AnimateDiff - INFO - Hacking i2i-batch.
Civitai Helper: Set Proxy:
Creating model from config: E:\variousAI\stable-diffusion-webui\configs\v1-inference.yaml
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 323.3s (initial startup: 0.2s, prepare environment: 43.7s, import torch: 23.5s, import gradio: 9.7s, setup paths: 38.5s, import ldm: 0.1s, initialize shared: 1.2s, other imports: 7.1s, setup gfpgan: 0.1s, list SD models: 60.0s, load scripts: 118.0s, create ui: 21.8s, gradio launch: 0.5s, add APIs: 0.1s).
Loading VAE weights specified in settings: 
Applying attention optimization: xformers... done.
Model loaded in 31.3s (load weights from disk: 20.6s, create model: 1.6s, apply weights to model: 2.3s, load VAE: 1.8s, load textual inversion embeddings: 2.9s, calculate empty prompt: 2.0s).

Additional information

No response

w-e-w commented 8 months ago

I don't think the browser loading thumbnails is the cause as lazy load is used on the extra networks tab, if you open the browser debugger and monintor the network activities you would see that the thumbnails are only loaded when you scroll down

I'm guessing it has more to do with crawling through the entire extra networks directory listing all the available networks or the creating html of the extra networks tab

I made a PR that should reduce unnecessary re-indexed of extra networks directory https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/14512 can you try and see if this helps

freecoderwaifu commented 8 months ago

Thank you for looking into it. Just tried it and the first generation right after launching the UI goes through as expected, but after about a minute or so passes a second generation still gets stuck and the Extra Networks tabs still take about the same long time to load.

AUTOMATIC1111 commented 8 months ago

See if this PR https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/14528 makes it any better.

freecoderwaifu commented 8 months ago

Thank you, sadly still experiencing long load times. It could also be something on my side, maybe a specific Lora, Checkpoint or Embedding causing it, or even a thumbnail. Load times with these many models were always a bit longer than desired but tolerable, then it started being much worse recently, started noticing it on commits around or after the 31st of December. I'll keep checking if it's a file causing it on my side meanwhile.

freecoderwaifu commented 8 months ago

Just tested with a much lower but still substantial amount of Loras, about 7k from the full 31k. (4 of the 5 subfolders I have: Loras trained on my pictures, general Loras trained by me, character Loras and concept Loras, leaving out style Loras).

With just that amount of Loras, the Extra Networks tabs both refreshed and loaded in about 3-5 seconds max, so can confirm both recent PRs do work as intended, as they would have loaded and refreshed noticeably slower before.

Adding the rest of the Loras (the much larger Style Loras subfolder) is when it chokes, so it might be a particular file in there, or also the sheer amount of extra files in that one folder, 48,557 to be exact including thumbnails.

AUTOMATIC1111 commented 8 months ago

Can you revert to a commit before 31st of December to check if it really would start working well again?

freecoderwaifu commented 8 months ago

Reverted to commits from the 28th and then 22nd of December, long load times still are there, so it is definitely something on my side, either a particular Lora downloaded recently or even something else system related.

I'd say this is closed for now as the commits to address this issue did help, and with a more manageable amount of Loras the Extra Networks tabs do load substantially faster than before.

Will update later on if I find something else related.

haru832 commented 8 months ago

I'm actually still having this issue after getting the latest dev commit on Jan 05, were you able to find the problematic LORA?

freecoderwaifu commented 8 months ago

I'm actually still having this issue after getting the latest dev commit on Jan 05, were you able to find the problematic LORA?

Not a particular Lora but I had a hunch it might be related to certain characters in their filename and it seems to be the case. I used this regex [^a-zA-Z0-9()_\-.\s] with Power Rename from Windows Powertoys to remove most special characters and now the Lora list loads in just 2 minutes max vs the 10+ minutes it was taking before. This is with 62,718 files total spread over 5 subfolders, including thumbnails.

I also DeepL translated some Loras with chinese characters beforehand so the renaming process didn't completely erase their name. Some files also outright had garbage characters for whatever Civitai reason.

Not sure which particular character or characters might have been causing it though, but leaving only those characters in the regex above fixed it completely.

Two other things I did, in case it's a compound issue: I had some thumbnails with .preview.preview.png extensions (just Civitai Helper things) and renamed them back to just .preview.png Removed a couple of thumbnails with no corresponding .safetensors

haru832 commented 8 months ago

I see, what if you re-download the same lora, but you've altered the filename, how do you keep track to prevent a duplicate?

freecoderwaifu commented 8 months ago

Hashes could help, Civitai Helper also has a duplicates finder that uses hashes but I don't know how good it is since haven't tried it. A lot of files on Civitai have really unhelpful names, and even using their metadata names instead of their filenames doesn't really help and makes things worse sometimes, so it's better to rename them anyway.

Now what needs to be tested is which particular character or characters cause the Extra Networks list to take considerably longer to load, when I have more time I'll do some tests. But that might be a general filesystem issue rather than a particular bug with the web UI.

haru832 commented 8 months ago

Yeah I wasn't sure if it was a good idea to change their names, but I suppose it's worth a shot. I'm just worried about having to re-reference any foreign language on an image I found later, but if I just left foreign characters alone, would it have a drastic difference in loading times. But thank you for the tips though, very helpful!

AUTOMATIC1111 / stable-diffusion-webui