search

AbdBarho / stable-diffusion-webui-docker

Easy Docker setup for Stable Diffusion with user-friendly UI
Other
6.57k stars 1.09k forks source link

Model loading race condition with other parts of scripts? #317

Closed DevilaN closed 1 year ago

DevilaN commented 1 year ago

Has this issue been opened before?

Describe the bug

From time to time when starting container it seems that scripts do not see models. It fails loading model despite it is there, and fallbacks to "default one". 

webui-docker-auto-1  | Error loading script: sonar.py
webui-docker-auto-1  | Traceback (most recent call last):
webui-docker-auto-1  |   File "/stable-diffusion-webui/modules/scripts.py", line 229, in load_scripts
webui-docker-auto-1  |     script_module = script_loading.load_module(scriptfile.path)
webui-docker-auto-1  |   File "/stable-diffusion-webui/modules/script_loading.py", line 11, in load_module
webui-docker-auto-1  |     module_spec.loader.exec_module(module)
webui-docker-auto-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
webui-docker-auto-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
webui-docker-auto-1  |   File "/stable-diffusion-webui/extensions/stable-diffusion-webui-sonar/scripts/sonar.py", line 17, in <module>
webui-docker-auto-1  |     from modules.sd_samplers_common import setup_img2img_steps, SamplerData
webui-docker-auto-1  | ModuleNotFoundError: No module named 'modules.sd_samplers_common'
webui-docker-auto-1  | 
webui-docker-auto-1  | Checkpoint PaperCut_v1.ckpt [17b4578e45] not found; loading fallback protogenX53OfficialR_10.ckpt [f762cdef02]
webui-docker-auto-1  | Loading weights [f762cdef02] from /stable-diffusion-webui/models/Stable-diffusion/protogenX53OfficialR_10.ckpt
webui-docker-auto-1  | Creating model from config: /stable-diffusion-webui/configs/v1-inference.yaml
webui-docker-auto-1  | LatentDiffusion: Running in eps-prediction mode
webui-docker-auto-1  | DiffusionWrapper has 859.52 M params.
webui-docker-auto-1  | Applying xformers cross attention optimization.

Which UI

auto Hardware / Software

Steps to Reproduce

  1. docker compose --profile auto up
  2. See error

Additional context It started appearing when I've noticed accelerate is being used to run everything, but it might me coincidence.

AbdBarho commented 1 year ago

I will remove accelerate, it did not affect the performance from what I have seen.

Hopefully this fixes the issue.

320

DevilaN commented 1 year ago

Nope. That's not it. Upgraded to #320 and still got error:

webui-docker-auto-1  | Removing empty folder: /stable-diffusion-webui/models/BSRGAN
webui-docker-auto-1  | [AddNet] Updating model hashes...
0it [00:00, ?it/s]1  | 
webui-docker-auto-1  | (Unprompted v7.0.0 by Therefore Games)
webui-docker-auto-1  | (SETUP) Initializing Unprompted object...
webui-docker-auto-1  | (SETUP) Loading configuration files...
webui-docker-auto-1  | (SETUP) Debug mode is False
webui-docker-auto-1  | Checkpoint woolitize-3998-0869-1116.safetensors [a3b78e47c0] not found; loading fallback seek_art_mega_v1-4044-0869-0090.safetensors
webui-docker-auto-1  | Calculating sha256 for /stable-diffusion-webui/models/Stable-diffusion/seek_art_mega_v1-4044-0869-0090.safetensors: c556654e181f3aa681b59cb6e4918be5614c333161f6f91a49f1320f5bcb6090
webui-docker-auto-1  | Loading weights [c556654e18] from /stable-diffusion-webui/models/Stable-diffusion/seek_art_mega_v1-4044-0869-0090.safetensors
webui-docker-auto-1  | Creating model from config: /stable-diffusion-webui/configs/v1-inference.yaml

Of course model exists:

$ ls -alh data/StableDiffusion/woolitize-3998-0869-1116.safetensors 
-rw-r--r-- 1 root root 2,0G 01-31 11:31 data/StableDiffusion/woolitize-3998-0869-1116.safetensors

This happened only 3 times and then "it decided not to happen again right now" :(

DevilaN commented 1 year ago

It happens in a middle of regular usage when changing models as well.

webui-docker-auto-1  | Checkpoint elldrethSLucidMix_v10-0347-0000-0629.safetensors [c324325401] not found; loading fallback seek_art_mega_v1-4044-0869-0090.safetensors [c556654e18]
webui-docker-auto-1  | Loading weights [c556654e18] from /stable-diffusion-webui/models/Stable-diffusion/seek_art_mega_v1-4044-0869-0090.safetensors
webui-docker-auto-1  | Applying xformers cross attention optimization.
webui-docker-auto-1  | Weights loaded in 2.1s (load weights from disk: 0.2s, apply weights to model: 1.8s).
webui-docker-auto-1  | Checkpoint elldrethSLucidMix_v10-0347-0000-0629.safetensors [c324325401] not found; loading fallback seek_art_mega_v1-4044-0869-0090.safetensors [c556654e18]

Maybe this is some issue with auto... I will look for bug reports on A1111 repo later.

DevilaN commented 1 year ago

it still happens with .safetensor models but infrequently.

Possible duplicate of https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7064 and https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5582

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 7 days.

github-actions[bot] commented 1 year ago

This issue was closed because it has been stalled for 7 days with no activity.