search

AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
141.63k stars 26.76k forks source link

[Bug]: Calculating sha256 hashes constantly #12826

Open bdszoke opened 1 year ago

bdszoke commented 1 year ago

Is there an existing issue for this?

What happened?

When I launch webui.bat or webui-user.bat it often starts to recalculate the hashes of all of the models I have downloaded, which is problematic, as this can take 1-2 days, given the number of Loras and the fact that I keep the checkpoints on an external hdd (which is always on, and always plugged into the computer via usb), due to the size of them.

An example of what is happening can be found below:

Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\Low-poly-A.safetensors: Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\Low-poly-A.safetensors: Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\Low-poly-A.safetensors: Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\Low-poly-A.safetensors: Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\Low-poly-A.safetensors: 20a8876f87c23b20676ea6720ac7a2de52fa709ec1b82ce34ac4bb757d96b4be 20a8876f87c23b20676ea6720ac7a2de52fa709ec1b82ce34ac4bb757d96b4be 20a8876f87c23b20676ea6720ac7a2de52fa709ec1b82ce34ac4bb757d96b4be 20a8876f87c23b20676ea6720ac7a2de52fa709ec1b82ce34ac4bb757d96b4be 20a8876f87c23b20676ea6720ac7a2de52fa709ec1b82ce34ac4bb757d96b4be

Steps to reproduce the problem

  1. Go to .... Webui.bat
  2. Launch .... Webui.bat or Webui-user.bat
  3. ...

What should have happened?

It should have calculated the hashes once, when downloading them through the Civitai helper plugin, or after they were put into the appropriate folder (Lora, Stable-Diffusion, etc) and then not recalculate the hashes at random intervals.

Version or Commit where the problem happens

1.51

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Windows

What device are you running WebUI on?

No response

Cross attention optimization

Automatic

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

No

List of extensions

Stable-Diffusion-Webui-Civitai-Helper stable-diffusion-webui-anti-burn stable-diffusion-webui-aesthetic-gradients sd_civitai_extension sd-webui-gelbooru-prompt sd-webui-e621-prompt sd-webui-controlnet sd-webui-additional-networks sd-model-preview-xd openpose-editor model-keyword danbooru-prompt clip-interrogator-ext civitai

Screenshot 2023-08-28 at 04-37-05 Stable Diffusion

Console logs

Console log is literally too long for it to even fit inside the cmd prompt. Since the bug will continue until all the hashes have been calculated, I am putting a representative sample here:

[AddNet] Updating model hashes...
0it [00:00, ?it/s]
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
2023-08-28 04:49:01,878 - ControlNet - INFO - ControlNet v1.1.312
ControlNet preprocessor location: D:\Stable Diffusion\Automatic1111 Webui 1.51\extensions\sd-webui-controlnet\annotator\downloads
2023-08-28 04:49:02,239 - ControlNet - INFO - ControlNet v1.1.312
Civitai: API loaded
Checkpoint v1-5-pruned(1).ckpt [e1441589a6] not found; loading fallback ghostmix_v20Bakedvae.safetensors
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Stable-diffusion\ghostmix_v20Bakedvae.safetensors: [AddNet] Updating model hashes...
100%|███████████████████████████████████████████████████████████████████████████| 3211/3211 [00:00<00:00, 76457.92it/s]
e3edb8a26f44dfddc124129004b5ded3378be974b8d3bf82f43b9660d416d3d1
Loading weights [e3edb8a26f] from D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Stable-diffusion\ghostmix_v20Bakedvae.safetensors
Creating model from config: D:\Stable Diffusion\Automatic1111 Webui 1.51\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying attention optimization: Doggettx... done.
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\80sdarkfantasysd15b.pt: 91d4217080fd6aff6be0f13f5c56b4e94e4fd89ed175d716377e963b7cccf553
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\aid210.pt: afa9c1d8e13cbf5779505e81ec40e0ab62dae18fe0844214564d5e49a5b1cd59
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\akinav1.pt: 5a8909755a30e6fe1596c0eb47d69295276e69cd2a5447faa08423d62a4d42e1
5d558ec2fb1ecdbdde06329355560cf6bca29868f59ff81dae49611a915db19e
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\Asian-Less-Neg.pt: 22d2f003e76f94dcf891b821a3f447f25c73b2e0542f089427b33ff344070a96
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\Asian-Less-Toon.pt: 779eac4f1d05e74f646ecbe1ecfbc944f1cdf45007ced90b13566a5c1c70de5b
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\Asian-Less.pt: 965d8e05baeb16776451587b57a14decc421bf0220afaf8631117168073325dd
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\atompunkstylesd15.pt: d95e92dd77e19897f3b8771a96397c4e176254c5111f6fdc069b830bf356e05e
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\bad-picture-chill-75v.pt: 7d9cc5f549d7972f24803a3a9880a923fb9a1b68c1443da3ce1ff2f7eff25ae9
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\BadDream.pt: 758aac44351557ccfae2fc6bdf3a29670464e4e4eabb61f08c5b8531c221649c
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\badhandv4.pt: 5e40d722fc3d0c2decb62debfaf8058db30ccdae9ab00ff64b183907b435708e
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\chunli_alpha.pt: 318fd8ed0fe01cdcf8d8247bf04f01bc33647e91bca81e737a7b2ba30e22cd38
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\cindy_aurumTI.pt: ab27b1c71a20eef1bfa2a26d7076c7bab8284bf3ad5a3488966eb1616134dcc0
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\Clutter-Home.pt: af80173633cf9b9deee468a752acefbb9dcfca41a1adc315058104183a693532
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\Clutter-Mechanical.pt: 7bfdb20388cd3511db6fb0b0d0f2868729b9fca121a1314c7ebb1b180fa6b43d
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\difConsistency_negative_v2.pt: ccba1c92906555500163d3bc2883f4ad5b9f4aa51f5c3d7e2df42f9eac9d072b
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\easynegative.safetensors: c74b4e810b030f6b75fde959e2db678c268d07115b85356d3c0138ba5eb42340
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\EmWat69.pt: 3ea9855ce0a534420507f31dad36c31a1c46be49dcb3aa6d24ae00879fe4e66d
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\ErinNobodySD15.pt: 73ca248cb9267e2058cbb32b796059a7a7f34638af9c1bde9d759e2973c69807
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\FastNegativeV2.pt: a7465e7cc2a2a27571ee020053f307b5af54e6b60a0b0235d773f6bd55f7d078
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\fcNeg-neg.pt: f257fca13352b7e00cd12d4d2257719c8a8944e4e8565c5eb17a5c1f673c9458
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\GS-DeMasculate-neg.pt: 06d0ac10994835de3e9362bc5588fee7cd9b285de0170be271bcd853ba38525b
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\Hermione_Granger_Emma_Watson-4400.pt: c875dfb2f4930abb2e5c35d419fe91806b9caecd2807c635f5151627192189e9
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\kbxll.pt: 5ca4b187759a539c5e3399fe96f1194757983c81dbee872bad194444bc5affec
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\KimberlyNobodySD15.pt: c2b00023d40aef4f4859f56e97df2fdbdf4d591eff0d830754ace04b78ef67f5
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\krface1.pt: 94cd7dc26325a1811f701e4bb6ac72e58c9f16fe9680359492b23d54fa2e46e7
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\lr.pt: fe5a4dfc4a76152ecf822156ee178a84e296cd888a08cb3e95112cc498abdafc
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\lucy_kushinadaV2.pt: 1263ab39896af8d242923187f94cd3887c090581209b517f78f183b19fb5734c
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\nami.pt: 755e53ac02eff5c0fe52701b62eb47ec44f8998a5400e0e6534f8d9ff92f5836
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\ng_deepnegative_v1_2t.pt: 43483653fd2abc807aff08f95d31ce62b3a3cf9c2a2fe43d0c6760dee55ed8e6
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\ng_deepnegative_v1_64t.pt: 3f2693e92692b4cc01e9c3166986db52c215ab4df68be8c8424e41f9bbd4c070
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\ng_deepnegative_v1_75t.pt: 54e7e4826d53949a3d0dde40aea023b1e456a618c608a7630e3999fd38f93245
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\opt-6000.pt: b111ec11fe14d3eb3c417f08a339ecf2364065a93a5f945a59eb2dcb02625ca1
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\pureerosface_v1.pt: dedb4322e42e360fe01775ba817be03ac6a6c307744562bb0d6759368bc681da
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\Rinko Iori.pt: 10c69b43d0574df563344a884cc49157547edebe8c0c32eec577948893c02cdd
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\robin_v2_1600.pt: 4efc2b7864e27bd8654b7e8e206c96536e3384ca414fc51958e936645fce63d8
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\rosalinenobodysd15.bin: 4ae66b9ade14467cef73e5b3194975bb7b5bdb7f9f9b88fa29fc4c9d88f6c41c
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\rukia.pt: ff9e7e30a592e261704ae1c440524afe6d720ad4e4b9d68253941b64bd44a76e
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\rurinobodysd15.bin: b2bf305433a6f6e54c65b83024c7f63cf0017e013a9bee0096b1e229338e25a2
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\sashanobodysd15.pt: 131cf5219f6b2ca188dda37d62922abad8e70ce44bb9ae828ba8c88cf83e3d4f
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\southofthebordersd15.pt: 9fe3116c0bab18b520a8da1bc90fdf031d1b390c75f1139d31ba8fd4507e2f9c
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\southofthebordersd15_2.pt: 9fe3116c0bab18b520a8da1bc90fdf031d1b390c75f1139d31ba8fd4507e2f9c
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\sxz-ashe-ow.pt: 0a041dabb0ada3bf5c2c69a7b9bd52201249121214e02b902a5e6409c9f67d49
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\sxz-jaina-v1.pt: e12afc9bc151e89ac6b834a78f4cfc777ec549c20ef73f3250283366cc73e213
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\sxz-jaina-v2.pt: f5a972b33007aa7bc94ede5c33823924c0ac3b058268bb215a7e808a219e7ad4
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\sxz-raiden-mgr.pt: 7a186774b57d76d05f9c83f91b1e922fdc105d5ffeb5e381382c1eb0f17fff4b
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\tifa_4900.pt: 468abc93b92ee29ee5e3fcdb9ec4cf606752db3f556bcb385edd936294b8b24b
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\UnrealisticDream.pt: a77451e7ea075c7f72d488d2b740b3d3970c671c0ac39dd3155f3c3b129df959
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\wano_style_30100.pt: b539c85c160938fdf1a8c0187e04f6a02b76cb694cd2bd185908eb2244b94ccf
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\yamato_5000.pt: ec6dba8c4c163be7acaa493e0bce1e56d035976b133fbede19643f42d450d8fc
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\embeddings\yu-jinnobodysd15.bin: 4bfc0bae3e0b1644ed0cf4ff00f23480b86729d5e98201f6db70717d2aeee9c0
Model loaded in 56.8s (calculate hash: 25.1s, load weights from disk: 0.7s, create model: 1.2s, apply weights to model: 0.9s, apply half(): 0.9s, load VAE: 0.1s, move model to device: 11.8s, load textual inversion embeddings: 6.0s, calculate empty prompt: 9.9s).
[AddNet] Updating model hashes...
100%|███████████████████████████████████████████████████████████████████████████| 3211/3211 [00:00<00:00, 82341.31it/s]
Running on local URL:  http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Civitai: Check resources for missing info files
Civitai Link: Connecting to Civitai Link Server
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\042.safetensors: a3ecf0ef30338c4e1d5a299b2f67f77b542acb841bcdf1bb44e25414eb51dffc
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\100FHASSAN-v2.safetensors: Civitai Link: Connected to Civitai Link Server
Civitai: Check resources for missing preview images
Startup time: 225.1s (launcher: 25.4s, import torch: 5.1s, import gradio: 3.7s, setup paths: 3.0s, other imports: 3.6s, setup codeformer: 0.3s, list SD models: 0.2s, load scripts: 122.5s, create ui: 59.4s, gradio launch: 0.5s, app_started_callback: 1.4s).
Civitai Link: Joined room 1b08908b814f64f96cc8f0979bd473707c686c4eb1fa84434073d2b8c4db4b463411c7d7e00cb1d9d30202ad2f3655e7060eb42287c6bcae73f654cc19e85ece
Civitai Link: Presence update: SD: 1, Clients: 0
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\100FHASSAN-v2.safetensors: 1398033b0f6049ab2f17ff52d118923e686d6fe5c210ffc37b9a67678e00f7a1
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\103423215.safetensors: 1398033b0f6049ab2f17ff52d118923e686d6fe5c210ffc37b9a67678e00f7a1
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\103423215.safetensors: e5ab3986a41a2eb4d73128e89ee11843e29b4e86f53d454a6cbdb4dfc264444f
e5ab3986a41a2eb4d73128e89ee11843e29b4e86f53d454a6cbdb4dfc264444f
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\16dimepoch6.safetensors: Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\16dimepoch6.safetensors: 4639d4fe6ea02b5a9538efd2657ae04a60898270c3bcf5765aeefe90357e428d
4639d4fe6ea02b5a9538efd2657ae04a60898270c3bcf5765aeefe90357e428d
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\1990.safetensors: Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\1990.safetensors: 8bbe66484628df2df7e26c918187843611f82de05597144dca5558a862128ebb
8bbe66484628df2df7e26c918187843611f82de05597144dca5558a862128ebb
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\1990s.safetensors: Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\1990s.safetensors: a3057ac92c817cb92e707e22c13b3f0196eba458d5d25cada18e69e32bd6eadf
a3057ac92c817cb92e707e22c13b3f0196eba458d5d25cada18e69e32bd6eadf
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\1fate平景清.safetensors: Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\1fate平景清.safetensors: 9925add8c483fb590ffec2189f2e5fe7a767a11efd04051d2defce49f494b87e
9925add8c483fb590ffec2189f2e5fe7a767a11efd04051d2defce49f494b87e
Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\1三体动漫_庄颜.safetensors: Calculating sha256 for D:\Stable Diffusion\Automatic1111 Webui 1.51\models\Lora\1三体动漫_庄颜.safetensors: 4449657f91b9da56d639a59b93024116c5f4d55f1e1f6f16450fdf8424e97120
4449657f91b9da56d639a59b93024116c5f4d55f1e1f6f16450fdf8424e97120

Additional information

No response

dhwz commented 1 year ago

As you can see in the log this is not done by WebUI but by the additional network extension you've installed. WebUi doesn't rehash models which are already stored in cache.json