search

Mikubill / sd-webui-controlnet

WebUI extension for ControlNet
GNU General Public License v3.0
16.51k stars 1.91k forks source link

[Bug]: depth_zoe not working and showing errors, probably incompatibility with some other extension #1463

Closed JaysonPhoenix closed 7 months ago

JaysonPhoenix commented 1 year ago

Is there an existing issue for this?

What happened?

yesterday i updated automatic1111 and all extensions at startup and before that everything worked fine and afterwards the depth_zoe preprocessor shows errors (which i dont understand) and is stuck on loading a preview till i reload the ui. the other preprocessors including other depth preprocessors (especially midas, see additional information part) still work fine. because during that update process i had some github connection problems i thought maybe some dependency got corrupted and because my installation is an old one anyway which could profit from some cleaning i decided to make a fresh clean install of the whole stable diffusion including the virtual environment and made a backup moving the folder somewhere else, then i did the clean install and only installed controlnet and its models (without configs since they are now part of the extension) and tested the preprocessors and everything worked fine again, then i did a clean install of all extensions i had and then tested controlnet again and i got the same error. so i would guess its some kind of incompatibility that was introduced during the update i made yesterday (since i didnt install any new extensions or made any changes, only updated while restarting the ui, including installing/updating dependencies).

Steps to reproduce the problem

check pixel perfect and allow preview load picture select depth_zoe preprocessor click generate preview error and no depthmap

What should have happened?

generate depthmap instead of error ^^

Commit where the problem happens

webui:
version: v1.3.0  •  python: 3.10.6  •  torch: 2.0.1+cu118  •  xformers: N/A  •  gradio: 3.31.0  •  checkpoint: 2bd89c4fad

controlnet: v1.1.202 (main)

What browsers do you use to access the UI ?

No response

Command Line Arguments

only custom folders and --api

List of enabled extensions

active extensions:

a1111-sd-webui-lycoris a1111-sd-webui-tagcomplete Auto-Photoshop-StableDiffusion-Plugin canvas-zoom Config-Presets embedding-inspector Hypernetwork-MonkeyPatch-Extension multidiffusion-upscaler-for-automatic1111 openOutpaint-webUI-extension openpose-editor PBRemTools prompt-fusion-extension sd-dynamic-prompts sd-dynamic-thresholding sd-model-preview-xd sd-webui-3d-open-pose-editor sd-webui-additional-networks sd-webui-aspect-ratio-helper sd-webui-controlnet sd-webui-cutoff sd-webui-infinite-image-browsing sd-webui-llul sd-webui-neutral-prompt sd-webui-photopea-embed sd-webui-prompt-all-in-one sd-webui-regional-prompter stable-diffusion-webui-anti-burn Stable-Diffusion-Webui-Civitai-Helper stable-diffusion-webui-composable-lora stable-diffusion-webui-state stable-diffusion-webui-tokenizer stable-diffusion-webui-two-shot StylePile ultimate-upscale-for-automatic1111 unprompted weight_gradient

disabled extensions:

sd-web-ui-quickcss sd-webui-ar sd-webui-model-converter stable-diffusion-webui-aesthetic-gradients stable-diffusion-webui-dataset-tag-editor stable-diffusion-webui-embedding-editor stable-diffusion-webui-images-browser stable-diffusion-webui-model-toolkit

Console logs

venv "R:\Programs\Automatic1111\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.3.0
Commit hash: 20ae71faa8ef035c31aa3a410b707d792c8203a3
Installing requirements
[Auto-Photoshop-SD] Attempting auto-update...
[Auto-Photoshop-SD] switch branch to extension branch.
checkout_result: Your branch is up to date with 'origin/master'.

[Auto-Photoshop-SD] Current Branch.
branch_result: * master

[Auto-Photoshop-SD] Fetch upstream.
fetch_result:
[Auto-Photoshop-SD] Pull upstream.
pull_result: Already up to date.

No module 'xformers'. Proceeding without it.

Installing sd-webui-infinite-image-browsing requirement: python-dotenv
Installing sd-webui-infinite-image-browsing requirement: Pillow

Installing requirements for Unprompted - natural language processing: pattern@git+https://github.com/NicolasBizzozzero/pattern
Installing requirements for Unprompted - [img2pez]: sentence-transformers
Installing requirements for Unprompted - [pix2pix_zero]: salesforce-lavis
Installing requirements for Unprompted - [zoom_enhance]: color-matcher

Launching Web UI with arguments: --autolaunch --ckpt-dir R:\Workspace\Stable Diffusion\SD1.5 Stuff\Checkpoints --vae-path R:\Workspace\Stable Diffusion\SD1.5 Stuff\VAEs\vae-ft-mse-840000-ema-pruned.ckpt --api --lora-dir R:\Workspace\Stable Diffusion\SD1.5 Stuff\Loras --embeddings-dir R:\Workspace\Stable Diffusion\SD1.5 Stuff\Embeddings --hypernetwork-dir R:\Workspace\Stable Diffusion\SD1.5 Stuff\Hypernetworks --lyco-dir R:\Workspace\Stable Diffusion\SD1.5 Stuff\Loras\Lycoris
No module 'xformers'. Proceeding without it.
python_server_full_path:  R:\Programs\Automatic1111\stable-diffusion-webui\extensions\Auto-Photoshop-StableDiffusion-Plugin\server/python_server
Civitai Helper: Get Custom Model Folder
Civitai Helper: Load setting from: R:\Programs\Automatic1111\stable-diffusion-webui\extensions\Stable-Diffusion-Webui-Civitai-Helper\setting.json
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
ControlNet v1.1.202
ControlNet v1.1.202
sd-webui-prompt-all-in-one background API service started successfully.
(SETUP) Loading Unprompted v9.2.0 by Therefore Games
(SETUP) Initializing Unprompted object...
(SETUP) Loading configuration files...
(SETUP) Logging enabled for the following message types: RESULT,ERROR
Loading weights [28ced0bf89] from R:\Workspace\Stable Diffusion\SD1.5 Stuff\Checkpoints\LazyMix+ (Real Amateur Nudes) - (by kaylazy) - (v1.0 2023-02-24)\lazymixRealAmateur_v20_BakedVAE.inpainting.ckpt
Create LRU cache (max_size=16) for preprocessor results.
Creating model from config: R:\Programs\Automatic1111\stable-diffusion-webui\configs\v1-inpainting-inference.yaml
LatentInpaintDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.54 M params.
Create LRU cache (max_size=16) for preprocessor results.
Loading VAE weights from commandline argument: R:\Workspace\Stable Diffusion\SD1.5 Stuff\VAEs\vae-ft-mse-840000-ema-pruned.ckpt
Applying optimization: sdp-no-mem... done.
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
Textual inversion embeddings loaded(80): ahegaoface-7500, alyhan, ANAT0J3, angry512, anime512, anjelika, arcan3, awful512, badhandsv5-neg, beauty512, belledephine, BiL13L1SH, cherrynobodysd15, clrs, corneo_anal, corneo_covering_breasts_arms_crossed, corneo_covering_breasts_one_arm, corneo_covering_breasts_two_hands, corneo_cowgirl, corneo_nipple_rings, corneo_pov_anal, corneo_side_deepthroat, corneo_x_pasties, defiance512, dense-forest, dollienobodysd15, easynegative, em1ly-browning-t2, emb-avril, emb-babs, emwatz, follynobodysd15, footjob_positionv2, gape , glow-forest, grin512, happy512, hecavill, kristennobodysd15, laugh512, maranobodysd15, microkini, milazul97, mulpty2, ne_nataliedormer, Neg512-neg, neg_anime512, neg_facelift512, neg_hands512-neg, neg_realism512, neon-ground-astronaut, nervous512, ng_deepnegative_v1_75t, No512-neg, photozoov15, pink_blacksuit, Portrait512, povblowjob3v3, pureerosface_v1, qualityzoov15, render512, renderzoov15, rfktr_bontrex-150, rmadanegative4_sd15-neg, sad512, selfie512, shirtp5, shock512, smile512, spreadassms, structurezoov15, ulzzang-6500, underworld_selene, Unspeakable-Horrors-16v, Unspeakable-Horrors-24v, Unspeakable-Horrors-32v, Unspeakable-Horrors-48v, Unspeakable-Horrors-64v, Unspeakable-Horrors-Composition, Unspeakable-Horrors-Composition-4v
Model loaded in 48.2s (load weights from disk: 12.4s, create model: 1.7s, apply weights to model: 3.1s, apply half(): 3.0s, load VAE: 3.1s, move model to device: 1.1s, load textual inversion embeddings: 23.8s).
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Create LRU cache (max_size=16) for preprocessor results.
Startup time: 82.9s (import torch: 3.0s, import gradio: 1.9s, import ldm: 1.0s, other imports: 2.0s, setup codeformer: 0.2s, list SD models: 3.1s, load scripts: 15.5s, refresh VAE: 0.1s, create ui: 50.9s, gradio launch: 4.9s, scripts app_started_callback: 0.2s).
Pixel Perfect Mode Enabled In Preview.
resize_mode = Crop and Resize
raw_H = 1440
raw_W = 960
target_H = 512
target_W = 512
estimation = 512.0
Preview Resolution = 512
Calling preprocessor depth_zoe outside of cache.
ControlNet preprocessor location: R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
img_size [384, 512]
Traceback (most recent call last):
  File "R:\Programs\Automatic1111\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 414, in run_predict
    output = await app.get_blocks().process_api(
  File "R:\Programs\Automatic1111\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1323, in process_api
    result = await self.call_function(
  File "R:\Programs\Automatic1111\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1051, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "R:\Programs\Automatic1111\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "R:\Programs\Automatic1111\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "R:\Programs\Automatic1111\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 521, in run_annotator
    result, is_image = preprocessor(img, res=pres, thr_a=pthr_a, thr_b=pthr_b,
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 70, in decorated_func
    return cached_func(*args, **kwargs)
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 58, in cached_func
    return func(*args, **kwargs)
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\global_state.py", line 35, in unified_preprocessor
    return preprocessor_modules[preprocessor_name](*args, **kwargs)
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\processor.py", line 477, in zoe_depth
    result = model_zoe_depth(img)
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\zoe\__init__.py", line 38, in __call__
    self.load_model()
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\zoe\__init__.py", line 27, in load_model
    model = ZoeDepth.build_from_config(conf)
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\zoe\zoedepth\models\zoedepth\zoedepth_v1.py", line 250, in build_from_config
    return ZoeDepth.build(**config)
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\zoe\zoedepth\models\zoedepth\zoedepth_v1.py", line 240, in build
    core = MidasCore.build(midas_model_type=midas_model_type, use_pretrained_midas=use_pretrained_midas,
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\zoe\zoedepth\models\base_models\midas.py", line 343, in build
    midas = torch.hub.load(midas_path, midas_model_type,
  File "R:\Programs\Automatic1111\stable-diffusion-webui\venv\lib\site-packages\torch\hub.py", line 558, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "R:\Programs\Automatic1111\stable-diffusion-webui\venv\lib\site-packages\torch\hub.py", line 584, in _load_local
    hub_module = _import_module(MODULE_HUBCONF, hubconf_path)
  File "R:\Programs\Automatic1111\stable-diffusion-webui\venv\lib\site-packages\torch\hub.py", line 98, in _import_module
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\zoe\zoedepth\models\base_models\midas_repo\hubconf.py", line 5, in <module>
    from midas.dpt_depth import DPTDepthModel
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\zoe\zoedepth\models\base_models\midas_repo\midas\dpt_depth.py", line 5, in <module>
    from .blocks import (
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\zoe\zoedepth\models\base_models\midas_repo\midas\blocks.py", line 4, in <module>
    from .backbones.beit import (
  File "R:\Programs\Automatic1111\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\zoe\zoedepth\models\base_models\midas_repo\midas\backbones\beit.py", line 9, in <module>
    from timm.models.beit import gen_relative_position_index
ModuleNotFoundError: No module named 'timm.models.beit'

Additional information

i removed 2 extensions because they were throwing errors after clean stable diffusion installation seemingly because of some failing mm-something dependency installation on every restart even when the extensions are disabled (but they were working before the clean installation). i'll still mention them because they seem related, especially the depth2img one using midas and i remember reading words on their gits like beit and timm which are mentioned in the controlnet error. i also remember some extension cloned something with midas from controlnet during its installation and i would guess it was the depth2img one but i cant tell for sure, too many extensions, maybe its unrelated but maybe it helps. the 2 extensions were ddetailer and depthmap2mask

2blackbar commented 1 year ago

https://github.com/Mikubill/sd-webui-controlnet/issues/1047

JaysonPhoenix commented 1 year ago

1047

well i dont quiet understand, does that mention the extension that salesforce-lavis thing belongs to (which is only unprompted i think) wont work when i change the version to the one mentioned in that thread but controlnet zoe will work again? will it check dependencies on startup and change the timm version again so i would have to remove the unprompted extension to avoid it?

anyways, i did what was mentioned in that thread, i used pip install timm==0.6.13 which gave this warning:

Installing collected packages: timm Attempting uninstall: timm Found existing installation: timm 0.4.12 Uninstalling timm-0.4.12: Successfully uninstalled timm-0.4.12 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. salesforce-lavis 1.0.2 requires timm==0.4.12, but you have timm 0.6.7 which is incompatible. Successfully installed timm-0.6.7

then started automatic1111 which didnt mention updating to another timm version and ran controlnet again but i still got that bug (the same one i think), so that didnt work.

till i get a working solution, would it work to switch the automatic1111 folder again and do the fresh install without ever installing the unprompted extension if its only because of that salesforce thing?

2blackbar commented 1 year ago

works for me, i deleted venv, unpacked newest auto11 into folder and ran user webui bat file, zoe works , i had same issue before

JaysonPhoenix commented 1 year ago

yeah well as i described i know it works if i do a fresh install and only install controlnet but not when installing my extensions. question is which extension causes the problem and from that other thread i would guess its unprompted (and maybe depthmask2img too idk) and its salesforce dependency which installs the wrong timm if i get that right, so that would temporary help if its only that extension so i could just delete venv on the clean installation with all extensions and if no other extension installs the same thing it should work for now. still id prefer a proper solution so i can use unprompted again and since i dont really understand whats going on under the hood maybe somebody who does should talk to their devs about em upgrading their dependency if possible or whatever each side would have to do to to solve the issue.

lllyasviel commented 1 year ago

well. welcome to a1111. and i think other lib does not really call timm so it is safe to just use another version

JaysonPhoenix commented 1 year ago

i just copied the old installation of before the clean install and removed venv folder and the unprompted extension and the 2 other extensions in question i mentioned that caused errors in clean install to make sure (depthmap2img and ddetailer), because only deleting venv but keeping the extensions would just have the console reinstall the salesforce dependency with its wrong timm lib again and cause the same error. controlnet zoe works now, so it must have been at least one of those 3 extensions causing these incompatibility issues, for at least unprompted i think we can say for sure, so maybe somebody should warn on the controlnet git mainpage temporary not to install that extension till theres a proper solution.