search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.28k stars 374 forks source link

[Extension]: Depthmap extension does not register itself into the API #2801

Closed aaronsantiago closed 4 months ago

aaronsantiago commented 4 months ago

Issue Description

The method that the extension uses to detect whether its working within an installation of the webui seems to not work with sdnext: https://github.com/thygate/stable-diffusion-webui-depthmap-script/blob/5250c8b7ea895a4b15575d17a278c88c38cb2132/src/backbone.py#L76

This causes it to run in standalone mode and--i assume--skip the part where it adds itself to the API.

The depth endpoint is missing from the docs and 404s

Otherwise, the extension works completely within the UI.

Version Platform Description

13:13:09-805188 INFO Starting SD.Next 13:13:09-806801 INFO Logger: file="/home/sensorium/workspace/automatic/sdnext.log" level=INFO size=565010 mode=append 13:13:09-807406 INFO Python 3.9.18 on Linux 13:13:09-815080 INFO Version: app=sd.next updated=2024-02-05 hash=e32220cc url=https://github.com/vladmandic/automatic/tree/dev 13:13:09-923478 INFO Latest published version: 9bf08c5493925b6ddb3451f6a6e00460ca92aedd 2024-01-30T14:29:51Z 13:13:09-933298 INFO Platform: arch=x86_64 cpu=x86_64 system=Linux release=6.5.0-15-generic python=3.9.18 13:13:09-937808 INFO nVidia CUDA toolkit detected: nvidia-smi present 13:13:09-949219 INFO Extensions: disabled=[] 13:13:09-949940 INFO Extensions: enabled=['stable-diffusion-webui-rembg', 'sd-webui-agent-scheduler', 'sd-extension-chainner', 'Lora', 'stable-diffusion-webui-images-browser', 'sd-extension-system-info', 'sd-webui-controlnet'] extensions-builtin 13:13:09-950760 INFO Extensions: enabled=['stable-diffusion-webui-depthmap-script', 'asymmetric-tiling-sd-webui'] extensions 13:13:09-951467 INFO Startup: quick launch 13:13:09-951852 INFO Verifying requirements 13:13:09-959459 INFO Verifying packages 13:13:09-960330 INFO Extensions: disabled=[] 13:13:09-960856 INFO Extensions: enabled=['stable-diffusion-webui-rembg', 'sd-webui-agent-scheduler', 'sd-extension-chainner', 'Lora', 'stable-diffusion-webui-images-browser', 'sd-extension-system-info', 'sd-webui-controlnet'] extensions-builtin 13:13:09-961576 INFO Extensions: enabled=['stable-diffusion-webui-depthmap-script', 'asymmetric-tiling-sd-webui'] extensions 13:13:09-963117 INFO Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0} 13:13:09-963924 INFO Command line args: ['--listen', '--docs', '--cors-origins', '', '--insecure'] insecure=True listen=True docs=True cors_origins= 13:13:11-812717 INFO Load packages: torch=2.2.0+cu121 diffusers=0.26.1 gradio=3.43.2 13:13:12-051189 INFO Engine: backend=Backend.ORIGINAL compute=cuda device=cuda attention="Scaled-Dot-Product" mode=no_grad 13:13:12-076559 INFO Device: device=NVIDIA GeForce RTX 4090 n=1 arch=sm_90 cap=(8, 9) cuda=12.1 cudnn=8902 driver=535.154.05 13:13:12-554101 INFO Available VAEs: path="models/VAE" items=0 13:13:12-555043 INFO Disabled extensions: [] 13:13:12-555952 INFO Available models: path="models/Stable-diffusion" items=2 time=0.00 13:13:12-648173 INFO LoRA networks: available=0 folders=2 13:13:12-881439 INFO Extension: script='extensions-builtin/sd-webui-agent-scheduler/scripts/task_scheduler.py' Using sqlite file: extensions-builtin/sd-webui-agent-scheduler/task_scheduler.sqlite3 13:13:12-925220 INFO Extension: script='extensions-builtin/sd-webui-controlnet/scripts/api.py' ControlNet preprocessor location: /home/sensorium/workspace/automatic/extensions-builtin/sd-webui-controlnet/annotator/ downloads 13:13:12-931628 INFO Extension: script='extensions-builtin/sd-webui-controlnet/scripts/controlnet.py' Warning: ControlNet failed to load SGM - will use LDM instead. 13:13:12-936074 INFO Extension: script='extensions-builtin/sd-webui-controlnet/scripts/hook.py' Warning: ControlNet failed to load SGM - will use LDM instead. 13:13:13-293664 INFO Extension: script='extensions/stable-diffusion-webui-depthmap-script/scripts/depthmap.py' The standalone mode is not on par with the stable-diffusion-webui mode. 13:13:13-294634 INFO Extension: script='extensions/stable-diffusion-webui-depthmap-script/scripts/depthmap.py' Some features may be missing or work differently. Please report bugs. 13:13:13-299474 INFO Extensions init time: 0.66 sd-webui-agent-scheduler=0.22 sd-webui-controlnet=0.06

13:13:17-957982 INFO LDM: DiffusionWrapper params=859.52M 13:13:17-966186 INFO Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="/home/sensorium/workspace/automatic/models/Stable-diffusion/juggernaut_final.sa fetensors" size=2034MB 13:13:18-484522 INFO Compiling pipeline=DiffusionWrapper mode=inductor 13:13:18-489682 INFO Model complilation done. 13:13:18-490794 INFO Cross-attention: optimization=Scaled-Dot-Product options=[] 13:13:18-654622 INFO Load embeddings: loaded=0 skipped=0 time=0.16 13:13:18-656994 INFO Model loaded in 3.95 create=3.23 apply=0.16 vae=0.18 move=0.18 embeddings=0.16 13:13:18-816615 INFO Model load finished: {'ram': {'used': 5.55, 'total': 31.05}, 'gpu': {'used': 2.51, 'total': 23.65}, 'retries': 0, 'oom': 0} cached=0 13:13:18-835903 INFO Startup time: 8.87 torch=1.40 gradio=0.42 libraries=0.69 samplers=0.06 extensions=0.66 face-restore=0.08 ui-img2img=0.05 ui-settings=0.20 ui-extensions=0.58 launch=0.23 app-started=0.12 checkpoint=4.13 13:13:14-547241 INFO API Docs: http://localhost:7860/docs 13:13:14-547703 INFO API ReDocs: http://localhost:7860/redocs 13:13:14-548122 INFO Initializing middleware 13:13:14-637736 INFO [AgentScheduler] Task queue is empty 13:13:14-638410 INFO [AgentScheduler] Registering APIs 13:13:14-709721 INFO Select: model="juggernaut_final [88967f03f2]" Loading model: /home/sensorium/workspace/automatic/models/Stable-diffusion/juggernaut_final.safetensors ━ … -… … 13:13:14-729763 INFO Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad fp16=True bf16=None 13:13:17-957205 INFO LDM: LatentDiffusion: mode=eps 13:13:17-957982 INFO LDM: DiffusionWrapper params=859.52M 13:13:17-966186 INFO Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="/home/sensorium/workspace/automatic/models/Stable-diffusion/juggernaut_final.sa fetensors" size=2034MB 13:13:18-484522 INFO Compiling pipeline=DiffusionWrapper mode=inductor 13:13:18-489682 INFO Model complilation done. 13:13:18-490794 INFO Cross-attention: optimization=Scaled-Dot-Product options=[] 13:13:18-654622 INFO Load embeddings: loaded=0 skipped=0 time=0.16 13:13:18-656994 INFO Model loaded in 3.95 create=3.23 apply=0.16 vae=0.18 move=0.18 embeddings=0.16 13:13:18-816615 INFO Model load finished: {'ram': {'used': 5.55, 'total': 31.05}, 'gpu': {'used': 2.51, 'total': 23.65}, 'retries': 0, 'oom': 0} cached=0 13:13:18-835903 INFO Startup time: 8.87 torch=1.40 gradio=0.42 libraries=0.69 samplers=0.06 extensions=0.66 face-restore=0.08 ui-img2img=0.05 ui-settings=0.20 ui-extensions=0.58 launch=0.23 app-started=0.12 checkpoint=4.13 13:13:14-547241 INFO API Docs: http://localhost:7860/docs 13:13:14-547703 INFO API ReDocs: http://localhost:7860/redocs 13:13:14-548122 INFO Initializing middleware 13:13:14-637736 INFO [AgentScheduler] Task queue is empty 13:13:14-638410 INFO [AgentScheduler] Registering APIs 13:13:14-709721 INFO Select: model="juggernaut_final [88967f03f2]" Loading model: /home/sensorium/workspace/automatic/models/Stable-diffusion/juggernaut_final.safetensors ━ … -… … 13:13:14-729763 INFO Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad fp16=True bf16=None 13:13:17-957205 INFO LDM: LatentDiffusion: mode=eps 13:13:17-957982 INFO LDM: DiffusionWrapper params=859.52M 13:13:17-966186 INFO Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="/home/sensorium/workspace/automatic/models/Stable-diffusion/juggernaut_final.sa fetensors" size=2034MB 13:13:18-484522 INFO Compiling pipeline=DiffusionWrapper mode=inductor 13:13:18-489682 INFO Model complilation done. 13:13:18-490794 INFO Cross-attention: optimization=Scaled-Dot-Product options=[] 13:13:18-654622 INFO Load embeddings: loaded=0 skipped=0 time=0.16 13:13:18-656994 INFO Model loaded in 3.95 create=3.23 apply=0.16 vae=0.18 move=0.18 embeddings=0.16 13:13:18-816615 INFO Model load finished: {'ram': {'used': 5.55, 'total': 31.05}, 'gpu': {'used': 2.51, 'total': 23.65}, 'retries': 0, 'oom': 0} cached=0 13:13:18-835903 INFO Startup time: 8.87 torch=1.40 gradio=0.42 libraries=0.69 samplers=0.06 extensions=0.66 face-restore=0.08 ui-img2img=0.05 ui-settings=0.20 ui-extensions=0.58 launch=0.23 app-started=0.12 checkpoint=4.13 13:13:14-547241 INFO API Docs: http://localhost:7860/docs 13:13:14-547703 INFO API ReDocs: http://localhost:7860/redocs 13:13:14-548122 INFO Initializing middleware 13:13:14-637736 INFO [AgentScheduler] Task queue is empty 13:13:14-638410 INFO [AgentScheduler] Registering APIs 13:13:14-709721 INFO Select: model="juggernaut_final [88967f03f2]" Loading model: /home/sensorium/workspace/automatic/models/Stable-diffusion/juggernaut_final.safetensors ━ … -… … 13:13:14-729763 INFO Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad fp16=True bf16=None 13:13:17-957205 INFO LDM: LatentDiffusion: mode=eps 13:13:17-957982 INFO LDM: DiffusionWrapper params=859.52M 13:13:17-966186 INFO Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="/home/sensorium/workspace/automatic/models/Stable-diffusion/juggernaut_final.sa fetensors" size=2034MB 13:13:18-484522 INFO Compiling pipeline=DiffusionWrapper mode=inductor 13:13:18-489682 INFO Model complilation done. 13:13:18-490794 INFO Cross-attention: optimization=Scaled-Dot-Product options=[] 13:13:18-654622 INFO Load embeddings: loaded=0 skipped=0 time=0.16 13:13:18-656994 INFO Model loaded in 3.95 create=3.23 apply=0.16 vae=0.18 move=0.18 embeddings=0.16 13:13:18-816615 INFO Model load finished: {'ram': {'used': 5.55, 'total': 31.05}, 'gpu': {'used': 2.51, 'total': 23.65}, 'retries': 0, 'oom': 0} cached=0 13:13:18-835903 INFO Startup time: 8.87 torch=1.40 gradio=0.42 libraries=0.69 samplers=0.06 extensions=0.66 face-restore=0.08 ui-img2img=0.05 ui-settings=0.20 ui-extensions=0.58 launch=0.23 app-started=0.12 checkpoint=4.13 stable-diffusion-webui-images-browser=0.10 stable-diffusion-webui-depthmap-script=0.26 13:13:13-307737 INFO Load UI theme: name="black-teal" style=Auto base=sdnext.css 13:13:14-546578 INFO Local URL: http://localhost:7860/ 13:13:14-547241 INFO API Docs: http://localhost:7860/docs 13:13:14-547703 INFO API ReDocs: http://localhost:7860/redocs 13:13:14-548122 INFO Initializing middleware 13:13:14-637736 INFO [AgentScheduler] Task queue is empty 13:13:14-638410 INFO [AgentScheduler] Registering APIs 13:13:14-709721 INFO Select: model="juggernaut_final [88967f03f2]" Loading model: /home/sensorium/workspace/automatic/models/Stable-diffusion/juggernaut_final.safetensors ━ … -… … 13:13:14-729763 INFO Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad fp16=True bf16=None 13:13:17-957205 INFO LDM: LatentDiffusion: mode=eps 13:13:17-957982 INFO LDM: DiffusionWrapper params=859.52M 13:13:17-966186 INFO Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="/home/sensorium/workspace/automatic/models/Stable-diffusion/juggernaut_final.sa fetensors" size=2034MB 13:13:18-484522 INFO Compiling pipeline=DiffusionWrapper mode=inductor 13:13:18-489682 INFO Model complilation done. 13:13:18-490794 INFO Cross-attention: optimization=Scaled-Dot-Product options=[] 13:13:18-654622 INFO Load embeddings: loaded=0 skipped=0 time=0.16 13:13:18-656994 INFO Model loaded in 3.95 create=3.23 apply=0.16 vae=0.18 move=0.18 embeddings=0.16 13:13:18-816615 INFO Model load finished: {'ram': {'used': 5.55, 'total': 31.05}, 'gpu': {'used': 2.51, 'total': 23.65}, 'retries': 0, 'oom': 0} cached=0 13:13:18-835903 INFO Startup time: 8.87 torch=1.40 gradio=0.42 libraries=0.69 samplers=0.06 extensions=0.66 face-restore=0.08 ui-img2img=0.05 ui-settings=0.20 ui-extensions=0.58 launch=0.23 app-started=0.12 checkpoint=4.13

URL link of the extension

https://github.com/thygate/stable-diffusion-webui-depthmap-script/tree/5250c8b7ea895a4b15575d17a278c88c38cb2132

URL link of the issue reported in the extension repository

https://github.com/thygate/stable-diffusion-webui-depthmap-script/issues/404

Acknowledgements

vladmandic commented 4 months ago

if it actually printed the exception instead of just simple string "The standalone mode is not on par with the stable-diffusion-webui mode.\n", then I could see what's incompatible. and author says "no time to fix".

aaronsantiago commented 4 months ago

i'll grab the exception for you today

aaronsantiago commented 4 months ago 
14:19:06-004398 INFO     Extension:
                         script='extensions/stable-diffusion-webui-depthmap-script/scripts/depthmap.py' cannot
                         import name 'listfiles' from 'modules.shared'
                         (/home/sensorium/workspace/automatic/modules/shared.py)

I assume the imports at the top of this section are used to "detect" the webui environment: https://github.com/thygate/stable-diffusion-webui-depthmap-script/blob/5250c8b7ea895a4b15575d17a278c88c38cb2132/src/backbone.py#L16

aaronsantiago commented 4 months ago

If there's a modification that you want me to test with I can and can get you the outputs

aaronsantiago commented 4 months ago

Not sure if there's anything to be done on the sd.next end but I was able to get it to work by commenting out the listfiles import and defining it manually (copying it from the except section below)

Perhaps listfiles is in sd.next and needs to be surfaced to extensions via this import but otherwise the extension can easily be modified to support sd.next specifically by detecting listfiles being missing and creating it separately from the rest of the import checks

aaronsantiago commented 4 months ago

For posterity--with the modifications to make it work this error reappears: https://github.com/thygate/stable-diffusion-webui-depthmap-script/issues/15

but setting sd.next to use PNG allows the endpoint to work fully

vladmandic commented 4 months ago

can you test simply adding listfiles = listdir to the end of modules/shared.py and without any other modifications?

aaronsantiago commented 4 months ago

Seemed to work perfectly for me

vladmandic commented 4 months ago

OK, I've added it to dev branch.