search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.56k stars 409 forks source link

[Issue]: Very Slow Startup Time (280+ seconds) in Docker #2239

Closed lodorg closed 1 year ago

lodorg commented 1 year ago

Issue Description

I was using Docker to run SD,Next, but it takes more than 280 seconds to start up. I have seen other people's logs, the normal startup time is below 30 seconds. Any way to fix this?

Here is the startup time in logs:

06:18:11-841469 INFO     Startup time: 283.92s { torch=113.44s gradio=7.30s diffusers=11.29s libraries=17.24s vae=0.07s extensions=80.33s models=0.12s codeformer=1.73s gfpgan=0.62s upscalers=0.08s onchange=0.27s extra-networks=0.08s ui-txt2img=0.74s ui-img2img=0.37s ui-settings=0.19s ui-extensions=3.45s ui-defaults=0.07s  
                         launch=0.58s api=0.10s checkpoint=45.66s }

(As you can see, the torch and extensions itself took ~200 seconds to load)

Here is the detail of my host system:

Here is the detail of my Docker system:

Here is the full logs:

venv) vscode ➜ /workspaces/sd-playground/vladmandic-automatic (master) $ ./webui.sh --listen --insecure --debug
Create and activate python venv
Launching launch.py...
06:13:20-209413 INFO     Starting SD.Next                                                                                                                                                                                                                                                                                           
06:13:20-223065 INFO     Python 3.10.13 on Linux                                                                                                                                                                                                                                                                                    
06:13:20-575891 INFO     Version: app=sd.next updated=2023-09-20 hash=89ba8e3c url=https://github.com/vladmandic/automatic/tree/master                                                                                                                                                                                              
06:13:22-276134 INFO     Platform: arch=x86_64 cpu= system=Linux release=5.15.90.1-microsoft-standard-WSL2 python=3.10.13                                                                                                                                                                                                           
06:13:22-279177 DEBUG    Setting environment tuning                                                                                                                                                                                                                                                                                 
06:13:22-281757 DEBUG    Torch overrides: cuda=False rocm=False ipex=False diml=False openvino=False                                                                                                                                                                                                                                
06:13:22-284408 DEBUG    Torch allowed: cuda=True rocm=True ipex=True diml=True openvino=True                                                                                                                                                                                                                                       
06:13:22-289739 INFO     nVidia CUDA toolkit detected                                                                                                                                                                                                                                                                               
06:13:27-168681 DEBUG    Repository update time: Wed Sep 20 12:39:56 2023                                                                                                                                                                                                                                                           
06:13:27-170959 DEBUG    Previous setup time: Thu Sep 21 13:23:59 2023                                                                                                                                                                                                                                                              
06:13:27-174610 INFO     Extensions: disabled=['stable-diffusion-webui-images-browser', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'SwinIR', 'SwinIR', 'ScuNET', 'ScuNET', 'Lora', 'stable-diffusion-webui-rembg', 'clip-interrogator-ext', 'fantasy-full']                                                            
06:13:27-181915 INFO     Extensions: enabled=['a1111-sd-webui-lycoris', 'LDSR', 'multidiffusion-upscaler-for-automatic1111', 'sd-webui-controlnet'] extensions-builtin                                                                                                                                                              
06:13:27-367258 INFO     Extensions: enabled=['immersive-fantasy'] extensions                                                                                                                                                                                                                                                       
06:13:27-394486 DEBUG    Latest extensions time: Thu Sep 21 13:01:14 2023                                                                                                                                                                                                                                                           
06:13:27-396823 DEBUG    Timestamps: version:1695213596 setup:1695302639 extension:1695301274                                                                                                                                                                                                                                       
06:13:27-398707 INFO     No changes detected: Quick launch active                                                                                                                                                                                                                                                                   
06:13:27-400340 INFO     Verifying requirements                                                                                                                                                                                                                                                                                     
06:13:27-421470 INFO     Verifying packages                                                                                                                                                                                                                                                                                         
06:13:27-426272 INFO     Extensions: disabled=['stable-diffusion-webui-images-browser', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'SwinIR', 'SwinIR', 'ScuNET', 'ScuNET', 'Lora', 'stable-diffusion-webui-rembg', 'clip-interrogator-ext', 'fantasy-full']                                                            
06:13:27-433608 INFO     Extensions: enabled=['a1111-sd-webui-lycoris', 'LDSR', 'multidiffusion-upscaler-for-automatic1111', 'sd-webui-controlnet'] extensions-builtin                                                                                                                                                              
06:13:27-597569 INFO     Extensions: enabled=['immersive-fantasy'] extensions                                                                                                                                                                                                                                                       
06:13:27-788100 INFO     Extension preload: {'extensions-builtin': 0.14, 'extensions': 0.01}                                                                                                                                                                                                                                        
06:13:27-794059 DEBUG    Starting module: <module 'webui' from '/workspaces/sd-playground/vladmandic-automatic/webui.py'>                                                                                                                                                                                                           
06:13:27-805957 INFO     Command line args: ['--listen', '--insecure', '--debug'] insecure=True listen=True debug=True                                                                                                                                                                                                              
06:15:39-933104 DEBUG    Loaded packages: torch=2.0.1+cu118 diffusers=0.20.2 gradio=3.43.2                                                                                                                                                                                                                                          
06:15:50-800284 DEBUG    Reading: config.json len=16                                                                                                                                                                                                                                                                                
06:15:50-804503 INFO     Engine: backend=Backend.ORIGINAL compute=cuda mode=no_grad device=cuda                                                                                                                                                                                                                                     
06:15:51-068875 INFO     Device: device=NVIDIA GeForce RTX 2070 SUPER n=1 arch=sm_90 cap=(7, 5) cuda=11.8 cudnn=8700 driver=536.40                                                                                                                                                                                                  
06:15:57-174056 DEBUG    Entering start sequence                                                                                                                                                                                                                                                                                    
06:15:57-200299 DEBUG    Initializing                                                                                                                                                                                                                                                                                               
06:15:57-268563 INFO     Available VAEs: models/VAE items=0                                                                                                                                                                                                                                                                         
06:15:57-347572 DEBUG    Reading: cache.json len=2                                                                                                                                                                                                                                                                                  
06:15:57-362561 DEBUG    Reading: metadata.json len=66                                                                                                                                                                                                                                                                              
06:15:57-433240 INFO     Available models: models/Stable-diffusion items=17 time=0.10s                                                                                                                                                                                                                                              
06:15:59-788135 DEBUG    Loading extensions                                                                                                                                                                                                                                                                                         
06:17:19-762741 INFO     Extension: script='extensions-builtin/sd-webui-controlnet/scripts/controlnet.py' Warning: ControlNet failed to load SGM - will use LDM instead.                                                                                                                                                            
06:17:19-768307 INFO     Extension: script='extensions-builtin/sd-webui-controlnet/scripts/controlnet.py' ControlNet preprocessor location: /workspaces/sd-playground/vladmandic-automatic/extensions-builtin/sd-webui-controlnet/annotator/downloads                                                                               
06:17:19-873891 INFO     Extension: script='extensions-builtin/sd-webui-controlnet/scripts/hook.py' Warning: ControlNet failed to load SGM - will use LDM instead.                                                                                                                                                                  
06:17:20-121854 INFO     Extensions time: 80.33s { vladmandic-automatic=0.39s a1111-sd-webui-lycoris=74.80s LDSR=0.33s multidiffusion-upscaler-for-automatic1111=0.19s sd-webui-controlnet=4.44s immersive-fantasy=0.17s }                                                                                                          
06:17:20-185603 DEBUG    FS walk error: [Errno 2] No such file or directory: '/workspaces/sd-playground/vladmandic-automatic/models/RealESRGAN' /workspaces/sd-playground/vladmandic-automatic/models/RealESRGAN                                                                                                                    
06:17:20-196202 DEBUG    Loaded upscalers: items=11                                                                                                                                                                                                                                                                                 
06:17:20-470797 INFO     Loading UI theme: name=gradio/base style=Auto                                                                                                                                                                                                                                                              
06:17:20-486463 DEBUG    Loaded styles: folder=models/styles items=0                                                                                                                                                                                                                                                                
06:17:20-555566 DEBUG    Creating UI                                                                                                                                                                                                                                                                                                
06:17:20-840378 DEBUG    Reading: ui-config.json len=0                                                                                                                                                                                                                                                                              
06:17:21-108149 DEBUG    Extra networks: page='model' items=17 subdirs=2 tab=txt2img dirs=['models/Stable-diffusion', 'models/Diffusers', '/workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion'] time=0.25                                                                                                       
06:17:21-127434 DEBUG    Extra networks: page='style' items=0 subdirs=0 tab=txt2img dirs=['models/styles'] time=0.02                                                                                                                                                                                                                
06:17:21-152874 DEBUG    Extra networks: page='embedding' items=0 subdirs=0 tab=txt2img dirs=['models/embeddings'] time=0.02                                                                                                                                                                                                        
06:17:21-177040 DEBUG    Extra networks: page='hypernetwork' items=0 subdirs=0 tab=txt2img dirs=['models/hypernetworks'] time=0.02                                                                                                                                                                                                  
06:17:21-203313 DEBUG    Extra networks: page='lycoris' items=0 subdirs=0 tab=txt2img dirs=['models/LyCORIS'] time=0.02                                                                                                                                                                                                             
06:17:21-813562 DEBUG    Reading: ui-config.json len=0                                                                                                                                                                                                                                                                              
06:17:21-875920 INFO     Themes: builtin=6 default=5 external=53                                                                                                                                                                                                                                                                    
06:17:21-972107 DEBUG    Reading: ui-config.json len=0                                                                                                                                                                                                                                                                              
06:17:22-254686 DEBUG    Extra networks: page='model' items=17 subdirs=2 tab=txt2img dirs=['models/Stable-diffusion', 'models/Diffusers', '/workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion'] time=0.27                                                                                                       
06:17:22-266204 DEBUG    Extra networks: page='style' items=0 subdirs=0 tab=txt2img dirs=['models/styles'] time=0.01                                                                                                                                                                                                                
06:17:22-281584 DEBUG    Extra networks: page='embedding' items=0 subdirs=0 tab=txt2img dirs=['models/embeddings'] time=0.01                                                                                                                                                                                                        
06:17:22-293574 DEBUG    Extra networks: page='hypernetwork' items=0 subdirs=0 tab=txt2img dirs=['models/hypernetworks'] time=0.01                                                                                                                                                                                                  
06:17:22-305430 DEBUG    Extra networks: page='lycoris' items=0 subdirs=0 tab=txt2img dirs=['models/LyCORIS'] time=0.01                                                                                                                                                                                                             
06:17:22-418371 DEBUG    Script: 0.45s ui_tabs /workspaces/sd-playground/vladmandic-automatic/extensions/immersive-fantasy/scripts/immersive_fantasy_tab.py                                                                                                                                                                         
06:17:22-424486 DEBUG    Extensions list failed to load: /workspaces/sd-playground/vladmandic-automatic/html/extensions.json                                                                                                                                                                                                        
06:17:25-392149 DEBUG    Extension list refresh: processed=13 installed=13 enabled=5 disabled=8 visible=13 hidden=0                                                                                                                                                                                                                 
06:17:26-041847 INFO     Local URL: http://localhost:7860/                                                                                                                                                                                                                                                                          
06:17:26-045726 DEBUG    Gradio registered functions: 1410                                                                                                                                                                                                                                                                          
06:17:26-047904 INFO     Initializing middleware                                                                                                                                                                                                                                                                                    
06:17:26-071464 DEBUG    Creating API                                                                                                                                                                                                                                                                                               
06:17:26-174451 DEBUG    Scripts setup: ['X/Y/Z Grid:0.011s', 'controlnet m2m:0.006s', 'Tiled Diffusion:0.059s', 'Tiled VAE:0.005s', 'ControlNet:0.354s']                                                                                                                                                                           
06:17:26-176567 DEBUG    Model metadata: metadata.json no changes                                                                                                                                                                                                                                                                   
06:17:26-179054 DEBUG    Select checkpoint: model="meinahentai_v4.safetensors [8145104977]"                                                                                                                                                                                                                                         
06:17:26-182236 DEBUG    Load model weights: existing=False target=/workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion/meinahentai_v4.safetensors info=None                                                                                                                                                      
Loading weights: /workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion/meinahentai_v4.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.1 GB -:--:--
06:17:28-071696 DEBUG    Load model: name=/workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion/meinahentai_v4.safetensors dict=True                                                                                                                                                                               
06:17:28-343821 DEBUG    Desired Torch parameters: dtype=FP16 no-half=False no-half-vae=False upscast=False                                                                                                                                                                                                                         
06:17:28-346100 INFO     Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad fp16=True bf16=False                                                                                                                                                                        
06:17:28-348797 DEBUG    Model dict loaded: {'ram': {'used': 1.21, 'total': 23.45}, 'gpu': {'used': 1.05, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                                    
06:17:28-369612 DEBUG    Model config loaded: {'ram': {'used': 1.21, 'total': 23.45}, 'gpu': {'used': 1.05, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                                  
06:17:29-234265 INFO     LDM: LatentDiffusion: Running in eps-prediction mode                                                                                                                                                                                                                                                       
06:17:29-236727 INFO     LDM: DiffusionWrapper has 859.52 M params.                                                                                                                                                                                                                                                                 
06:17:29-238757 DEBUG    Model created from config: /workspaces/sd-playground/vladmandic-automatic/configs/v1-inference.yaml                                                                                                                                                                                                        
06:17:29-240941 DEBUG    Model autodetect model: /workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion/meinahentai_v4.safetensors pipeline=Stable Diffusion size=1.99 GB                                                                                                                                           
06:17:29-244351 DEBUG    Model weights loading: {'ram': {'used': 2.23, 'total': 23.45}, 'gpu': {'used': 1.05, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                                
06:18:10-437602 DEBUG    Model weights loaded: {'ram': {'used': 6.04, 'total': 23.45}, 'gpu': {'used': 1.05, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                                 
06:18:11-082951 DEBUG    Model weights moved: {'ram': {'used': 5.61, 'total': 23.45}, 'gpu': {'used': 3.08, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                                  
06:18:11-092941 INFO     Cross-attention: optimization=Scaled-Dot-Product options=[]                                                                                                                                                                                                                                                
06:18:11-393384 INFO     Loaded embeddings: loaded=0 skipped=0                                                                                                                                                                                                                                                                      
06:18:11-398859 INFO     Model loaded in 45.21s { load=1.89s config=0.30s create=0.87s apply=40.67s vae=0.52s move=0.64s embeddings=0.30s }                                                                                                                                                                                         
06:18:11-671367 DEBUG    gc: collected=2823 device=cuda {'ram': {'used': 5.61, 'total': 23.45}, 'gpu': {'used': 3.08, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                        
06:18:11-674506 INFO     Model load finished: {'ram': {'used': 5.61, 'total': 23.45}, 'gpu': {'used': 3.08, 'total': 8.0}, 'retries': 0, 'oom': 0} cached=0                                                                                                                                                                         
06:18:11-836176 DEBUG    Saving: config.json len=1120                                                                                                                                                                                                                                                                               
06:18:11-841469 INFO     Startup time: 283.92s { torch=113.44s gradio=7.30s diffusers=11.29s libraries=17.24s vae=0.07s extensions=80.33s models=0.12s codeformer=1.73s gfpgan=0.62s upscalers=0.08s onchange=0.27s extra-networks=0.08s ui-txt2img=0.74s ui-img2img=0.37s ui-settings=0.19s ui-extensions=3.45s ui-defaults=0.07s  
                         launch=0.58s api=0.10s checkpoint=45.66s }

Version Platform Description

Here is the detail of my host system:

Here is the detail of my Docker system:

Relevant log output

venv) vscode ➜ /workspaces/sd-playground/vladmandic-automatic (master) $ ./webui.sh --listen --insecure --debug
Create and activate python venv
Launching launch.py...
06:13:20-209413 INFO     Starting SD.Next                                                                                                                                                                                                                                                                                           
06:13:20-223065 INFO     Python 3.10.13 on Linux                                                                                                                                                                                                                                                                                    
06:13:20-575891 INFO     Version: app=sd.next updated=2023-09-20 hash=89ba8e3c url=https://github.com/vladmandic/automatic/tree/master                                                                                                                                                                                              
06:13:22-276134 INFO     Platform: arch=x86_64 cpu= system=Linux release=5.15.90.1-microsoft-standard-WSL2 python=3.10.13                                                                                                                                                                                                           
06:13:22-279177 DEBUG    Setting environment tuning                                                                                                                                                                                                                                                                                 
06:13:22-281757 DEBUG    Torch overrides: cuda=False rocm=False ipex=False diml=False openvino=False                                                                                                                                                                                                                                
06:13:22-284408 DEBUG    Torch allowed: cuda=True rocm=True ipex=True diml=True openvino=True                                                                                                                                                                                                                                       
06:13:22-289739 INFO     nVidia CUDA toolkit detected                                                                                                                                                                                                                                                                               
06:13:27-168681 DEBUG    Repository update time: Wed Sep 20 12:39:56 2023                                                                                                                                                                                                                                                           
06:13:27-170959 DEBUG    Previous setup time: Thu Sep 21 13:23:59 2023                                                                                                                                                                                                                                                              
06:13:27-174610 INFO     Extensions: disabled=['stable-diffusion-webui-images-browser', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'SwinIR', 'SwinIR', 'ScuNET', 'ScuNET', 'Lora', 'stable-diffusion-webui-rembg', 'clip-interrogator-ext', 'fantasy-full']                                                            
06:13:27-181915 INFO     Extensions: enabled=['a1111-sd-webui-lycoris', 'LDSR', 'multidiffusion-upscaler-for-automatic1111', 'sd-webui-controlnet'] extensions-builtin                                                                                                                                                              
06:13:27-367258 INFO     Extensions: enabled=['immersive-fantasy'] extensions                                                                                                                                                                                                                                                       
06:13:27-394486 DEBUG    Latest extensions time: Thu Sep 21 13:01:14 2023                                                                                                                                                                                                                                                           
06:13:27-396823 DEBUG    Timestamps: version:1695213596 setup:1695302639 extension:1695301274                                                                                                                                                                                                                                       
06:13:27-398707 INFO     No changes detected: Quick launch active                                                                                                                                                                                                                                                                   
06:13:27-400340 INFO     Verifying requirements                                                                                                                                                                                                                                                                                     
06:13:27-421470 INFO     Verifying packages                                                                                                                                                                                                                                                                                         
06:13:27-426272 INFO     Extensions: disabled=['stable-diffusion-webui-images-browser', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'SwinIR', 'SwinIR', 'ScuNET', 'ScuNET', 'Lora', 'stable-diffusion-webui-rembg', 'clip-interrogator-ext', 'fantasy-full']                                                            
06:13:27-433608 INFO     Extensions: enabled=['a1111-sd-webui-lycoris', 'LDSR', 'multidiffusion-upscaler-for-automatic1111', 'sd-webui-controlnet'] extensions-builtin                                                                                                                                                              
06:13:27-597569 INFO     Extensions: enabled=['immersive-fantasy'] extensions                                                                                                                                                                                                                                                       
06:13:27-788100 INFO     Extension preload: {'extensions-builtin': 0.14, 'extensions': 0.01}                                                                                                                                                                                                                                        
06:13:27-794059 DEBUG    Starting module: <module 'webui' from '/workspaces/sd-playground/vladmandic-automatic/webui.py'>                                                                                                                                                                                                           
06:13:27-805957 INFO     Command line args: ['--listen', '--insecure', '--debug'] insecure=True listen=True debug=True                                                                                                                                                                                                              
06:15:39-933104 DEBUG    Loaded packages: torch=2.0.1+cu118 diffusers=0.20.2 gradio=3.43.2                                                                                                                                                                                                                                          
06:15:50-800284 DEBUG    Reading: config.json len=16                                                                                                                                                                                                                                                                                
06:15:50-804503 INFO     Engine: backend=Backend.ORIGINAL compute=cuda mode=no_grad device=cuda                                                                                                                                                                                                                                     
06:15:51-068875 INFO     Device: device=NVIDIA GeForce RTX 2070 SUPER n=1 arch=sm_90 cap=(7, 5) cuda=11.8 cudnn=8700 driver=536.40                                                                                                                                                                                                  
06:15:57-174056 DEBUG    Entering start sequence                                                                                                                                                                                                                                                                                    
06:15:57-200299 DEBUG    Initializing                                                                                                                                                                                                                                                                                               
06:15:57-268563 INFO     Available VAEs: models/VAE items=0                                                                                                                                                                                                                                                                         
06:15:57-347572 DEBUG    Reading: cache.json len=2                                                                                                                                                                                                                                                                                  
06:15:57-362561 DEBUG    Reading: metadata.json len=66                                                                                                                                                                                                                                                                              
06:15:57-433240 INFO     Available models: models/Stable-diffusion items=17 time=0.10s                                                                                                                                                                                                                                              
06:15:59-788135 DEBUG    Loading extensions                                                                                                                                                                                                                                                                                         
06:17:19-762741 INFO     Extension: script='extensions-builtin/sd-webui-controlnet/scripts/controlnet.py' Warning: ControlNet failed to load SGM - will use LDM instead.                                                                                                                                                            
06:17:19-768307 INFO     Extension: script='extensions-builtin/sd-webui-controlnet/scripts/controlnet.py' ControlNet preprocessor location: /workspaces/sd-playground/vladmandic-automatic/extensions-builtin/sd-webui-controlnet/annotator/downloads                                                                               
06:17:19-873891 INFO     Extension: script='extensions-builtin/sd-webui-controlnet/scripts/hook.py' Warning: ControlNet failed to load SGM - will use LDM instead.                                                                                                                                                                  
06:17:20-121854 INFO     Extensions time: 80.33s { vladmandic-automatic=0.39s a1111-sd-webui-lycoris=74.80s LDSR=0.33s multidiffusion-upscaler-for-automatic1111=0.19s sd-webui-controlnet=4.44s immersive-fantasy=0.17s }                                                                                                          
06:17:20-185603 DEBUG    FS walk error: [Errno 2] No such file or directory: '/workspaces/sd-playground/vladmandic-automatic/models/RealESRGAN' /workspaces/sd-playground/vladmandic-automatic/models/RealESRGAN                                                                                                                    
06:17:20-196202 DEBUG    Loaded upscalers: items=11                                                                                                                                                                                                                                                                                 
06:17:20-470797 INFO     Loading UI theme: name=gradio/base style=Auto                                                                                                                                                                                                                                                              
06:17:20-486463 DEBUG    Loaded styles: folder=models/styles items=0                                                                                                                                                                                                                                                                
06:17:20-555566 DEBUG    Creating UI                                                                                                                                                                                                                                                                                                
06:17:20-840378 DEBUG    Reading: ui-config.json len=0                                                                                                                                                                                                                                                                              
06:17:21-108149 DEBUG    Extra networks: page='model' items=17 subdirs=2 tab=txt2img dirs=['models/Stable-diffusion', 'models/Diffusers', '/workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion'] time=0.25                                                                                                       
06:17:21-127434 DEBUG    Extra networks: page='style' items=0 subdirs=0 tab=txt2img dirs=['models/styles'] time=0.02                                                                                                                                                                                                                
06:17:21-152874 DEBUG    Extra networks: page='embedding' items=0 subdirs=0 tab=txt2img dirs=['models/embeddings'] time=0.02                                                                                                                                                                                                        
06:17:21-177040 DEBUG    Extra networks: page='hypernetwork' items=0 subdirs=0 tab=txt2img dirs=['models/hypernetworks'] time=0.02                                                                                                                                                                                                  
06:17:21-203313 DEBUG    Extra networks: page='lycoris' items=0 subdirs=0 tab=txt2img dirs=['models/LyCORIS'] time=0.02                                                                                                                                                                                                             
06:17:21-813562 DEBUG    Reading: ui-config.json len=0                                                                                                                                                                                                                                                                              
06:17:21-875920 INFO     Themes: builtin=6 default=5 external=53                                                                                                                                                                                                                                                                    
06:17:21-972107 DEBUG    Reading: ui-config.json len=0                                                                                                                                                                                                                                                                              
06:17:22-254686 DEBUG    Extra networks: page='model' items=17 subdirs=2 tab=txt2img dirs=['models/Stable-diffusion', 'models/Diffusers', '/workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion'] time=0.27                                                                                                       
06:17:22-266204 DEBUG    Extra networks: page='style' items=0 subdirs=0 tab=txt2img dirs=['models/styles'] time=0.01                                                                                                                                                                                                                
06:17:22-281584 DEBUG    Extra networks: page='embedding' items=0 subdirs=0 tab=txt2img dirs=['models/embeddings'] time=0.01                                                                                                                                                                                                        
06:17:22-293574 DEBUG    Extra networks: page='hypernetwork' items=0 subdirs=0 tab=txt2img dirs=['models/hypernetworks'] time=0.01                                                                                                                                                                                                  
06:17:22-305430 DEBUG    Extra networks: page='lycoris' items=0 subdirs=0 tab=txt2img dirs=['models/LyCORIS'] time=0.01                                                                                                                                                                                                             
06:17:22-418371 DEBUG    Script: 0.45s ui_tabs /workspaces/sd-playground/vladmandic-automatic/extensions/immersive-fantasy/scripts/immersive_fantasy_tab.py                                                                                                                                                                         
06:17:22-424486 DEBUG    Extensions list failed to load: /workspaces/sd-playground/vladmandic-automatic/html/extensions.json                                                                                                                                                                                                        
06:17:25-392149 DEBUG    Extension list refresh: processed=13 installed=13 enabled=5 disabled=8 visible=13 hidden=0                                                                                                                                                                                                                 
06:17:26-041847 INFO     Local URL: http://localhost:7860/                                                                                                                                                                                                                                                                          
06:17:26-045726 DEBUG    Gradio registered functions: 1410                                                                                                                                                                                                                                                                          
06:17:26-047904 INFO     Initializing middleware                                                                                                                                                                                                                                                                                    
06:17:26-071464 DEBUG    Creating API                                                                                                                                                                                                                                                                                               
06:17:26-174451 DEBUG    Scripts setup: ['X/Y/Z Grid:0.011s', 'controlnet m2m:0.006s', 'Tiled Diffusion:0.059s', 'Tiled VAE:0.005s', 'ControlNet:0.354s']                                                                                                                                                                           
06:17:26-176567 DEBUG    Model metadata: metadata.json no changes                                                                                                                                                                                                                                                                   
06:17:26-179054 DEBUG    Select checkpoint: model="meinahentai_v4.safetensors [8145104977]"                                                                                                                                                                                                                                         
06:17:26-182236 DEBUG    Load model weights: existing=False target=/workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion/meinahentai_v4.safetensors info=None                                                                                                                                                      
Loading weights: /workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion/meinahentai_v4.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.1 GB -:--:--
06:17:28-071696 DEBUG    Load model: name=/workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion/meinahentai_v4.safetensors dict=True                                                                                                                                                                               
06:17:28-343821 DEBUG    Desired Torch parameters: dtype=FP16 no-half=False no-half-vae=False upscast=False                                                                                                                                                                                                                         
06:17:28-346100 INFO     Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad fp16=True bf16=False                                                                                                                                                                        
06:17:28-348797 DEBUG    Model dict loaded: {'ram': {'used': 1.21, 'total': 23.45}, 'gpu': {'used': 1.05, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                                    
06:17:28-369612 DEBUG    Model config loaded: {'ram': {'used': 1.21, 'total': 23.45}, 'gpu': {'used': 1.05, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                                  
06:17:29-234265 INFO     LDM: LatentDiffusion: Running in eps-prediction mode                                                                                                                                                                                                                                                       
06:17:29-236727 INFO     LDM: DiffusionWrapper has 859.52 M params.                                                                                                                                                                                                                                                                 
06:17:29-238757 DEBUG    Model created from config: /workspaces/sd-playground/vladmandic-automatic/configs/v1-inference.yaml                                                                                                                                                                                                        
06:17:29-240941 DEBUG    Model autodetect model: /workspaces/sd-playground/vladmandic-automatic/models/Stable-diffusion/meinahentai_v4.safetensors pipeline=Stable Diffusion size=1.99 GB                                                                                                                                           
06:17:29-244351 DEBUG    Model weights loading: {'ram': {'used': 2.23, 'total': 23.45}, 'gpu': {'used': 1.05, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                                
06:18:10-437602 DEBUG    Model weights loaded: {'ram': {'used': 6.04, 'total': 23.45}, 'gpu': {'used': 1.05, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                                 
06:18:11-082951 DEBUG    Model weights moved: {'ram': {'used': 5.61, 'total': 23.45}, 'gpu': {'used': 3.08, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                                  
06:18:11-092941 INFO     Cross-attention: optimization=Scaled-Dot-Product options=[]                                                                                                                                                                                                                                                
06:18:11-393384 INFO     Loaded embeddings: loaded=0 skipped=0                                                                                                                                                                                                                                                                      
06:18:11-398859 INFO     Model loaded in 45.21s { load=1.89s config=0.30s create=0.87s apply=40.67s vae=0.52s move=0.64s embeddings=0.30s }                                                                                                                                                                                         
06:18:11-671367 DEBUG    gc: collected=2823 device=cuda {'ram': {'used': 5.61, 'total': 23.45}, 'gpu': {'used': 3.08, 'total': 8.0}, 'retries': 0, 'oom': 0}                                                                                                                                                                        
06:18:11-674506 INFO     Model load finished: {'ram': {'used': 5.61, 'total': 23.45}, 'gpu': {'used': 3.08, 'total': 8.0}, 'retries': 0, 'oom': 0} cached=0                                                                                                                                                                         
06:18:11-836176 DEBUG    Saving: config.json len=1120                                                                                                                                                                                                                                                                               
06:18:11-841469 INFO     Startup time: 283.92s { torch=113.44s gradio=7.30s diffusers=11.29s libraries=17.24s vae=0.07s extensions=80.33s models=0.12s codeformer=1.73s gfpgan=0.62s upscalers=0.08s onchange=0.27s extra-networks=0.08s ui-txt2img=0.74s ui-img2img=0.37s ui-settings=0.19s ui-extensions=3.45s ui-defaults=0.07s  
                         launch=0.58s api=0.10s checkpoint=45.66s }

Backend

Original

Model

SD 1.5

Acknowledgements

vladmandic commented 1 year ago

are you using wsl2 (i'm guessing since your docker host is windows)? docker inside wsl2 inside windows is really not optimized up to a point where random seek latency becomes an issue. for example, you say you have an ssd but model load takes 45s - it should take 6-9sec tops. other than that, really search on the net how to optimize docker on windows, this is not sdnext specific item - its clear that any load operation is extremely slow on your system.

lodorg commented 1 year ago

Yep docker with wsl2... Thanks for the insights

vladmandic commented 1 year ago

i'll have to close this as there is not much i can do here to help - issue is storage layer in this case is just working with too much latency (regardless if you have fastest ssd, latency is due to multiple unoptimized software translations).