vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.48k stars 400 forks source link

[Issue]: AMD GPU (rx7600) on a ubuntu 20.04.5 with pyton 3.10.12 not used #1973

Closed ConfusedMerlin closed 1 year ago

ConfusedMerlin commented 1 year ago

Issue Description

Tried to install vladmandic's automatic on an Ubuntu yesterday to see, if the ROCm backend performs better than automatics1111's openML on windows.

It... kind of worked. After a lot of problems with the Python 3.8/3.10 versions, it finally started. I immediately issued a 512x512 test image (happy cat sitting on a computer), but was a bit disappointed when it claimed to need 4 Minutes to do it. The image appeared after said time.

Which is 7 times the windows openML counterpart needed. but then the CPU fan gave away, that not the GPU was thinking, but the CPU. The system monitor agreed with that observation, when it showed pretty graphs for all my cpu cores above 50%. This was astounding and concerning at the same time.

Astounding, because the openML automatic1111 version estimated 40m+ for that test image with CPU backend and clogged up my CPU with next to 100% for each core; your version had each core around 60% with a lot of fluctuations. Concerning, because I realized that the GPU was idle the whole time. Looking at the systeminfo page (thanks for including that!) I realized that the backend in use was called CPU.

I looked around the interwebs a bit; somebody here posted a similar issue some time ago (https://github.com/vladmandic/automatic/issues/816), but failed to offer the required log files. But there were some instructions inside this ticket, like "remove venv, delete setup.log". Which I did.

While I had a hickup at one try, where it failed to find the CLIP thingy (this didn't happen the next time), this does not resolve the issue. Also, there is no setup.log, as far as I can remember.

Still, the output during the startup sounds kind of promising, as it says "rocm roolkit detected" and stuff like that. But even with the --use-rocm switch, it falls back to GPU without a highly visible error message around.

As far as I can tell, the GPU should be ready to use; its kernel moduls are compiled and activated. But this being the first time I try to get an AMD GPU to run on Linux, I may draw wrong conclusions about this. But if you google "check if AMD GPU works on ubuntu", all answers are about "doing lspci" and stuff, which did after the drivers claimed to be installed. But I guess if you have a dedicated "try to to check if it works" test at hand, I will do that one too.

Finally... I am sorry, but I cannot offer logs right now. The test system being a new one, I managed to forget my gitlab pw yesterday evening, until gitlab locked the ip... Now I am at work, where I cannot access the test system (but the pw manager knows my password) I will add it to this ticket later this day.

Version Platform Description

ubuntu 20.04.5 (tried a 22.04 first, but the gpu driver installation failed... very hard; not your problem) python 3.10.12 (from that inofficial repo, with fitting pip, also keeping the 3.8 as alternative for ubuntu) radeon rx 7600, 23.10.3 for Ubuntu 20.04.5 HWE (see https://www.amd.com/en/support/linux-drivers) the firefiox that comes with ubuntu 20.04.5 (dunno which version that is)

the vladmandic is cloned fresh (yesterday evening), and the webui.sh seemed to have no problems getting its stuff.

Relevant log output

Create and activate python venv
Launching launch.py...
16:13:18-400774 INFO     Starting SD.Next                                       
16:13:18-404421 INFO     Python 3.10.12 on Linux                                
16:13:18-418094 INFO     Version: 417ef540 Tue Aug 8 12:05:30 2023 -0400        
16:13:18-424700 INFO     AMD ROCm toolkit detected                              
16:13:18-426475 INFO     Installing package: torch==2.0.1 torchvision==0.15.2   
                         --index-url https://download.pytorch.org/whl/rocm5.4.2 
16:14:26-590914 WARNING  Modified files: ['webui.sh']                           
16:14:26-596262 INFO     Verifying requirements                                 
16:14:26-598947 INFO     Installing package: addict                             
16:14:27-292499 INFO     Installing package: aenum                              
16:14:28-045902 INFO     Installing package: aiohttp                            
16:14:29-940152 INFO     Installing package: anyio                              
16:14:30-755362 INFO     Installing package: appdirs                            
16:14:31-403205 INFO     Installing package: astunparse                         
16:14:32-236579 INFO     Installing package: bitsandbytes                       
16:14:34-554593 INFO     Installing package: blendmodes                         
16:14:35-334493 INFO     Installing package: clean-fid                          
16:14:39-099783 INFO     Installing package: easydev                            
16:14:40-158669 INFO     Installing package: extcolors                          
16:14:41-108496 INFO     Installing package: facexlib                           
16:14:48-915052 INFO     Installing package: filetype                           
16:14:49-700933 INFO     Installing package: future                             
16:14:50-668659 INFO     Installing package: gdown                              
16:14:51-689222 INFO     Installing package: gfpgan                             
16:14:58-705860 INFO     Installing package: GitPython                          
16:14:59-907192 INFO     Installing package: httpcore                           
16:15:00-975424 INFO     Installing package: inflection                         
16:15:01-899175 INFO     Installing package: jsonmerge                          
16:15:03-331173 INFO     Installing package: kornia                             
16:15:04-671572 INFO     Installing package: lark                               
16:15:05-712136 INFO     Installing package: lmdb                               
16:15:06-702318 INFO     Installing package: lpips                              
16:15:07-729868 INFO     Installing package: omegaconf                          
16:15:08-959674 INFO     Installing package: open-clip-torch                    
16:15:12-999176 INFO     Installing package: opencv-contrib-python-headless     
16:15:24-136080 INFO     Installing package: piexif                             
16:15:25-296931 INFO     Installing package: psutil                             
16:15:26-620939 INFO     Installing package: pyyaml                             
16:15:27-757667 INFO     Installing package: realesrgan                         
16:15:29-131372 INFO     Installing package: resize-right                       
16:15:30-244588 INFO     Installing package: rich                               
16:15:31-414002 INFO     Installing package: safetensors                        
16:15:32-574226 INFO     Installing package: scipy                              
16:15:33-819556 INFO     Installing package: tb_nightly                         
16:15:35-448721 INFO     Installing package: toml                               
16:15:36-608949 INFO     Installing package: torchdiffeq                        
16:15:37-776375 INFO     Installing package: torchsde                           
16:15:39-103227 INFO     Installing package: voluptuous                         
16:15:40-262033 INFO     Installing package: yapf                               
16:15:41-396815 INFO     Installing package: scikit-image                       
16:15:42-603857 INFO     Installing package: basicsr                            
16:15:43-844599 INFO     Installing package: compel                             
16:15:49-766072 INFO     Installing package: fasteners                          
16:15:51-318559 INFO     Installing package: typing-extensions==4.7.1           
16:15:53-027547 INFO     Installing package: antlr4-python3-runtime==4.9.3      
16:15:54-537777 INFO     Installing package: requests==2.31.0                   
16:15:56-177455 INFO     Installing package: tqdm==4.65.0                       
16:15:57-818081 INFO     Installing package: accelerate==0.20.3                 
16:15:59-542304 INFO     Installing package: opencv-python-headless==4.7.0.72   
16:16:02-098483 INFO     Installing package: diffusers==0.19.3                  
16:16:03-675325 INFO     Installing package: einops==0.4.1                      
16:16:05-258442 INFO     Installing package: gradio==3.32.0                     
16:16:14-776778 INFO     Installing package: huggingface_hub==0.16.4            
16:16:16-533080 INFO     Installing package: numexpr==2.8.4                     
16:16:18-375387 INFO     Installing package: numpy==1.23.5                      
16:16:21-994259 INFO     Installing package: numba==0.57.0                      
16:16:25-601736 INFO     Installing package: pandas==1.5.3                      
16:16:30-781894 INFO     Installing package: protobuf==3.20.3                   
16:16:32-690203 INFO     Installing package: pytorch_lightning==1.9.4           
16:16:35-452447 INFO     Installing package: transformers==4.31.0               
16:16:37-357468 INFO     Installing package: tomesd==0.1.3                      
16:16:39-344207 INFO     Installing package: urllib3==1.26.15                   
16:16:41-289599 INFO     Installing package: Pillow==9.5.0                      
16:16:43-689820 INFO     Installing package: timm==0.6.13                       
16:16:45-948046 INFO     Installing package: pydantic==1.10.11                  
16:16:48-125905 INFO     Verifying packages                                     
16:16:48-127152 INFO     Installing package:                                    
                         git+https://github.com/openai/CLIP.git                 
16:16:52-070316 INFO     Installing package:                                    
                         git+https://github.com/patrickvonplaten/invisible-water
                         mark.git@remove_onnxruntime_depedency                  
16:16:58-555656 INFO     Installing package: onnxruntime==1.15.1                
16:17:01-026876 INFO     Installing package: pi-heif                            
16:17:03-407166 INFO     Installing package: tensorflow-rocm                    
16:17:22-051364 INFO     Verifying repositories                                 
16:17:23-533092 INFO     Verifying submodules                                   
16:17:29-217792 INFO     Extension installed packages:                          
                         stable-diffusion-webui-rembg ['rembg==2.0.38',         
                         'pooch==1.7.0', 'PyMatting==1.1.8']                    
16:17:31-405299 INFO     Extension installed packages:                          
                         stable-diffusion-webui-images-browser                  
                         ['Send2Trash==1.8.2']                                  
16:17:42-802950 INFO     Extension installed packages: sd-webui-controlnet      
                         ['lxml==4.9.3', 'opencv-contrib-python==4.8.0.76',     
                         'reportlab==4.0.4', 'pycparser==2.21',                 
                         'portalocker==2.7.0', 'cffi==1.15.1', 'svglib==1.5.1', 
                         'tinycss2==1.2.1', 'mediapipe==0.10.3',                
                         'tabulate==0.9.0', 'cssselect2==0.7.0',                
                         'webencodings==0.5.1', 'sounddevice==0.4.6',           
                         'iopath==0.1.9', 'yacs==0.1.8',                        
                         'fvcore==0.1.5.post20221221']                          
16:17:46-451774 INFO     Extension installed packages: sd-webui-agent-scheduler 
                         ['SQLAlchemy==2.0.19', 'greenlet==2.0.2']              
16:17:49-072610 INFO     Extension installed packages: clip-interrogator-ext    
                         ['clip-interrogator==0.6.0']                           
16:17:49-149886 INFO     Extensions enabled:                                    
                         ['multidiffusion-upscaler-for-automatic1111',          
                         'stable-diffusion-webui-rembg', 'LDSR', 'Lora',        
                         'stable-diffusion-webui-images-browser',               
                         'sd-webui-controlnet', 'ScuNET',                       
                         'sd-webui-agent-scheduler', 'sd-extension-system-info',
                         'sd-dynamic-thresholding', 'clip-interrogator-ext',    
                         'SwinIR', 'a1111-sd-webui-lycoris']                    
16:17:49-151285 INFO     Verifying packages                                     
16:17:49-152917 INFO     Installing package: tensorflow-rocm                    
16:17:51-272046 INFO     Extension preload: 0.0s                                
                         /opt/ai/automatic/extensions-builtin                   
16:17:51-273821 INFO     Extension preload: 0.0s /opt/ai/automatic/extensions   
16:17:51-290690 INFO     Server arguments: []                                   
No module 'xformers'. Proceeding without it.
16:17:56-493034 INFO     Pipeline: Backend.ORIGINAL                             
16:17:56-806918 INFO     Libraries loaded                                       
16:17:56-808190 INFO     Using data path: /opt/ai/automatic                     
16:17:56-809781 INFO     Available VAEs: /opt/ai/automatic/models/VAE 0         
16:17:56-812199 INFO     Available models:                                      
                         /opt/ai/automatic/models/Stable-diffusion 1            
16:17:58-541250 INFO     ControlNet v1.1.234                                    
ControlNet v1.1.234
ControlNet preprocessor location: /opt/ai/automatic/extensions-builtin/sd-webui-controlnet/annotator/downloads
16:17:58-736080 INFO     ControlNet v1.1.234                                    
ControlNet v1.1.234
16:18:15-376578 INFO     Loading UI theme: name=black-orange style=Auto         
Running on local URL:  http://127.0.0.1:7860
16:18:16-446889 INFO     Local URL: http://127.0.0.1:7860/                      
16:18:16-448551 INFO     Initializing middleware                                
16:18:17-020224 INFO     [AgentScheduler] Task queue is empty                   
16:18:17-021243 INFO     [AgentScheduler] Registering APIs                      
Loading weights: /opt/ai/automatic/models/Stable-diffusion/v1-5-pruned-emaonly… 
16:18:18-814744 INFO     Torch override dtype: no-half set                      
16:18:18-815960 INFO     Torch override VAE dtype: no-half set                  
16:18:18-816998 INFO     Setting Torch parameters: dtype=torch.float32          
                         vae=torch.float32 unet=torch.float32                   
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
16:18:23-435481 INFO     Applying Doggettx cross attention optimization         
16:18:23-441643 INFO     Embeddings: loaded=0 skipped=0                         
16:18:23-447903 INFO     Model loaded in 5.9s (load=1.3s create=0.5s apply=4.1s)
16:18:23-846156 INFO     Model load finished: {'ram': {'used': 9.36, 'total':   
                         31.24}} cached=0                                       
16:18:24-294232 INFO     Startup time: 33.0s (torch=3.9s gradio=0.5s            
                         libraries=1.1s scripts=18.4s onchange=0.1s             
                         ui-txt2img=0.1s ui-img2img=0.1s ui-settings=0.1s       
                         ui-extensions=0.4s ui-defaults=0.1s launch=0.2s        
                         app-started=0.7s checkpoint=7.1s) 

the changed webui.sh contains now this line (instead of only python3, which points to python 3.8, which was declared unsuppored somewhere during my first installation tries)

python_cmd="python3.10"
2023-08-09 16:13:18,400 | sd | INFO | launch | Starting SD.Next
2023-08-09 16:13:18,404 | sd | INFO | installer | Python 3.10.12 on Linux
2023-08-09 16:13:18,418 | sd | INFO | installer | Version: 417ef540 Tue Aug 8 12:05:30 2023 -0400
2023-08-09 16:13:18,422 | sd | DEBUG | installer | Setting environment tuning
2023-08-09 16:13:18,423 | sd | DEBUG | installer | Torch overrides: cuda=False rocm=False ipex=False diml=False
2023-08-09 16:13:18,423 | sd | DEBUG | installer | Torch allowed: cuda=True rocm=True ipex=True diml=True
2023-08-09 16:13:18,424 | sd | INFO | installer | AMD ROCm toolkit detected
2023-08-09 16:13:18,426 | sd | DEBUG | installer | Package version not found: torch
2023-08-09 16:13:18,426 | sd | DEBUG | installer | Package version not found: torchvision
2023-08-09 16:13:18,426 | sd | INFO | installer | Installing package: torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/rocm5.4.2
2023-08-09 16:13:18,427 | sd | DEBUG | installer | Running pip: install --upgrade torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/rocm5.4.2
2023-08-09 16:14:26,590 | sd | WARNING | installer | Modified files: ['webui.sh']
2023-08-09 16:14:26,596 | sd | DEBUG | installer | Repository update time: Tue Aug  8 18:05:30 2023
2023-08-09 16:14:26,596 | sd | INFO | installer | Verifying requirements
2023-08-09 16:14:26,598 | sd | DEBUG | installer | Package version not found: addict
2023-08-09 16:14:26,598 | sd | INFO | installer | Installing package: addict
2023-08-09 16:14:26,600 | sd | DEBUG | installer | Running pip: install --upgrade addict
2023-08-09 16:14:27,292 | sd | DEBUG | installer | Package version not found: aenum
2023-08-09 16:14:27,292 | sd | INFO | installer | Installing package: aenum
2023-08-09 16:14:27,294 | sd | DEBUG | installer | Running pip: install --upgrade aenum
2023-08-09 16:14:28,045 | sd | DEBUG | installer | Package version not found: aiohttp
2023-08-09 16:14:28,045 | sd | INFO | installer | Installing package: aiohttp
2023-08-09 16:14:28,047 | sd | DEBUG | installer | Running pip: install --upgrade aiohttp
2023-08-09 16:14:29,939 | sd | DEBUG | installer | Package version not found: anyio
2023-08-09 16:14:29,940 | sd | INFO | installer | Installing package: anyio
2023-08-09 16:14:29,941 | sd | DEBUG | installer | Running pip: install --upgrade anyio
2023-08-09 16:14:30,755 | sd | DEBUG | installer | Package version not found: appdirs
2023-08-09 16:14:30,755 | sd | INFO | installer | Installing package: appdirs
2023-08-09 16:14:30,757 | sd | DEBUG | installer | Running pip: install --upgrade appdirs
2023-08-09 16:14:31,402 | sd | DEBUG | installer | Package version not found: astunparse
2023-08-09 16:14:31,403 | sd | INFO | installer | Installing package: astunparse
2023-08-09 16:14:31,404 | sd | DEBUG | installer | Running pip: install --upgrade astunparse
2023-08-09 16:14:32,236 | sd | DEBUG | installer | Package version not found: bitsandbytes
2023-08-09 16:14:32,236 | sd | INFO | installer | Installing package: bitsandbytes
2023-08-09 16:14:32,238 | sd | DEBUG | installer | Running pip: install --upgrade bitsandbytes
2023-08-09 16:14:34,554 | sd | DEBUG | installer | Package version not found: blendmodes
2023-08-09 16:14:34,554 | sd | INFO | installer | Installing package: blendmodes
2023-08-09 16:14:34,556 | sd | DEBUG | installer | Running pip: install --upgrade blendmodes
2023-08-09 16:14:35,334 | sd | DEBUG | installer | Package version not found: clean-fid
2023-08-09 16:14:35,334 | sd | INFO | installer | Installing package: clean-fid
2023-08-09 16:14:35,336 | sd | DEBUG | installer | Running pip: install --upgrade clean-fid
2023-08-09 16:14:39,099 | sd | DEBUG | installer | Package version not found: easydev
2023-08-09 16:14:39,099 | sd | INFO | installer | Installing package: easydev
2023-08-09 16:14:39,101 | sd | DEBUG | installer | Running pip: install --upgrade easydev
2023-08-09 16:14:40,158 | sd | DEBUG | installer | Package version not found: extcolors
2023-08-09 16:14:40,158 | sd | INFO | installer | Installing package: extcolors
2023-08-09 16:14:40,160 | sd | DEBUG | installer | Running pip: install --upgrade extcolors
2023-08-09 16:14:41,108 | sd | DEBUG | installer | Package version not found: facexlib
2023-08-09 16:14:41,108 | sd | INFO | installer | Installing package: facexlib
2023-08-09 16:14:41,110 | sd | DEBUG | installer | Running pip: install --upgrade facexlib
2023-08-09 16:14:48,914 | sd | DEBUG | installer | Package version not found: filetype
2023-08-09 16:14:48,915 | sd | INFO | installer | Installing package: filetype
2023-08-09 16:14:48,916 | sd | DEBUG | installer | Running pip: install --upgrade filetype
2023-08-09 16:14:49,700 | sd | DEBUG | installer | Package version not found: future
2023-08-09 16:14:49,700 | sd | INFO | installer | Installing package: future
2023-08-09 16:14:49,702 | sd | DEBUG | installer | Running pip: install --upgrade future
2023-08-09 16:14:50,668 | sd | DEBUG | installer | Package version not found: gdown
2023-08-09 16:14:50,668 | sd | INFO | installer | Installing package: gdown
2023-08-09 16:14:50,670 | sd | DEBUG | installer | Running pip: install --upgrade gdown
2023-08-09 16:14:51,689 | sd | DEBUG | installer | Package version not found: gfpgan
2023-08-09 16:14:51,689 | sd | INFO | installer | Installing package: gfpgan
2023-08-09 16:14:51,690 | sd | DEBUG | installer | Running pip: install --upgrade gfpgan
2023-08-09 16:14:58,705 | sd | DEBUG | installer | Package version not found: GitPython
2023-08-09 16:14:58,705 | sd | INFO | installer | Installing package: GitPython
2023-08-09 16:14:58,707 | sd | DEBUG | installer | Running pip: install --upgrade GitPython
2023-08-09 16:14:59,906 | sd | DEBUG | installer | Package version not found: httpcore
2023-08-09 16:14:59,907 | sd | INFO | installer | Installing package: httpcore
2023-08-09 16:14:59,908 | sd | DEBUG | installer | Running pip: install --upgrade httpcore
2023-08-09 16:15:00,975 | sd | DEBUG | installer | Package version not found: inflection
2023-08-09 16:15:00,975 | sd | INFO | installer | Installing package: inflection
2023-08-09 16:15:00,977 | sd | DEBUG | installer | Running pip: install --upgrade inflection
2023-08-09 16:15:01,898 | sd | DEBUG | installer | Package version not found: jsonmerge
2023-08-09 16:15:01,899 | sd | INFO | installer | Installing package: jsonmerge
2023-08-09 16:15:01,900 | sd | DEBUG | installer | Running pip: install --upgrade jsonmerge
2023-08-09 16:15:03,330 | sd | DEBUG | installer | Package version not found: kornia
2023-08-09 16:15:03,331 | sd | INFO | installer | Installing package: kornia
2023-08-09 16:15:03,332 | sd | DEBUG | installer | Running pip: install --upgrade kornia
2023-08-09 16:15:04,671 | sd | DEBUG | installer | Package version not found: lark
2023-08-09 16:15:04,671 | sd | INFO | installer | Installing package: lark
2023-08-09 16:15:04,673 | sd | DEBUG | installer | Running pip: install --upgrade lark
2023-08-09 16:15:05,711 | sd | DEBUG | installer | Package version not found: lmdb
2023-08-09 16:15:05,712 | sd | INFO | installer | Installing package: lmdb
2023-08-09 16:15:05,713 | sd | DEBUG | installer | Running pip: install --upgrade lmdb
2023-08-09 16:15:06,702 | sd | DEBUG | installer | Package version not found: lpips
2023-08-09 16:15:06,702 | sd | INFO | installer | Installing package: lpips
2023-08-09 16:15:06,703 | sd | DEBUG | installer | Running pip: install --upgrade lpips
2023-08-09 16:15:07,729 | sd | DEBUG | installer | Package version not found: omegaconf
2023-08-09 16:15:07,729 | sd | INFO | installer | Installing package: omegaconf
2023-08-09 16:15:07,731 | sd | DEBUG | installer | Running pip: install --upgrade omegaconf
2023-08-09 16:15:08,959 | sd | DEBUG | installer | Package version not found: open-clip-torch
2023-08-09 16:15:08,959 | sd | INFO | installer | Installing package: open-clip-torch
2023-08-09 16:15:08,961 | sd | DEBUG | installer | Running pip: install --upgrade open-clip-torch
2023-08-09 16:15:12,998 | sd | DEBUG | installer | Package version not found: opencv-contrib-python-headless
2023-08-09 16:15:12,999 | sd | INFO | installer | Installing package: opencv-contrib-python-headless
2023-08-09 16:15:13,000 | sd | DEBUG | installer | Running pip: install --upgrade opencv-contrib-python-headless
2023-08-09 16:15:24,135 | sd | DEBUG | installer | Package version not found: piexif
2023-08-09 16:15:24,136 | sd | INFO | installer | Installing package: piexif
2023-08-09 16:15:24,137 | sd | DEBUG | installer | Running pip: install --upgrade piexif
2023-08-09 16:15:25,296 | sd | DEBUG | installer | Package version not found: psutil
2023-08-09 16:15:25,296 | sd | INFO | installer | Installing package: psutil
2023-08-09 16:15:25,298 | sd | DEBUG | installer | Running pip: install --upgrade psutil
2023-08-09 16:15:26,620 | sd | DEBUG | installer | Package version not found: pyyaml
2023-08-09 16:15:26,620 | sd | INFO | installer | Installing package: pyyaml
2023-08-09 16:15:26,622 | sd | DEBUG | installer | Running pip: install --upgrade pyyaml
2023-08-09 16:15:27,757 | sd | DEBUG | installer | Package version not found: realesrgan
2023-08-09 16:15:27,757 | sd | INFO | installer | Installing package: realesrgan
2023-08-09 16:15:27,759 | sd | DEBUG | installer | Running pip: install --upgrade realesrgan
2023-08-09 16:15:29,131 | sd | DEBUG | installer | Package version not found: resize-right
2023-08-09 16:15:29,131 | sd | INFO | installer | Installing package: resize-right
2023-08-09 16:15:29,132 | sd | DEBUG | installer | Running pip: install --upgrade resize-right
2023-08-09 16:15:30,244 | sd | DEBUG | installer | Package version not found: rich
2023-08-09 16:15:30,244 | sd | INFO | installer | Installing package: rich
2023-08-09 16:15:30,246 | sd | DEBUG | installer | Running pip: install --upgrade rich
2023-08-09 16:15:31,413 | sd | DEBUG | installer | Package version not found: safetensors
2023-08-09 16:15:31,414 | sd | INFO | installer | Installing package: safetensors
2023-08-09 16:15:31,415 | sd | DEBUG | installer | Running pip: install --upgrade safetensors
2023-08-09 16:15:32,573 | sd | DEBUG | installer | Package version not found: scipy
2023-08-09 16:15:32,574 | sd | INFO | installer | Installing package: scipy
2023-08-09 16:15:32,575 | sd | DEBUG | installer | Running pip: install --upgrade scipy
2023-08-09 16:15:33,819 | sd | DEBUG | installer | Package version not found: tb_nightly
2023-08-09 16:15:33,819 | sd | INFO | installer | Installing package: tb_nightly
2023-08-09 16:15:33,821 | sd | DEBUG | installer | Running pip: install --upgrade tb_nightly
2023-08-09 16:15:35,448 | sd | DEBUG | installer | Package version not found: toml
2023-08-09 16:15:35,448 | sd | INFO | installer | Installing package: toml
2023-08-09 16:15:35,450 | sd | DEBUG | installer | Running pip: install --upgrade toml
2023-08-09 16:15:36,608 | sd | DEBUG | installer | Package version not found: torchdiffeq
2023-08-09 16:15:36,608 | sd | INFO | installer | Installing package: torchdiffeq
2023-08-09 16:15:36,609 | sd | DEBUG | installer | Running pip: install --upgrade torchdiffeq
2023-08-09 16:15:37,776 | sd | DEBUG | installer | Package version not found: torchsde
2023-08-09 16:15:37,776 | sd | INFO | installer | Installing package: torchsde
2023-08-09 16:15:37,777 | sd | DEBUG | installer | Running pip: install --upgrade torchsde
2023-08-09 16:15:39,103 | sd | DEBUG | installer | Package version not found: voluptuous
2023-08-09 16:15:39,103 | sd | INFO | installer | Installing package: voluptuous
2023-08-09 16:15:39,104 | sd | DEBUG | installer | Running pip: install --upgrade voluptuous
2023-08-09 16:15:40,261 | sd | DEBUG | installer | Package version not found: yapf
2023-08-09 16:15:40,262 | sd | INFO | installer | Installing package: yapf
2023-08-09 16:15:40,263 | sd | DEBUG | installer | Running pip: install --upgrade yapf
2023-08-09 16:15:41,396 | sd | DEBUG | installer | Package version not found: scikit-image
2023-08-09 16:15:41,396 | sd | INFO | installer | Installing package: scikit-image
2023-08-09 16:15:41,398 | sd | DEBUG | installer | Running pip: install --upgrade scikit-image
2023-08-09 16:15:42,603 | sd | DEBUG | installer | Package version not found: basicsr
2023-08-09 16:15:42,603 | sd | INFO | installer | Installing package: basicsr
2023-08-09 16:15:42,605 | sd | DEBUG | installer | Running pip: install --upgrade basicsr
2023-08-09 16:15:43,844 | sd | DEBUG | installer | Package version not found: compel
2023-08-09 16:15:43,844 | sd | INFO | installer | Installing package: compel
2023-08-09 16:15:43,846 | sd | DEBUG | installer | Running pip: install --upgrade compel
2023-08-09 16:15:49,765 | sd | DEBUG | installer | Package version not found: fasteners
2023-08-09 16:15:49,766 | sd | INFO | installer | Installing package: fasteners
2023-08-09 16:15:49,767 | sd | DEBUG | installer | Running pip: install --upgrade fasteners
2023-08-09 16:15:51,318 | sd | DEBUG | installer | Package version not found: typing-extensions
2023-08-09 16:15:51,318 | sd | INFO | installer | Installing package: typing-extensions==4.7.1
2023-08-09 16:15:51,320 | sd | DEBUG | installer | Running pip: install --upgrade typing-extensions==4.7.1
2023-08-09 16:15:53,027 | sd | DEBUG | installer | Package version not found: antlr4-python3-runtime
2023-08-09 16:15:53,027 | sd | INFO | installer | Installing package: antlr4-python3-runtime==4.9.3
2023-08-09 16:15:53,029 | sd | DEBUG | installer | Running pip: install --upgrade antlr4-python3-runtime==4.9.3
2023-08-09 16:15:54,537 | sd | DEBUG | installer | Package version not found: requests
2023-08-09 16:15:54,537 | sd | INFO | installer | Installing package: requests==2.31.0
2023-08-09 16:15:54,539 | sd | DEBUG | installer | Running pip: install --upgrade requests==2.31.0
2023-08-09 16:15:56,177 | sd | DEBUG | installer | Package version not found: tqdm
2023-08-09 16:15:56,177 | sd | INFO | installer | Installing package: tqdm==4.65.0
2023-08-09 16:15:56,179 | sd | DEBUG | installer | Running pip: install --upgrade tqdm==4.65.0
2023-08-09 16:15:57,817 | sd | DEBUG | installer | Package version not found: accelerate
2023-08-09 16:15:57,818 | sd | INFO | installer | Installing package: accelerate==0.20.3
2023-08-09 16:15:57,819 | sd | DEBUG | installer | Running pip: install --upgrade accelerate==0.20.3
2023-08-09 16:15:59,542 | sd | DEBUG | installer | Package version not found: opencv-python-headless
2023-08-09 16:15:59,542 | sd | INFO | installer | Installing package: opencv-python-headless==4.7.0.72
2023-08-09 16:15:59,544 | sd | DEBUG | installer | Running pip: install --upgrade opencv-python-headless==4.7.0.72
2023-08-09 16:16:02,098 | sd | DEBUG | installer | Package version not found: diffusers
2023-08-09 16:16:02,098 | sd | INFO | installer | Installing package: diffusers==0.19.3
2023-08-09 16:16:02,101 | sd | DEBUG | installer | Running pip: install --upgrade diffusers==0.19.3
2023-08-09 16:16:03,675 | sd | DEBUG | installer | Package version not found: einops
2023-08-09 16:16:03,675 | sd | INFO | installer | Installing package: einops==0.4.1
2023-08-09 16:16:03,677 | sd | DEBUG | installer | Running pip: install --upgrade einops==0.4.1
2023-08-09 16:16:05,258 | sd | DEBUG | installer | Package version not found: gradio
2023-08-09 16:16:05,258 | sd | INFO | installer | Installing package: gradio==3.32.0
2023-08-09 16:16:05,260 | sd | DEBUG | installer | Running pip: install --upgrade gradio==3.32.0
2023-08-09 16:16:14,776 | sd | DEBUG | installer | Package version not found: huggingface_hub
2023-08-09 16:16:14,776 | sd | INFO | installer | Installing package: huggingface_hub==0.16.4
2023-08-09 16:16:14,778 | sd | DEBUG | installer | Running pip: install --upgrade huggingface_hub==0.16.4
2023-08-09 16:16:16,532 | sd | DEBUG | installer | Package version not found: numexpr
2023-08-09 16:16:16,533 | sd | INFO | installer | Installing package: numexpr==2.8.4
2023-08-09 16:16:16,534 | sd | DEBUG | installer | Running pip: install --upgrade numexpr==2.8.4
2023-08-09 16:16:18,375 | sd | DEBUG | installer | Package version not found: numpy
2023-08-09 16:16:18,375 | sd | INFO | installer | Installing package: numpy==1.23.5
2023-08-09 16:16:18,376 | sd | DEBUG | installer | Running pip: install --upgrade numpy==1.23.5
2023-08-09 16:16:21,993 | sd | DEBUG | installer | Package version not found: numba
2023-08-09 16:16:21,994 | sd | INFO | installer | Installing package: numba==0.57.0
2023-08-09 16:16:21,995 | sd | DEBUG | installer | Running pip: install --upgrade numba==0.57.0
2023-08-09 16:16:25,601 | sd | DEBUG | installer | Package version not found: pandas
2023-08-09 16:16:25,601 | sd | INFO | installer | Installing package: pandas==1.5.3
2023-08-09 16:16:25,603 | sd | DEBUG | installer | Running pip: install --upgrade pandas==1.5.3
2023-08-09 16:16:30,781 | sd | DEBUG | installer | Package version not found: protobuf
2023-08-09 16:16:30,781 | sd | INFO | installer | Installing package: protobuf==3.20.3
2023-08-09 16:16:30,783 | sd | DEBUG | installer | Running pip: install --upgrade protobuf==3.20.3
2023-08-09 16:16:32,689 | sd | DEBUG | installer | Package version not found: pytorch_lightning
2023-08-09 16:16:32,690 | sd | INFO | installer | Installing package: pytorch_lightning==1.9.4
2023-08-09 16:16:32,692 | sd | DEBUG | installer | Running pip: install --upgrade pytorch_lightning==1.9.4
2023-08-09 16:16:35,452 | sd | DEBUG | installer | Package version not found: transformers
2023-08-09 16:16:35,452 | sd | INFO | installer | Installing package: transformers==4.31.0
2023-08-09 16:16:35,454 | sd | DEBUG | installer | Running pip: install --upgrade transformers==4.31.0
2023-08-09 16:16:37,357 | sd | DEBUG | installer | Package version not found: tomesd
2023-08-09 16:16:37,357 | sd | INFO | installer | Installing package: tomesd==0.1.3
2023-08-09 16:16:37,359 | sd | DEBUG | installer | Running pip: install --upgrade tomesd==0.1.3
2023-08-09 16:16:39,344 | sd | DEBUG | installer | Package version not found: urllib3
2023-08-09 16:16:39,344 | sd | INFO | installer | Installing package: urllib3==1.26.15
2023-08-09 16:16:39,345 | sd | DEBUG | installer | Running pip: install --upgrade urllib3==1.26.15
2023-08-09 16:16:41,289 | sd | DEBUG | installer | Package version not found: Pillow
2023-08-09 16:16:41,289 | sd | INFO | installer | Installing package: Pillow==9.5.0
2023-08-09 16:16:41,291 | sd | DEBUG | installer | Running pip: install --upgrade Pillow==9.5.0
2023-08-09 16:16:43,689 | sd | DEBUG | installer | Package version not found: timm
2023-08-09 16:16:43,689 | sd | INFO | installer | Installing package: timm==0.6.13
2023-08-09 16:16:43,690 | sd | DEBUG | installer | Running pip: install --upgrade timm==0.6.13
2023-08-09 16:16:45,947 | sd | DEBUG | installer | Package version not found: pydantic
2023-08-09 16:16:45,948 | sd | INFO | installer | Installing package: pydantic==1.10.11
2023-08-09 16:16:45,949 | sd | DEBUG | installer | Running pip: install --upgrade pydantic==1.10.11
2023-08-09 16:16:48,125 | sd | INFO | installer | Verifying packages
2023-08-09 16:16:48,127 | sd | DEBUG | installer | Package version not found: clip
2023-08-09 16:16:48,127 | sd | INFO | installer | Installing package: git+https://github.com/openai/CLIP.git
2023-08-09 16:16:48,127 | sd | DEBUG | installer | Running pip: install --upgrade git+https://github.com/openai/CLIP.git
2023-08-09 16:16:52,070 | sd | DEBUG | installer | Package version not found: invisible-watermark
2023-08-09 16:16:52,070 | sd | INFO | installer | Installing package: git+https://github.com/patrickvonplaten/invisible-watermark.git@remove_onnxruntime_depedency
2023-08-09 16:16:52,072 | sd | DEBUG | installer | Running pip: install --upgrade git+https://github.com/patrickvonplaten/invisible-watermark.git@remove_onnxruntime_depedency
2023-08-09 16:16:58,555 | sd | DEBUG | installer | Package version not found: onnxruntime
2023-08-09 16:16:58,555 | sd | INFO | installer | Installing package: onnxruntime==1.15.1
2023-08-09 16:16:58,557 | sd | DEBUG | installer | Running pip: install --upgrade onnxruntime==1.15.1
2023-08-09 16:17:01,026 | sd | DEBUG | installer | Package version not found: pi_heif
2023-08-09 16:17:01,026 | sd | INFO | installer | Installing package: pi-heif
2023-08-09 16:17:01,028 | sd | DEBUG | installer | Running pip: install --upgrade pi-heif
2023-08-09 16:17:03,406 | sd | DEBUG | installer | Package version not found: tensorflow
2023-08-09 16:17:03,407 | sd | INFO | installer | Installing package: tensorflow-rocm
2023-08-09 16:17:03,408 | sd | DEBUG | installer | Running pip: install --upgrade tensorflow-rocm
2023-08-09 16:17:22,051 | sd | INFO | installer | Verifying repositories
2023-08-09 16:17:22,061 | sd | DEBUG | installer | Submodule: /opt/ai/automatic/repositories/stable-diffusion-stability-ai / main
2023-08-09 16:17:22,340 | sd | DEBUG | installer | Submodule: /opt/ai/automatic/repositories/taming-transformers / master
2023-08-09 16:17:22,659 | sd | DEBUG | installer | Submodule: /opt/ai/automatic/repositories/k-diffusion / master
2023-08-09 16:17:23,201 | sd | DEBUG | installer | Submodule: /opt/ai/automatic/repositories/BLIP / main
2023-08-09 16:17:23,533 | sd | INFO | installer | Verifying submodules
2023-08-09 16:17:24,063 | sd | DEBUG | installer | Submodule: extensions-builtin/a1111-sd-webui-lycoris / main
2023-08-09 16:17:24,071 | sd | DEBUG | installer | Submodule: extensions-builtin/clip-interrogator-ext / main
2023-08-09 16:17:24,078 | sd | DEBUG | installer | Submodule: extensions-builtin/multidiffusion-upscaler-for-automatic1111 / main
2023-08-09 16:17:24,085 | sd | DEBUG | installer | Submodule: extensions-builtin/sd-dynamic-thresholding / master
2023-08-09 16:17:24,092 | sd | DEBUG | installer | Submodule: extensions-builtin/sd-extension-system-info / main
2023-08-09 16:17:24,099 | sd | DEBUG | installer | Submodule: extensions-builtin/sd-webui-agent-scheduler / main
2023-08-09 16:17:24,117 | sd | DEBUG | installer | Submodule: extensions-builtin/sd-webui-controlnet / main
2023-08-09 16:17:24,129 | sd | DEBUG | installer | Submodule: extensions-builtin/stable-diffusion-webui-images-browser / main
2023-08-09 16:17:24,135 | sd | DEBUG | installer | Submodule: extensions-builtin/stable-diffusion-webui-rembg / master
2023-08-09 16:17:24,143 | sd | DEBUG | installer | Submodule: modules/lora / main
2023-08-09 16:17:24,151 | sd | DEBUG | installer | Submodule: modules/lycoris / main
2023-08-09 16:17:24,160 | sd | DEBUG | installer | Submodule: wiki / master
2023-08-09 16:17:24,209 | sd | DEBUG | installer | Installed packages: 197
2023-08-09 16:17:24,209 | sd | DEBUG | installer | Extensions all: ['multidiffusion-upscaler-for-automatic1111', 'stable-diffusion-webui-rembg', 'LDSR', 'Lora', 'stable-diffusion-webui-images-browser', 'sd-webui-controlnet', 'ScuNET', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'sd-dynamic-thresholding', 'clip-interrogator-ext', 'SwinIR', 'a1111-sd-webui-lycoris']
2023-08-09 16:17:24,247 | sd | DEBUG | installer | Running extension installer: /opt/ai/automatic/extensions-builtin/stable-diffusion-webui-rembg/install.py
2023-08-09 16:17:29,217 | sd | INFO | installer | Extension installed packages: stable-diffusion-webui-rembg ['rembg==2.0.38', 'pooch==1.7.0', 'PyMatting==1.1.8']
2023-08-09 16:17:29,297 | sd | DEBUG | installer | Running extension installer: /opt/ai/automatic/extensions-builtin/stable-diffusion-webui-images-browser/install.py
2023-08-09 16:17:31,405 | sd | INFO | installer | Extension installed packages: stable-diffusion-webui-images-browser ['Send2Trash==1.8.2']
2023-08-09 16:17:31,406 | sd | DEBUG | installer | Running extension installer: /opt/ai/automatic/extensions-builtin/sd-webui-controlnet/install.py
2023-08-09 16:17:42,802 | sd | INFO | installer | Extension installed packages: sd-webui-controlnet ['lxml==4.9.3', 'opencv-contrib-python==4.8.0.76', 'reportlab==4.0.4', 'pycparser==2.21', 'portalocker==2.7.0', 'cffi==1.15.1', 'svglib==1.5.1', 'tinycss2==1.2.1', 'mediapipe==0.10.3', 'tabulate==0.9.0', 'cssselect2==0.7.0', 'webencodings==0.5.1', 'sounddevice==0.4.6', 'iopath==0.1.9', 'yacs==0.1.8', 'fvcore==0.1.5.post20221221']
2023-08-09 16:17:42,842 | sd | DEBUG | installer | Running extension installer: /opt/ai/automatic/extensions-builtin/sd-webui-agent-scheduler/install.py
2023-08-09 16:17:46,451 | sd | INFO | installer | Extension installed packages: sd-webui-agent-scheduler ['SQLAlchemy==2.0.19', 'greenlet==2.0.2']
2023-08-09 16:17:46,452 | sd | DEBUG | installer | Running extension installer: /opt/ai/automatic/extensions-builtin/sd-extension-system-info/install.py
2023-08-09 16:17:46,678 | sd | DEBUG | installer | Running extension installer: /opt/ai/automatic/extensions-builtin/clip-interrogator-ext/install.py
2023-08-09 16:17:49,072 | sd | INFO | installer | Extension installed packages: clip-interrogator-ext ['clip-interrogator==0.6.0']
2023-08-09 16:17:49,149 | sd | DEBUG | installer | Extensions all: []
2023-08-09 16:17:49,149 | sd | INFO | installer | Extensions enabled: ['multidiffusion-upscaler-for-automatic1111', 'stable-diffusion-webui-rembg', 'LDSR', 'Lora', 'stable-diffusion-webui-images-browser', 'sd-webui-controlnet', 'ScuNET', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'sd-dynamic-thresholding', 'clip-interrogator-ext', 'SwinIR', 'a1111-sd-webui-lycoris']
2023-08-09 16:17:49,151 | sd | INFO | installer | Verifying packages
2023-08-09 16:17:49,152 | sd | DEBUG | installer | Package version not found: tensorflow
2023-08-09 16:17:49,152 | sd | INFO | installer | Installing package: tensorflow-rocm
2023-08-09 16:17:49,153 | sd | DEBUG | installer | Running pip: install --upgrade tensorflow-rocm
2023-08-09 16:17:51,256 | sd | DEBUG | launch | Setup complete without errors: 1691590671
2023-08-09 16:17:51,272 | sd | INFO | installer | Extension preload: 0.0s /opt/ai/automatic/extensions-builtin
2023-08-09 16:17:51,273 | sd | INFO | installer | Extension preload: 0.0s /opt/ai/automatic/extensions
2023-08-09 16:17:51,290 | sd | INFO | launch | Server arguments: []
2023-08-09 16:17:56,493 | sd | INFO | shared | Pipeline: Backend.ORIGINAL
2023-08-09 16:17:56,806 | sd | INFO | webui | Libraries loaded
2023-08-09 16:17:56,808 | sd | INFO | webui | Using data path: /opt/ai/automatic
2023-08-09 16:17:56,809 | sd | INFO | sd_vae | Available VAEs: /opt/ai/automatic/models/VAE 0
2023-08-09 16:17:56,812 | sd | INFO | sd_models | Available models: /opt/ai/automatic/models/Stable-diffusion 1
2023-08-09 16:17:58,541 | ControlNet | INFO | controlnet_version | ControlNet v1.1.234
2023-08-09 16:17:58,736 | ControlNet | INFO | controlnet_version | ControlNet v1.1.234
2023-08-09 16:18:15,376 | sd | INFO | shared | Loading UI theme: name=black-orange style=Auto
2023-08-09 16:18:16,446 | sd | INFO | webui | Local URL: http://127.0.0.1:7860/
2023-08-09 16:18:16,448 | sd | INFO | middleware | Initializing middleware
2023-08-09 16:18:17,020 | sd | INFO | task_runner | [AgentScheduler] Task queue is empty
2023-08-09 16:18:17,021 | sd | INFO | api | [AgentScheduler] Registering APIs
2023-08-09 16:18:18,814 | sd | INFO | devices | Torch override dtype: no-half set
2023-08-09 16:18:18,815 | sd | INFO | devices | Torch override VAE dtype: no-half set
2023-08-09 16:18:18,816 | sd | INFO | devices | Setting Torch parameters: dtype=torch.float32 vae=torch.float32 unet=torch.float32
2023-08-09 16:18:23,435 | sd | INFO | sd_hijack | Applying Doggettx cross attention optimization
2023-08-09 16:18:23,441 | sd | INFO | textual_inversion | Embeddings: loaded=0 skipped=0
2023-08-09 16:18:23,447 | sd | INFO | sd_models | Model loaded in 5.9s (load=1.3s create=0.5s apply=4.1s)
2023-08-09 16:18:23,846 | sd | INFO | sd_models | Model load finished: {'ram': {'used': 9.36, 'total': 31.24}} cached=0
2023-08-09 16:18:24,294 | sd | INFO | webui | Startup time: 33.0s (torch=3.9s gradio=0.5s libraries=1.1s scripts=18.4s onchange=0.1s ui-txt2img=0.1s ui-img2img=0.1s ui-settings=0.1s ui-extensions=0.4s ui-defaults=0.1s launch=0.2s app-started=0.7s checkpoint=7.1s)

EDIT: Added log and console outpu

Acknowledgements

vladmandic commented 1 year ago

@ConfusedMerlin

they are on an ext-4 fs, that also hosts about... everything else. I know, not the best setup, But after a dozen or so not quite working manuel partition setups I just stuffed everything on one big fs. EDiT: would that memmap/stream not be kind of reproducable with ease?

yeah, mmap is not an issue here. regardless of your partititioning, ext4 is definitely good enough - and yes, it would be reproducible.