search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.56k stars 409 forks source link

[Issue]: Accelerate version 0.22.0 and above causes a bug in inpaint mode. #2559

Closed mozlima closed 10 months ago

mozlima commented 10 months ago

Issue Description

Inpaint mode fail when using Accelerate version 0.22.0 and above

21:45:15-408496 ERROR    Diffusers failed loading:                                                  
                         model=/stable-diffusion-data/models/stable-diffusion/i
                         npaintClothing_uberrealisticinpaint-inpainting.safetensors                 
                         pipeline=Autodetect/NoneType Trying to set a tensor of shape               
                         torch.Size([320, 9, 3, 3]) in "weight" (which has shape torch.Size([320, 4,
                         3, 3])), this look incorrect. 

21:55:10-778978 ERROR    Diffusers failed loading:                                                  
                         model=/stable-diffusion-data/models/stable-diffusion/a
                         ZovyaPhotoreal_v2InpaintVAE.safetensors pipeline=Autodetect/NoneType Trying
                         to set a tensor of shape torch.Size([320, 9, 3, 3]) in "weight" (which has 
                         shape torch.Size([320, 4, 3, 3])), this look incorrect.

Works good if rollback to: accelerate==0.21.0 Fail with: 0.22.0, 0.23.0, 0.24.0, 0.24.1

Version Platform Description

Linux 6.6.2-arch1-1 Intel Arc A380 AMD Ryzen 7 2700

Relevant log output

Setting OneAPI environment

:: initializing oneAPI environment ...
   webui.sh: BASH_VERSION = 5.2.21(1)-release
   args: Using "$@" for setvars.sh arguments: --data-dir /apps/sd.next --debug --use-ipex 
:: advisor -- latest
:: ccl -- latest
:: compiler -- latest
:: dal -- latest
:: debugger -- latest
:: dev-utilities -- latest
:: dnnl -- latest
:: dpcpp-ct -- latest
:: dpl -- latest
:: ipp -- latest
:: ippcp -- latest
:: ipp -- latest
:: mkl -- latest
:: mpi -- latest
:: tbb -- latest
:: vtune -- latest
:: oneAPI environment initialized ::

Launching ipexrun launch.py...
/apps/sd.next/venv/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: ''If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
2023-11-25 09:20:38.420725: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-11-25 09:20:38.590728: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-11-25 09:20:38.591196: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-25 09:20:39.829275: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-11-25 09:20:42.824176: I itex/core/wrapper/itex_gpu_wrapper.cc:35] Intel Extension for Tensorflow* GPU backend is loaded.
2023-11-25 09:20:42.903986: W itex/core/ops/op_init.cc:58] Op: _QuantizedMaxPool3D is already registered in Tensorflow
2023-11-25 09:20:42.923395: I itex/core/devices/gpu/itex_gpu_runtime.cc:129] Selected platform: Intel(R) Level-Zero
2023-11-25 09:20:42.923740: I itex/core/devices/gpu/itex_gpu_runtime.cc:154] number of sub-devices is zero, expose root device.
2023-11-25 09:20:43,326 - numexpr.utils - INFO - Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-11-25 09:20:43,326 - numexpr.utils - INFO - NumExpr defaulting to 8 threads.
/apps/sd.next/venv/lib/python3.11/site-packages/intel_extension_for_pytorch/launcher.py:102: UserWarning: Backend is not specified, it will automatically default to cpu.
  warnings.warn(
2023-11-25 09:20:43,816 - intel_extension_for_pytorch.cpu.launch.launch - INFO - Use 'jemalloc' memory allocator.
2023-11-25 09:20:43,817 - intel_extension_for_pytorch.cpu.launch.launch - WARNING - 'intel' OpenMP runtime is not found in ['/apps/sd.next/venv/lib/', '/apps/.local/lib/', '/usr/local/lib/', '/usr/local/lib64/', '/usr/lib/', '/usr/lib64/', '/usr/lib/x86_64-linux-gnu/'].
2023-11-25 09:20:43,818 - intel_extension_for_pytorch.cpu.launch.launch - INFO - Use 'default' OpenMP runtime.
2023-11-25 09:20:43,823 - intel_extension_for_pytorch.cpu.launch.launch - INFO - Use 'taskset' multi-task manager.
2023-11-25 09:20:43,823 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: Untouched preset environment variables are not displayed.
2023-11-25 09:20:43,823 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: LD_PRELOAD=/usr/lib/libjemalloc.so
2023-11-25 09:20:43,823 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: MALLOC_CONF=oversize_threshold:1,background_thread:true,metadata_thp:auto
2023-11-25 09:20:43,823 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: OMP_SCHEDULE=STATIC
2023-11-25 09:20:43,823 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: OMP_PROC_BIND=CLOSE
2023-11-25 09:20:43,823 - intel_extension_for_pytorch.cpu.launch.launch - INFO - env: OMP_NUM_THREADS=8
2023-11-25 09:20:43,823 - intel_extension_for_pytorch.cpu.launch.launch - INFO - cmd: taskset -c 0-7 /apps/sd.next/venv/bin/python -u launch.py --data-dir /apps/sd.next --debug --use-ipex 
09:20:44-320907 DEBUG    Logger: file=/apps/sd.next/sdnext.log       
                         level=10 size=0 mode=create                                                
09:20:44-324349 INFO     Starting SD.Next                                                           
09:20:44-325393 INFO     Python 3.11.6 on Linux                                                     
09:20:44-378649 INFO     Version: app=sd.next updated=2023-11-24 hash=a4a7a937                      
                         url=https://github.com/vladmandic/automatic/tree/master                    
09:20:45-268635 INFO     Platform: arch=x86_64 cpu= system=Linux release=6.6.2-arch1-1 python=3.11.6
09:20:45-271393 DEBUG    Setting environment tuning                                                 
09:20:45-273421 DEBUG    Cache folder: /apps/.cache/huggingface/hub  
09:20:45-275289 DEBUG    Torch overrides: cuda=False rocm=False ipex=True diml=False openvino=False 
09:20:45-277496 DEBUG    Torch allowed: cuda=False rocm=False ipex=True diml=False openvino=False   
09:20:45-280565 INFO     Intel OneAPI Toolkit detected                                              
09:20:45-283398 DEBUG    Package not found: onnxruntime-openvino                                    
09:20:45-285539 INFO     Installing package: onnxruntime-openvino                                   
09:20:45-287342 DEBUG    Running pip: install --upgrade onnxruntime-openvino                        
09:20:47-161053 DEBUG    Repository update time: Fri Nov 24 16:44:14 2023                           
09:20:47-163422 INFO     Startup: standard                                                          
09:20:47-164919 INFO     Verifying requirements                                                     
09:20:47-194984 WARNING  Package version mismatch: accelerate 0.21.0 required 0.24.1                
09:20:47-197707 INFO     Installing package: accelerate==0.24.1                                     
09:20:47-199518 DEBUG    Running pip: install --upgrade accelerate==0.24.1                          
09:20:50-683744 INFO     Verifying packages                                                         
09:20:50-686383 INFO     Verifying submodules                                                       
09:20:51-584283 DEBUG    Submodule: extensions-builtin/sd-extension-chainner / main                 
09:20:51-620198 DEBUG    Submodule: extensions-builtin/sd-extension-system-info / main              
09:20:51-659207 DEBUG    Submodule: extensions-builtin/sd-webui-agent-scheduler / main              
09:20:51-686062 DEBUG    Submodule: extensions-builtin/sd-webui-controlnet / main                   
09:20:51-738834 DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-images-browser / main 
09:20:51-777749 DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-rembg / master        
09:20:51-803998 DEBUG    Submodule: modules/k-diffusion / master                                    
09:20:51-833441 DEBUG    Submodule: modules/lora / main                                             
09:20:51-874046 DEBUG    Submodule: wiki / master                                                   
09:20:51-897701 DEBUG    Register paths                                                             
09:20:52-004481 DEBUG    Installed packages: 239                                                    
09:20:52-005438 DEBUG    Extensions all: ['Lora', 'sd-extension-chainner',                          
                         'sd-extension-system-info', 'sd-webui-agent-scheduler',                    
                         'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg']   
09:20:52-111643 DEBUG    Running extension installer:                                               
                         /apps/sd.next/extensions-builtin/sd-extensio
                         n-system-info/install.py                                                   
09:20:52-382004 DEBUG    Running extension installer:                                               
                         /apps/sd.next/extensions-builtin/sd-webui-ag
                         ent-scheduler/install.py                                                   
09:20:52-606912 DEBUG    Running extension installer:                                               
                         /apps/sd.next/extensions-builtin/stable-diff
                         usion-webui-images-browser/install.py                                      
09:20:52-853594 DEBUG    Running extension installer:                                               
                         /apps/sd.next/extensions-builtin/stable-diff
                         usion-webui-rembg/install.py                                               
09:20:53-151891 DEBUG    Extensions all: ['sd-webui-aspect-ratio-helper']                           
09:20:53-213062 INFO     Extensions enabled: ['Lora', 'sd-extension-chainner',                      
                         'sd-extension-system-info', 'sd-webui-agent-scheduler',                    
                         'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg',   
                         'sd-webui-aspect-ratio-helper']                                            
09:20:53-214346 INFO     Verifying requirements                                                     
09:20:53-247842 DEBUG    Setup complete without errors: 1700914853                                  
09:20:53-252480 INFO     Extension preload: {'extensions-builtin': 0.0,                             
                         '/apps/sd.next/extensions': 0.0}            
09:20:53-253639 DEBUG    Starting module: <module 'webui' from                                      
                         '/apps/sd.next/webui.py'>                   
09:20:53-254663 INFO     Command line args: ['--data-dir',                                          
                         '/apps/sd.next', '--debug', '--use-ipex']   
                         data_dir=/apps/sd.next debug=True           
                         use_ipex=True                                                              
/apps/sd.next/venv/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: ''If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
09:20:56-245312 DEBUG    Load IPEX==2.0.110+xpu                                                     
09:20:58-104814 INFO     Load packages: torch=2.0.1a0+cxx11.abi diffusers=0.23.1 gradio=3.43.2      
09:20:58-804397 DEBUG    Read: file="/apps/sd.next/config.json"      
                         json=38 bytes=2101                                                         
09:20:58-806602 INFO     Engine: backend=Backend.DIFFUSERS compute=ipex mode=no_grad device=xpu     
                         cross-optimization="Scaled-Dot-Product"                                    
09:20:58-808376 INFO     Device: device=Intel(R) Arc(TM) A380 Graphics n=1 ipex=2.0.110+xpu         
2023-11-25 09:21:01.712141: I itex/core/wrapper/itex_gpu_wrapper.cc:35] Intel Extension for Tensorflow* GPU backend is loaded.
2023-11-25 09:21:01.754231: W itex/core/ops/op_init.cc:58] Op: _QuantizedMaxPool3D is already registered in Tensorflow
2023-11-25 09:21:01.771236: I itex/core/devices/gpu/itex_gpu_runtime.cc:129] Selected platform: Intel(R) Level-Zero
2023-11-25 09:21:01.771487: I itex/core/devices/gpu/itex_gpu_runtime.cc:154] number of sub-devices is zero, expose root device.
09:21:02-475383 DEBUG    Entering start sequence                                                    
09:21:02-476439 INFO     Using data path: /apps/sd.next              
09:21:02-486159 DEBUG    Initializing                                                               
09:21:02-489285 INFO     Available VAEs:                                                            
                         path="/apps/.local/share/stable-diffusion-data/models/vae" items=0      
09:21:02-490385 INFO     Disabling uncompatible extensions: backend=Backend.DIFFUSERS               
                         ['multidiffusion-upscaler-for-automatic1111', 'a1111-sd-webui-lycoris']    
09:21:02-492451 DEBUG    Scanning diffusers cache:                                                  
                         /apps/.local/share/stable-diffusion-data/models/Diffusers               
                         /apps/.local/share/stable-diffusion-data/models/Diffusers items=1       
                         time=0.00                                                                  
09:21:02-494898 DEBUG    Read: file="/apps/sd.next/cache.json" json=2
                         bytes=49127                                                                
09:21:02-503657 DEBUG    Read: file="/apps/sd.next/metadata.json"    
                         json=252 bytes=870327                                                      
09:21:02-519797 INFO     Available models:                                                          
                         path="/apps/.local/share/stable-diffusion-data/models/stable-diffusion" 
                         items=44 time=0.03                                                         
09:21:02-785759 DEBUG    Load extensions                                                            
09:21:04-439179 INFO     Extension:                                                                 
                         script='extensions-builtin/sd-webui-agent-scheduler/scripts/task_scheduler.
                         py' Using sqlite file:                                                     
                         extensions-builtin/sd-webui-agent-scheduler/task_scheduler.sqlite3         
09:21:05-355978 INFO     Extensions time: 2.57 { sd-webui-aspect-ratio-helper=0.52 Lora=0.30        
                         sd-extension-chainner=0.08 sd-webui-agent-scheduler=0.70                   
                         stable-diffusion-webui-images-browser=0.12                                 
                         stable-diffusion-webui-rembg=0.80 }                                        
09:21:05-503967 DEBUG    Read: file="html/upscalers.json" json=4 bytes=2640                         
09:21:05-506209 DEBUG    Read: file="extensions-builtin/sd-extension-chainner/models.json" json=24  
                         bytes=2693                                                                 
09:21:05-507464 DEBUG    chaiNNer models:                                                           
                         path="/apps/.local/share/stable-diffusion-data/models/chaiNNer"         
                         defined=24 discovered=0 downloaded=0                                       
09:21:05-509850 DEBUG    Load upscalers: total=52 downloaded=1 user=0 ['None', 'Lanczos', 'Nearest',
                         'ChaiNNer', 'ESRGAN', 'LDSR', 'RealESRGAN', 'SCUNet', 'SD', 'SwinIR']      
09:21:05-518718 DEBUG    Load styles:                                                               
                         folder="/apps/sd.next/models/styles"        
                         items=288                                                                  
09:21:05-523022 DEBUG    Creating UI                                                                
09:21:06-129870 INFO     Load UI theme: name="gradio/base" style=Light base=base.css                
09:21:06-131268 WARNING  Using Gradio default theme which is not optimized for SD.Next              
09:21:06-196067 DEBUG    Read: file="html/reference.json" json=13 bytes=7896                        
09:21:06-399195 DEBUG    Extra networks: page='model' items=57 subfolders=3 tab=txt2img             
                         folders=['/apps/.local/share/stable-diffusion-data/models/stable-diffusi
                         on', '/apps/.local/share/stable-diffusion-data/models/Diffusers',       
                         'models/Reference',                                                        
                         '/apps/.local/share/stable-diffusion-data/models/Stable-diffusion']     
                         list=0.02 desc=0.01 info=0.03                                              
09:21:06-408660 DEBUG    Extra networks: page='style' items=288 subfolders=2 tab=txt2img            
                         folders=['/apps/sd.next/models/styles',     
                         'html'] list=0.01 desc=0.00 info=0.00                                      
09:21:06-412067 DEBUG    Extra networks: page='embedding' items=35 subfolders=1 tab=txt2img         
                         folders=['/apps/.local/share/stable-diffusion-data/models/embeddings']  
                         list=0.03 desc=0.01 info=0.13                                              
09:21:06-414648 DEBUG    Extra networks: page='hypernetwork' items=0 subfolders=1 tab=txt2img       
                         folders=['/apps/.local/share/stable-diffusion-data/models/hypernetworks'
                         ] list=0.00 desc=0.00 info=0.00                                            
09:21:06-417072 DEBUG    Extra networks: page='vae' items=0 subfolders=1 tab=txt2img                
                         folders=['/apps/.local/share/stable-diffusion-data/models/vae']         
                         list=0.00 desc=0.00 info=0.00                                              
09:21:06-425798 DEBUG    Extra networks: page='lora' items=209 subfolders=1 tab=txt2img             
                         folders=['/apps/.local/share/stable-diffusion-data/models/lora',        
                         '/apps/.local/share/stable-diffusion-data/models/LyCORIS'] list=0.16    
                         desc=0.06 info=0.83                                                        
09:21:06-625330 DEBUG    Read: file="/apps/sd.next/ui-config.json"   
                         json=0 bytes=2                                                             
09:21:06-809139 DEBUG    Themes: builtin=6 default=5 external=55                                    
09:21:07-400902 DEBUG    Script: 0.52 ui_tabs                                                       
                         /apps/sd.next/extensions-builtin/stable-diff
                         usion-webui-images-browser/scripts/image_browser.py                        
09:21:07-554275 DEBUG    Extension list: processed=10 installed=10 enabled=7 disabled=3 visible=10  
                         hidden=0                                                                   
09:21:07-931734 INFO     Local URL: http://127.0.0.1:7860/                                          
09:21:07-932688 DEBUG    Gradio functions: registered=1611                                          
09:21:07-933380 INFO     Initializing middleware                                                    
09:21:07-950538 DEBUG    Creating API                                                               
09:21:08-046642 DEBUG    SD-System-Info: benchmark data loaded:                                     
                         /apps/sd.next/extensions-builtin/sd-extensio
                         n-system-info/scripts/benchmark-data-local.json                            
09:21:08-376418 INFO     [AgentScheduler] Task queue is empty                                       
09:21:08-378829 INFO     [AgentScheduler] Registering APIs                                          
09:21:08-416376 DEBUG    Script: 0.32 app_started                                                   
                         /apps/sd.next/extensions-builtin/sd-webui-ag
                         ent-scheduler/scripts/task_scheduler.py                                    
09:21:08-528451 DEBUG    Scripts setup: ['X/Y/Z Grid:0.006']                                        
09:21:08-529444 DEBUG    Model metadata:                                                            
                         file="/apps/sd.next/metadata.json" no       
                         changes                                                                    
09:21:08-530231 DEBUG    Model auto load disabled                                                   
09:21:08-537183 DEBUG    Save: file="/apps/sd.next/config.json"      
                         json=38 bytes=2101                                                         
09:21:08-538114 INFO     Startup time: 15.27 { torch=3.98 gradio=0.84 libraries=4.37 extensions=2.57
                         face-restore=0.26 upscalers=0.15 ui-extra-networks=0.91 ui-img2img=0.07    
                         ui-settings=0.25 ui-extensions=0.71 ui-defaults=0.06 launch=0.32 api=0.09  
                         app-started=0.49 }                                                         
09:21:16-470029 INFO     MOTD: N/A                                                                  
09:21:19-191346 DEBUG    Themes: builtin=6 default=5 external=55                                    
09:21:19-417052 INFO     Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (X11; Linux  
                         x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0            
                         Safari/537.36                                                              
09:21:26-514781 INFO     Select: model="inpaintClothing_uberrealisticinpaint-inpainting             
                         [53beed7e09]"                                                              
09:21:26-516778 DEBUG    Load model weights: existing=False                                         
                         target=/apps/.local/share/stable-diffusion-data/models/stable-diffusion/
                         inpaintClothing_uberrealisticinpaint-inpainting.safetensors info=None      
Loading model: /apps/.local/share/stable-diffusion-data/models/stable-diffusion/inpaintClothing… 
09:21:30-754365 DEBUG    Desired Torch parameters: dtype=BF16 no-half=False no-half-vae=False       
                         upscast=False                                                              
09:21:30-755952 INFO     Setting Torch parameters: device=xpu dtype=torch.bfloat16                  
                         vae=torch.bfloat16 unet=torch.bfloat16 context=no_grad fp16=False bf16=True
09:21:30-757086 INFO     Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline         
                         file="/apps/.local/share/stable-diffusion-data/models/stable-diffusion/i
                         npaintClothing_uberrealisticinpaint-inpainting.safetensors" size=4068MB    
09:21:30-989089 ERROR    Diffusers failed loading:                                                  
                         model=/apps/.local/share/stable-diffusion-data/models/stable-diffusion/i
                         npaintClothing_uberrealisticinpaint-inpainting.safetensors                 
                         pipeline=Autodetect/NoneType Trying to set a tensor of shape               
                         torch.Size([320, 9, 3, 3]) in "weight" (which has shape torch.Size([320, 4,
                         3, 3])), this look incorrect.

Backend

Diffusers

Branch

Master

Model

SD 1.5

Acknowledgements

vladmandic commented 10 months ago

thanks for tracing it down - really helps with finding a solution. fixed in dev, will be merged to master in the next release.