search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.61k stars 413 forks source link

[Issue]: Error(s) in loading state_dict for LatentDiffusion: #2689

Closed vampireLibrarianMonk closed 9 months ago

vampireLibrarianMonk commented 9 months ago

Issue Description

I am having an issue with the following model upon initial install:

arget=/home/flaniganp/automatic/models/Stable-diffusion/sd_turbo.safetensors

Version Platform Description

I am working on a Ubuntu 22.4.03 and an Intel Sparkle 16GB GPU.

Relevant log output

Loading model: /home/flaniganp/automatic/models/Stable-diffusion/sd_turbo.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/5.2 GB -:--:--
07:23:26-813082 DEBUG    Load model: name=/home/flaniganp/automatic/models/Stable-diffusion/sd_turbo.safetensors dict=True                                                                                                                                                                
07:23:26-814978 DEBUG    Desired Torch parameters: dtype=BF16 no-half=False no-half-vae=False upscast=False                                                                                                                                                                               
07:23:26-816000 INFO     Setting Torch parameters: device=xpu dtype=torch.bfloat16 vae=torch.bfloat16 unet=torch.bfloat16 context=no_grad fp16=False bf16=True                                                                                                                            
07:23:26-817466 DEBUG    Model dict loaded: {'ram': {'used': 5.66, 'total': 125.71}, 'gpu': {'used': 0.0, 'total': 15.91}, 'retries': 0, 'oom': 0}                                                                                                                                        
07:23:26-831465 DEBUG    Model config loaded: {'ram': {'used': 5.66, 'total': 125.71}, 'gpu': {'used': 0.0, 'total': 15.91}, 'retries': 0, 'oom': 0}                                                                                                                                      
07:23:30-897451 INFO     LDM: LatentDiffusion: mode=eps                                                                                                                                                                                                                                   
07:23:30-898328 INFO     LDM: DiffusionWrapper params=859.52M                                                                                                                                                                                                                             
07:23:30-904779 DEBUG    Model created from config: /home/flaniganp/automatic/configs/v1-inference.yaml                                                                                                                                                                                   
07:23:30-905552 INFO     Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline file="/home/flaniganp/automatic/models/Stable-diffusion/sd_turbo.safetensors" size=4973MB                                                                                                     
07:23:30-906445 DEBUG    Model weights loading: {'ram': {'used': 9.66, 'total': 125.71}, 'gpu': {'used': 0.0, 'total': 15.91}, 'retries': 0, 'oom': 0}                                                                                                                                    
07:23:31-290783 ERROR    Error loading model weights: /home/flaniganp/automatic/models/Stable-diffusion/sd_turbo.safetensors                                                                                                                                                              
07:23:31-293273 ERROR    Error(s) in loading state_dict for LatentDiffusion:     size mismatch for model.diffusion_model.input_blocks.1.1.proj_in.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).  
07:23:31-302869 DEBUG    Unload weights model: {'ram': {'used': 12.99, 'total': 125.71}, 'gpu': {'used': 0.0, 'total': 15.91}, 'retries': 0, 'oom': 0}                                                                                                                                    
07:23:31-606338 DEBUG    gc: collected=951 device=xpu {'ram': {'used': 12.99, 'total': 125.71}, 'gpu': {'used': 0.0, 'total': 15.91}, 'retries': 0, 'oom': 0}                                                                                                                             
07:23:31-609503 DEBUG    Model weights unloaded: {'ram': {'used': 12.99, 'total': 125.71}, 'gpu': {'used': 0.0, 'total': 15.91}, 'retries': 0, 'oom': 0} op=model                                                                                                                         
07:23:31-873067 WARNING  Model not loaded                                                                                                                                                                                                                                                 
07:23:55-425973 INFO     MOTD: N/A                                                                                                                                                                                                                                                        
07:23:57-819103 DEBUG    Themes: builtin=9 default=5 external=55                                                                                                                                                                                                                          
07:23:58-027677 INFO     Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0                                                                                                                                 
07:24:00-012516 DEBUG    Server: alive=True jobs=1 requests=434 uptime=284 memory=7.2/125.71 backend=Backend.ORIGINAL state=idle

Backend

Original

Branch

Master

Model

SD 1.5

Acknowledgements

vladmandic commented 9 months ago

SD turbo si SDXL model and requires that you run in diffusers backend.

vampireLibrarianMonk commented 9 months ago

I am new to stable diffusion and this application how does one go about moving those models to the diffusers section? Currently it is in the Reference section as a default.

vladmandic commented 9 months ago

you don't move models - you set backend to diffusers in settings and restart the server.

vampireLibrarianMonk commented 9 months ago

Ok I changed to diffusers in the settings and restarted the server

got the following error

AttributeError: 'StableDiffusionPipeline' object has no attribute 'alphas_cumprod'

18:51:00-228448 DEBUG Setting changed: key=sd_model_checkpoint, value=stabilityai/stable-diffusion-2-1-base 18:51:05-234172 ERROR Running script process: extensions-builtin\sd-webui-controlnet\scripts\controlnet.py: AttributeError ┌───────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────┐ │ C:\Users\flaniganp\repos\automatic\modules\scripts.py:518 in process │ │ │ │ 517 │ │ │ │ args = p.per_script_args.get(script.title(), p.script_args[script.args_f │ │ > 518 │ │ │ │ script.process(p, *args, **kwargs) │ │ 519 │ │ │ except Exception as e: │ │ │ │ C:\Users\flaniganp\repos\automatic\extensions-builtin\sd-webui-controlnet\scripts\controlnet.py:1092 in process │ │ │ │ 1091 │ │ if not Script.process_has_sdxl_refiner(p): │ │ > 1092 │ │ │ self.controlnet_hack(p) │ │ 1093 │ │ return │ │ │ │ C:\Users\flaniganp\repos\automatic\extensions-builtin\sd-webui-controlnet\scripts\controlnet.py:1077 in controlnet_hack │ │ │ │ 1076 │ │ │ │ > 1077 │ │ self.controlnet_main_entry(p) │ │ 1078 │ │ if getattr(shared.cmd_opts, 'controlnet_tracemalloc', False): │ │ │ │ C:\Users\flaniganp\repos\automatic\extensions-builtin\sd-webui-controlnet\scripts\controlnet.py:717 in controlnet_main_entry │ │ │ │ 716 │ │ sd_ldm = p.sd_model │ │ > 717 │ │ unet = sd_ldm.model.diffusion_model │ │ 718 │ │ self.noise_modifier = None │ │ │ │ C:\Users\flaniganp\repos\automatic\venv\lib\site-packages\diffusers\configuration_utils.py:142 in getattr │ │ │ │ 141 │ │ │ │ > 142 │ │ raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'" │ │ 143 │ └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ AttributeError: 'StableDiffusionPipeline' object has no attribute 'model' 18:51:05-646630 ERROR Prompt parser encode: 'BaseModelOutputWithPooling' object has no attribute 'text_embeds' 18:51:05-646630 DEBUG Diffuser pipeline: StableDiffusionPipeline task=DiffusersTaskType.TEXT_2_IMAGE set={'prompt': 1, 'negative_prompt': 1, 'guidance_scale': 4, 'generator': device(type='xpu'), 'output_type': 'latent', 'num_inference_steps': 20, 'eta': 1.0, 'guidance_rescale': 0.7, 'width': 512, 'height': 512, 'parser': 'Fixed attention'} 18:51:05-646630 ERROR Exception: 'StableDiffusionPipeline' object has no attribute 'alphas_cumprod' 18:51:05-646630 ERROR Arguments: args=('task(g20evzb755o744d)', 'guinea pig', '', [], 20, None, None, True, False, False, 1, 1, 4, 4, 0.7, 1, -1.0, -1.0, 0, 0, 0, 512, 512, False, 0.5, 2, 'None', False, 20, 0, 0, 5, 0.8, '', '', False, 4, 0.95, False, 1, 1, False, 0.6, 1, [], 0, 3, 1, 1, 0.8, 8, 64, True, False, False, 'positive', 'comma', 0, False, False, '', 'None', True, 0, 'None', 2, True, 1, 0, 0, '', [], 0, '', [], 0, '', [], False, True, False, False, False, False, 0, None, None, False, None, None, False, None, None, False, 50, 'None', 16, 'None', 1, True, 'None', 2, True, 1, 0, True, 'none', 0.5, None, UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None)) kwargs={} 18:51:05-687500 ERROR gradio call: AttributeError ┌───────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────┐ │ C:\Users\flaniganp\repos\automatic\modules\call_queue.py:31 in f │ │ │ │ 30 │ │ │ try: │ │ > 31 │ │ │ │ res = func(*args, **kwargs) │ │ 32 │ │ │ │ progress.record_results(id_task, res) │ │ │ │ C:\Users\flaniganp\repos\automatic\modules\txt2img.py:88 in txt2img │ │ │ │ 87 │ if processed is None: │ │ > 88 │ │ processed = processing.process_images(p) │ │ 89 │ p.close() │ │ │ │ C:\Users\flaniganp\repos\automatic\modules\processing.py:760 in process_images │ │ │ │ 759 │ │ │ with context_hypertile_vae(p), context_hypertile_unet(p): │ │ > 760 │ │ │ │ res = process_images_inner(p) │ │ 761 │ │ │ │ C:\Users\flaniganp\repos\automatic\extensions-builtin\sd-webui-controlnet\scripts\batch_hijack.py:42 in processing_process_images_hijack │ │ │ │ 41 │ │ │ # we are not in batch mode, fallback to original function │ │ > 42 │ │ │ return getattr(processing, 'controlnet_original_process_images_inner')(p, │ │ 43 │ │ │ │ C:\Users\flaniganp\repos\automatic\modules\processing.py:922 in process_images_inner │ │ │ │ 921 │ │ │ │ from modules.processing_diffusers import process_diffusers │ │ > 922 │ │ │ │ x_samples_ddim = process_diffusers(p) │ │ 923 │ │ │ else: │ │ │ │ ... 2 frames hidden ... │ │ │ │ C:\Users\flaniganp\repos\automatic\modules\sd_samplers.py:63 in create_sampler │ │ │ │ 62 │ elif shared.backend == shared.Backend.DIFFUSERS: │ │ > 63 │ │ sampler = config.constructor(model) │ │ 64 │ │ if not hasattr(model, 'scheduler_config'): │ │ │ │ C:\Users\flaniganp\repos\automatic\modules\sd_samplers_compvis.py:11 in │ │ │ │ 10 samplers_data_compvis = [ │ │ > 11 │ sd_samplers_common.SamplerData('UniPC', lambda model: VanillaStableDiffusionSampler( │ │ 12 │ sd_samplers_common.SamplerData('DDIM', lambda model: VanillaStableDiffusionSampler(l │ │ │ │ C:\Users\flaniganp\repos\automatic\modules\sd_samplers_compvis.py:19 in init │ │ │ │ 18 │ def init__(self, constructor, sd_model): │ │ > 19 │ │ self.sampler = constructor(sd_model) │ │ 20 │ │ self.is_ddim = hasattr(self.sampler, 'p_sample_ddim') │ │ │ │ C:\Users\flaniganp\repos\automatic\modules\unipc\sampler.py:16 in init │ │ │ │ 15 │ │ self.after_sample = None │ │ > 16 │ │ self.register_buffer('alphas_cumprod', to_torch(model.alphas_cumprod)) │ │ 17 │ │ │ │ C:\Users\flaniganp\repos\automatic\venv\lib\site-packages\diffusers\configuration_utils.py:142 in getattr │ │ │ │ 141 │ │ │ │ > 142 │ │ raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'" │ │ 143 │ └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ AttributeError: 'StableDiffusionPipeline' object has no attribute 'alphas_cumprod' 18:51:59-878052 DEBUG Server: alive=True jobs=1 requests=264 uptime=2521 memory=4.39/127.93 backend=Backend.DIFFUSERS state=idle

I would like to follow this instruction set

https://www.reddit.com/r/StableDiffusion/comments/16tagbu/vlad_sdnext_sdxl_directml/?rdt=49186

However, where is the checkpoint folder and direction set for the first two files listed here:

https://github.com/vladmandic/automatic/wiki/SD-XL

vladmandic commented 9 months ago

do a full server restart. you just restarted from gui.

vampireLibrarianMonk commented 9 months ago

Ok i did a shutdown then restarted and its now working.