[Feature]: Add SDXL Support to the Original backend

[TeutonJon78]

Feature description

The original backend is tried and true and generally very stable, especially compared to diffusers support. It also tends to use less VRAM. They've also released all the files as single model .safetensors, so the original backend should work.

auto1111 WebUI seems to be using the original backend for SDXL support so it seems technically possible.

Obviously, only the safetensors model versions would be supported and not the diffusers models or other SD models with the original backend.

Version Platform Description

No response

[TeutonJon78]

Currently, you can load the model and it will go through the steps of generating an image, but it just outputs black squares (brown during image generation preview).

It also spits out a lot of errors on model loading:

10:38:13-660243 ERROR    Error loading model weights: D:\Graphic Design\AI\SD models\Stable-diffusion\SDXL\sd_xl_base_1.0.safetensors Error(s) in loading state_dict for LatentDiffusion:
                                 size mismatch for model.diffusion_model.input_blocks.4.1.proj_in.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([640, 640, 1, 1]).
                                 size mismatch for model.diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is
                         torch.Size([640, 768]).
                                 size mismatch for model.diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is
                         torch.Size([640, 768]).
                                 size mismatch for model.diffusion_model.input_blocks.4.1.proj_out.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([640, 640, 1, 1]).
                                 size mismatch for model.diffusion_model.input_blocks.5.1.proj_in.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([640, 640, 1, 1]).
                                 size mismatch for model.diffusion_model.input_blocks.5.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is
                         torch.Size([640, 768]).
                                 size mismatch for model.diffusion_model.input_blocks.5.1.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is
                         torch.Size([640, 768]).
                                 size mismatch for model.diffusion_model.input_blocks.5.1.proj_out.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([640, 640, 1, 1]).
                                 size mismatch for model.diffusion_model.input_blocks.7.1.proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1, 1]).
                                 size mismatch for model.diffusion_model.input_blocks.7.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([1280, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.input_blocks.7.1.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([1280, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.input_blocks.7.1.proj_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1,
                         1]).
                                 size mismatch for model.diffusion_model.input_blocks.8.1.proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1, 1]).
                                 size mismatch for model.diffusion_model.input_blocks.8.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([1280, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.input_blocks.8.1.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([1280, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.input_blocks.8.1.proj_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1,
                         1]).
                                 size mismatch for model.diffusion_model.middle_block.1.proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1, 1]).
                                 size mismatch for model.diffusion_model.middle_block.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([1280, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.middle_block.1.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([1280, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.middle_block.1.proj_out.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.2.0.in_layers.0.weight: copying a param with shape torch.Size([1920]) from checkpoint, the shape in current model is torch.Size([2560]).
                                 size mismatch for model.diffusion_model.output_blocks.2.0.in_layers.0.bias: copying a param with shape torch.Size([1920]) from checkpoint, the shape in current model is torch.Size([2560]).
                                 size mismatch for model.diffusion_model.output_blocks.2.0.in_layers.2.weight: copying a param with shape torch.Size([1280, 1920, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280,
                         2560, 3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.2.0.skip_connection.weight: copying a param with shape torch.Size([1280, 1920, 1, 1]) from checkpoint, the shape in current model is
                         torch.Size([1280, 2560, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.in_layers.0.weight: copying a param with shape torch.Size([1920]) from checkpoint, the shape in current model is torch.Size([2560]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.in_layers.0.bias: copying a param with shape torch.Size([1920]) from checkpoint, the shape in current model is torch.Size([2560]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.in_layers.2.weight: copying a param with shape torch.Size([640, 1920, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280,
                         2560, 3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.in_layers.2.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.emb_layers.1.weight: copying a param with shape torch.Size([640, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.emb_layers.1.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.out_layers.0.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.out_layers.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.out_layers.3.weight: copying a param with shape torch.Size([640, 640, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280,
                         1280, 3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.out_layers.3.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.skip_connection.weight: copying a param with shape torch.Size([640, 1920, 1, 1]) from checkpoint, the shape in current model is torch.Size([1280,
                         2560, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.3.0.skip_connection.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.norm.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.norm.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.proj_in.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.proj_in.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn1.to_q.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn1.to_k.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn1.to_v.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn1.to_out.0.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn1.to_out.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.ff.net.0.proj.weight: copying a param with shape torch.Size([5120, 640]) from checkpoint, the shape in current model is
                         torch.Size([10240, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.ff.net.0.proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is
                         torch.Size([10240]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.ff.net.2.weight: copying a param with shape torch.Size([640, 2560]) from checkpoint, the shape in current model is
                         torch.Size([1280, 5120]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.ff.net.2.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn2.to_q.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn2.to_out.0.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.attn2.to_out.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.norm1.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.norm1.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.norm2.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.norm2.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.norm3.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.transformer_blocks.0.norm3.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.proj_out.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.3.1.proj_out.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.in_layers.0.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2560]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.in_layers.0.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([2560]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.in_layers.2.weight: copying a param with shape torch.Size([640, 1280, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280,
                         2560, 3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.in_layers.2.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.emb_layers.1.weight: copying a param with shape torch.Size([640, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.emb_layers.1.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.out_layers.0.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.out_layers.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.out_layers.3.weight: copying a param with shape torch.Size([640, 640, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280,
                         1280, 3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.out_layers.3.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.skip_connection.weight: copying a param with shape torch.Size([640, 1280, 1, 1]) from checkpoint, the shape in current model is torch.Size([1280,
                         2560, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.4.0.skip_connection.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.norm.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.norm.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.proj_in.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.proj_in.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn1.to_q.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn1.to_k.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn1.to_v.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn1.to_out.0.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn1.to_out.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.ff.net.0.proj.weight: copying a param with shape torch.Size([5120, 640]) from checkpoint, the shape in current model is
                         torch.Size([10240, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.ff.net.0.proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is
                         torch.Size([10240]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.ff.net.2.weight: copying a param with shape torch.Size([640, 2560]) from checkpoint, the shape in current model is
                         torch.Size([1280, 5120]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.ff.net.2.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn2.to_q.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn2.to_out.0.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.attn2.to_out.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm1.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm1.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm2.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm2.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm3.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm3.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.proj_out.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.4.1.proj_out.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.in_layers.0.weight: copying a param with shape torch.Size([960]) from checkpoint, the shape in current model is torch.Size([1920]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.in_layers.0.bias: copying a param with shape torch.Size([960]) from checkpoint, the shape in current model is torch.Size([1920]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.in_layers.2.weight: copying a param with shape torch.Size([640, 960, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280,
                         1920, 3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.in_layers.2.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.emb_layers.1.weight: copying a param with shape torch.Size([640, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.emb_layers.1.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.out_layers.0.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.out_layers.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.out_layers.3.weight: copying a param with shape torch.Size([640, 640, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280,
                         1280, 3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.out_layers.3.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.skip_connection.weight: copying a param with shape torch.Size([640, 960, 1, 1]) from checkpoint, the shape in current model is torch.Size([1280,
                         1920, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.5.0.skip_connection.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.norm.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.norm.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.proj_in.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.proj_in.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn1.to_q.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn1.to_k.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn1.to_v.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn1.to_out.0.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn1.to_out.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.ff.net.0.proj.weight: copying a param with shape torch.Size([5120, 640]) from checkpoint, the shape in current model is
                         torch.Size([10240, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.ff.net.0.proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is
                         torch.Size([10240]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.ff.net.2.weight: copying a param with shape torch.Size([640, 2560]) from checkpoint, the shape in current model is
                         torch.Size([1280, 5120]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.ff.net.2.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn2.to_q.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is
                         torch.Size([1280, 768]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn2.to_out.0.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is
                         torch.Size([1280, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.attn2.to_out.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.norm1.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.norm1.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.norm2.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.norm2.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.norm3.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is
                         torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.transformer_blocks.0.norm3.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.proj_out.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.5.1.proj_out.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.5.2.conv.weight: copying a param with shape torch.Size([640, 640, 3, 3]) from checkpoint, the shape in current model is torch.Size([1280, 1280, 3,
                         3]).
                                 size mismatch for model.diffusion_model.output_blocks.5.2.conv.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.in_layers.0.weight: copying a param with shape torch.Size([960]) from checkpoint, the shape in current model is torch.Size([1920]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.in_layers.0.bias: copying a param with shape torch.Size([960]) from checkpoint, the shape in current model is torch.Size([1920]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.in_layers.2.weight: copying a param with shape torch.Size([320, 960, 3, 3]) from checkpoint, the shape in current model is torch.Size([640, 1920,
                         3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.in_layers.2.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.emb_layers.1.weight: copying a param with shape torch.Size([320, 1280]) from checkpoint, the shape in current model is torch.Size([640, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.emb_layers.1.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.out_layers.0.weight: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.out_layers.0.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.out_layers.3.weight: copying a param with shape torch.Size([320, 320, 3, 3]) from checkpoint, the shape in current model is torch.Size([640, 640,
                         3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.out_layers.3.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.skip_connection.weight: copying a param with shape torch.Size([320, 960, 1, 1]) from checkpoint, the shape in current model is torch.Size([640,
                         1920, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.6.0.skip_connection.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.in_layers.0.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.in_layers.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([1280]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.in_layers.2.weight: copying a param with shape torch.Size([320, 640, 3, 3]) from checkpoint, the shape in current model is torch.Size([640, 1280,
                         3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.in_layers.2.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.emb_layers.1.weight: copying a param with shape torch.Size([320, 1280]) from checkpoint, the shape in current model is torch.Size([640, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.emb_layers.1.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.out_layers.0.weight: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.out_layers.0.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.out_layers.3.weight: copying a param with shape torch.Size([320, 320, 3, 3]) from checkpoint, the shape in current model is torch.Size([640, 640,
                         3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.out_layers.3.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.skip_connection.weight: copying a param with shape torch.Size([320, 640, 1, 1]) from checkpoint, the shape in current model is torch.Size([640,
                         1280, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.7.0.skip_connection.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.in_layers.0.weight: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([960]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.in_layers.0.bias: copying a param with shape torch.Size([640]) from checkpoint, the shape in current model is torch.Size([960]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.in_layers.2.weight: copying a param with shape torch.Size([320, 640, 3, 3]) from checkpoint, the shape in current model is torch.Size([640, 960,
                         3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.in_layers.2.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.emb_layers.1.weight: copying a param with shape torch.Size([320, 1280]) from checkpoint, the shape in current model is torch.Size([640, 1280]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.emb_layers.1.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.out_layers.0.weight: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.out_layers.0.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.out_layers.3.weight: copying a param with shape torch.Size([320, 320, 3, 3]) from checkpoint, the shape in current model is torch.Size([640, 640,
                         3, 3]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.out_layers.3.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.skip_connection.weight: copying a param with shape torch.Size([320, 640, 1, 1]) from checkpoint, the shape in current model is torch.Size([640,
                         960, 1, 1]).
                                 size mismatch for model.diffusion_model.output_blocks.8.0.skip_connection.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640]).

[ShmuelRonen]

I agree to the request. My experience is the same. black squares.

[vladmandic]

The original backend is tried and true and generally very stable, especially compared to diffusers support. It also tends to use less VRAM. They've also released all the files as single model .safetensors, so the original backend should work.

auto1111 WebUI seems to be using the original backend for SDXL support so it seems technically possible.

i agree that eventually all backends should support SD-XL, but the assumptions made here are just wrong.

this is FYI, so you understand why I said that: "original" backend is actually LDM pipeline from RML heavily hijacked by a1111 code. that is not compatibile with SD-XL ever. what is possible is take new SGM pipeline from SAI and hijack it in a similar manner so it exists side by side with LDM (that is what a1111 does now). result? right now pretty bad. a1111 with sgm pipeline takes faaar more vram than sdnext with diffuers. and doesn't have integrated refiner workflow at all, you need to run full base and then load refiner and run it via img2img - that is not how base/refiner are even intended to be working.

once SGM pipeline is better optimized, i'm totally not against porting it to SDNext, but right now focus is on Diffusers.

[ShmuelRonen]

The original backend is tried and true and generally very stable, especially compared to diffusers support. It also tends to use less VRAM. They've also released all the files as single model .safetensors, so the original backend should work.

auto1111 WebUI seems to be using the original backend for SDXL support so it seems technically possible.

i agree that eventually all backends should support SD-XL, but the assumptions made here are just wrong.

this is FYI, so you understand why I said that: "original" backend is actually LDM pipeline from RML heavily hijacked by a1111 code. that is not compatibile with SD-XL ever. what is possible is take new SGM pipeline from SAI and hijack it in a similar manner so it exists side by side with LDM (that is what a1111 does now). result? right now pretty bad. a1111 with sgm pipeline takes faaar more vram than sdnext with diffuers. and doesn't have integrated refiner workflow at all, you need to run full base and then load refiner and run it via img2img - that is not how base/refiner are even intended to be working.

once SGM pipeline is better optimized, i'm totally not against porting it to SDNext, but right now focus is on Diffusers.

According to these things, how does the Quicksetting: Diffusers allow loading from safetensors files work and does it create an optimal result?

[vladmandic]

safetensors or not, that's just a format of a file. diffusers vs ldm (original) is how stuff are executed once its loaded. don't mix the two. originally diffusers had issues loading from safetensors due to different formats, but nowadays it just works, so i'll remove that option soon.

[MrsHorrid]

but why do i get black squares how do i fix this?

[vladmandic]

There's a thread for that, this is NOT it.

[VictorZakharov]

@vladmandic What's the thread? Without a solution/workaround people will create more issues on GitHub, and spam them with comments. Here is another one with the same issue:

• https://github.com/vladmandic/automatic/issues/1858

[vladmandic]

Like you said, theres one. So use that, it really does not fit here. Misqualified issues cannot be worked on.

[ronbere]

Feature description

The original backend is tried and true and generally very stable, especially compared to diffusers support. It also tends to use less VRAM. They've also released all the files as single model .safetensors, so the original backend should work.

auto1111 WebUI seems to be using the original backend for SDXL support so it seems technically possible.

Obviously, only the safetensors model versions would be supported and not the diffusers models or other SD models with the original backend.

Version Platform Description

No response

You're right, A1111 works perfectly with sdxl models without having to modify anything, I mainly use Vlad's sd next, but I'm switching to A1111 to use sdxl.

[vladmandic]

This work will happen, but under different issue as by now there is too much self-propagating incorrect info and assumptions.

Of course, feel free to use any app, I just don't like to see incorrect info.