[Bug]: New RunwayML Inpainting error

[iChristGit]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

Ive seen that it mentioned in the main page about the new Inpainting models supported within the webui Im on latest update, using the "sd-v1-5-inpainting.ckpt" from hugginface and just getting a red error, while all my normal models work (waifu, pokemon model etc) Is that still in progress or am i missing something?

Steps to reproduce the problem

1. Update to latest update
2. Download the inpainting model from hugginface
3. Try using it within webui

https://huggingface.co/runwayml/stable-diffusion-inpainting/tree/main

What should have happened?

I should be able to use it without any changes I think

Commit where the problem happens

No response

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

Command Line Arguments

No response

Additional information, context and logs

No response

[barleyj21]

I've managed to run it on latest commit https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/1ed227b3b57f06f4152be3bfc9f83b0a839a2604, but changing back to 1.4 model generated some weird pc behavior beside red error - i've got screen going black for a few seconds repeatedly and now my gpu disappeared from windows 10 - task manager - performance tab data

[iChristGit]

Seems like using first the " v1-5-pruned.ckpt" and then the "sd-v1-5-inpainting.ckpt" works fine.. If thats a bug or intentional I dont know buy just a heads up if someone has the same issue.

[Mozoloa]

Check out the log in the console when that happens and send it here !

[arpitest]

File "/home/arpi/stable/GUI/stable-diffusion-webui/webui.py", line 40, in f res = func(*args, **kwargs) File "/home/arpi/stable/GUI/stable-diffusion-webui/webui.py", line 85, in shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights(shared.sd_model))) File "/home/arpi/stable/GUI/stable-diffusion-webui/modules/sd_models.py", line 262, in reload_model_weights load_model_weights(sd_model, checkpoint_info) File "/home/arpi/stable/GUI/stable-diffusion-webui/modules/sd_models.py", line 170, in load_model_weights missing, extra = model.load_state_dict(sd, strict=False) File "/home/conda/lib/python3.8/site-packages/torch-1.12.1-py3.8-linux-x86_64.egg/torch/nn/modules/module.py", line 1604, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for LatentDiffusion: size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

according to model card it has more channels:

stable-diffusion-inpainting Resumed from stable-diffusion-v1-5 - then 440,000 steps of inpainting training at resolution 512x512 on “laion-aesthetics v2 5+” and 10% dropping of the text-conditioning. For inpainting, the UNet has 5 additional input channels (4 for the encoded masked-image and 1 for the mask itself) whose weights were zero-initialized after restoring the non-inpainting checkpoint. During training, we generate synthetic masks and in 25% mask everything.

[Mozoloa]

Looks like your inpainting is still only sending the 4 original inputs and not sending the 5 new ones associated with the inpainting model, I haven't tested myself but if no one else has this bug then maybe a clean install could fix it

[barleyj21]

this is the error i get trying to load it along with VAE weights which came along the model on huggingface:

...\webui\venv\Scripts\Python.exe" Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] Commit hash: Installing requirements for Web UI Launching Web UI with arguments: --xformers LatentDiffusion: Running in eps-prediction mode DiffusionWrapper has 859.52 M params. making attention of type 'vanilla' with 512 in_channels Working with z of shape (1, 4, 32, 32) = 4096 dimensions. making attention of type 'vanilla' with 512 in_channels Loading weights [a9263745] from E:\Automatic111\webui\models\Stable-diffusion\wsd-v1-5-pruned.ckpt Global Step: 840000 Loading VAE weights from: E:\Automatic111\webui\models\Stable-diffusion\wsd-v1-5-pruned.vae.pt Traceback (most recent call last): File "E:\Automatic111\webui\launch.py", line 206, in start_webui() File "E:\Automatic111\webui\launch.py", line 201, in start_webui webui.webui() File "E:\Automatic111\webui\webui.py", line 123, in webui initialize() File "E:\Automatic111\webui\webui.py", line 84, in initialize shared.sd_model = modules.sd_models.load_model() File "E:\Automatic111\webui\modules\sd_models.py", line 228, in load_model load_model_weights(sd_model, checkpoint_info) File "E:\Automatic111\webui\modules\sd_models.py", line 189, in load_model_weights vae_dict = {k: v for k, v in vae_ckpt["state_dict"].items() if k[0:4] != "loss"} KeyError: 'state_dict'

And here's the normal launch log with new model (weirdly it has exact size to the byte as 1.4) without adding VAE:

...webui\venv\Scripts\Python.exe" Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] Commit hash: Installing requirements for Web UI Launching Web UI with arguments: --xformers LatentDiffusion: Running in eps-prediction mode DiffusionWrapper has 859.52 M params. making attention of type 'vanilla' with 512 in_channels Working with z of shape (1, 4, 32, 32) = 4096 dimensions. making attention of type 'vanilla' with 512 in_channels Loading weights [a9263745] from E:\Automatic111\webui\models\Stable-diffusion\wsd-v1-5-pruned.ckpt Global Step: 840000 Applying xformers cross attention optimization. Model loaded. Loaded a total of 4 textual inversion embeddings. Embeddings: mNK, ln-1500, ln-4000, lny Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().

[ProGamerGov]

It's working fine for me.

You could try deleting the venv folder / directory, and then running the setup / install script again. That may fix your issue.

[arpitest]

i'm still getting this size mismatch ERROR when switching checkpoint:

RuntimeError: Error(s) in loading state_dict for LatentDiffusion: size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

[arpitest]

ok i've finaly found what is the problem: the filename does matter. after renaming inpaint-1.5.ckpt to sd-v1-5-inpainting.ckpt it loads. heh.

[ProducerMatt]

after renaming inpaint-1.5.ckpt to sd-v1-5-inpainting.ckpt it loads. heh.

Mine is also named that, but still getting the PyTorch size mismatch error. Does it need to match the name of the normal SD checkpoint or something?

[hunterfrerich]

I'm getting the same error with a 4GB 1050ti. Is anyone else getting the error using a 10xx card?

[ProducerMatt]

I'm getting the same error with a 4GB 1050ti. Is anyone else getting the error using a 10xx card?

I was with a 1080. However it went away once I updated Auto.

[ClashSAN]

@iChristGit still an issue for you with the latest updates? The code tweaks for inpainting model to work is designed to latch on to any model named "inpainting" so be sure you haven't changed the name.