[Bug]: Missing model directory, removing model

[huan085128]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

I encountered an error while creating the model. Trying to load both v1.5-pruned-emaonly.ckpt and v1.5-pruned.ckpt yields the following error:

Missing model directory, removing model: /home/stable-diffusion-webui/models/dreambooth/ph_panel/working/vae

Steps to reproduce the problem

I dont kown...

Commit and libraries

Dreambooth commit:
Dreambooth revision: [dc413a1](https://github.com/d8ahazard/sd_dreambooth_extension/commit/dc413a14379b165355502d9f65856c40a4bb5b6f)

pytorch: v2.0.0_cu117

Command Line Arguments

source venv/bin/activate
python webui.py

Console logs

Loading model from checkpoint.
Loading ckpt...
Pred and size are epsilon and 512, using config: /home/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/../configs/v1-training-default.yaml
v1 model loaded.
Trying to load: /home/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/../configs/v1-training-default.yaml
Converting unet...
Converting vae...
Exception setting up output: Error(s) in loading state_dict for AutoencoderKL:
        Missing key(s) in state_dict: "encoder.mid_block.attentions.0.to_q.weight", "encoder.mid_block.attentions.0.to_q.bias", "encoder.mid_block.attentions.0.to_k.weight", "encoder.mid_block.attentions.0.to_k.bias", "encoder.mid_block.attentions.0.to_v.weight", "encoder.mid_block.attentions.0.to_v.bias", "encoder.mid_block.attentions.0.to_out.0.weight", "encoder.mid_block.attentions.0.to_out.0.bias", "decoder.mid_block.attentions.0.to_q.weight", "decoder.mid_block.attentions.0.to_q.bias", "decoder.mid_block.attentions.0.to_k.weight", "decoder.mid_block.attentions.0.to_k.bias", "decoder.mid_block.attentions.0.to_v.weight", "decoder.mid_block.attentions.0.to_v.bias", "decoder.mid_block.attentions.0.to_out.0.weight", "decoder.mid_block.attentions.0.to_out.0.bias". 
        Unexpected key(s) in state_dict: "encoder.mid_block.attentions.0.query.weight", "encoder.mid_block.attentions.0.query.bias", "encoder.mid_block.attentions.0.key.weight", "encoder.mid_block.attentions.0.key.bias", "encoder.mid_block.attentions.0.value.weight", "encoder.mid_block.attentions.0.value.bias", "encoder.mid_block.attentions.0.proj_attn.weight", "encoder.mid_block.attentions.0.proj_attn.bias", "decoder.mid_block.attentions.0.query.weight", "decoder.mid_block.attentions.0.query.bias", "decoder.mid_block.attentions.0.key.weight", "decoder.mid_block.attentions.0.key.bias", "decoder.mid_block.attentions.0.value.weight", "decoder.mid_block.attentions.0.value.bias", "decoder.mid_block.attentions.0.proj_attn.weight", "decoder.mid_block.attentions.0.proj_attn.bias". 
Traceback (most recent call last):
  File "/home/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/sd_to_diff.py", line 1278, in extract_checkpoint
    vae.load_state_dict(converted_vae_checkpoint)
  File "/home/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1671, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for AutoencoderKL:
        Missing key(s) in state_dict: "encoder.mid_block.attentions.0.to_q.weight", "encoder.mid_block.attentions.0.to_q.bias", "encoder.mid_block.attentions.0.to_k.weight", "encoder.mid_block.attentions.0.to_k.bias", "encoder.mid_block.attentions.0.to_v.weight", "encoder.mid_block.attentions.0.to_v.bias", "encoder.mid_block.attentions.0.to_out.0.weight", "encoder.mid_block.attentions.0.to_out.0.bias", "decoder.mid_block.attentions.0.to_q.weight", "decoder.mid_block.attentions.0.to_q.bias", "decoder.mid_block.attentions.0.to_k.weight", "decoder.mid_block.attentions.0.to_k.bias", "decoder.mid_block.attentions.0.to_v.weight", "decoder.mid_block.attentions.0.to_v.bias", "decoder.mid_block.attentions.0.to_out.0.weight", "decoder.mid_block.attentions.0.to_out.0.bias". 
        Unexpected key(s) in state_dict: "encoder.mid_block.attentions.0.query.weight", "encoder.mid_block.attentions.0.query.bias", "encoder.mid_block.attentions.0.key.weight", "encoder.mid_block.attentions.0.key.bias", "encoder.mid_block.attentions.0.value.weight", "encoder.mid_block.attentions.0.value.bias", "encoder.mid_block.attentions.0.proj_attn.weight", "encoder.mid_block.attentions.0.proj_attn.bias", "decoder.mid_block.attentions.0.query.weight", "decoder.mid_block.attentions.0.query.bias", "decoder.mid_block.attentions.0.key.weight", "decoder.mid_block.attentions.0.key.bias", "decoder.mid_block.attentions.0.value.weight", "decoder.mid_block.attentions.0.value.bias", "decoder.mid_block.attentions.0.proj_attn.weight", "decoder.mid_block.attentions.0.proj_attn.bias". 
Missing model directory, removing model: /home/stable-diffusion-webui/models/dreambooth/ph_panel/working/vae
Restored system models.
Duration: 00:00:14

Additional information

No response

[xianweicui]

@huan085128 I had this error as well;

[huan085128]

@huan085128 I had this error as well;

I have solved this error:

in stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/sd_to_diff.py, in 1278 line. vae.load_state_dict(converted_vae_checkpoint, strict=False)

printi("Converting vae...")
# Convert the VAE model.
vae_config = create_vae_diffusers_config(original_config, image_size=image_size)
converted_vae_checkpoint = convert_ldm_vae_checkpoint(checkpoint, vae_config)

vae = AutoencoderKL(**vae_config)
vae.load_state_dict(converted_vae_checkpoint, strict=False)    # add strict=False
vae.save_pretrained(os.path.join(db_config.pretrained_model_name_or_path, "vae"), safe_serialization=True)
del vae

[Sniper199999]

@huan085128 I had this error as well;

I have solved this error:

in stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/sd_to_diff.py, in 1278 line. vae.load_state_dict(converted_vae_checkpoint, strict=False)

printi("Converting vae...")
# Convert the VAE model.
vae_config = create_vae_diffusers_config(original_config, image_size=image_size)
converted_vae_checkpoint = convert_ldm_vae_checkpoint(checkpoint, vae_config)

vae = AutoencoderKL(**vae_config)
vae.load_state_dict(converted_vae_checkpoint, strict=False)    # add strict=False
vae.save_pretrained(os.path.join(db_config.pretrained_model_name_or_path, "vae"), safe_serialization=True)
del vae

Can you tell me what will strict=False will do? Will it just ignore all the Missing and Unexpected keys? Because from what i can tell from the Errors is encoder.mid_block.attentions.0.to_q.weight and encoder.mid_block.attentions.0.query.weight should be the same keys but they have slightly different names. Ignoring these keys might affect the overall training right?

[huan085128]

@huan085128 I had this error as well;

I have solved this error: in stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/sd_to_diff.py, in 1278 line. vae.load_state_dict(converted_vae_checkpoint, strict=False)

printi("Converting vae...")
# Convert the VAE model.
vae_config = create_vae_diffusers_config(original_config, image_size=image_size)
converted_vae_checkpoint = convert_ldm_vae_checkpoint(checkpoint, vae_config)

vae = AutoencoderKL(**vae_config)
vae.load_state_dict(converted_vae_checkpoint, strict=False)    # add strict=False
vae.save_pretrained(os.path.join(db_config.pretrained_model_name_or_path, "vae"), safe_serialization=True)
del vae

Can you tell me what will strict=False will do? Will it just ignore all the Missing and Unexpected keys? Because from what i can tell from the Errors is encoder.mid_block.attentions.0.to_q.weight and encoder.mid_block.attentions.0.query.weight should be the same keys but they have slightly different names. Ignoring these keys might affect the overall training right?

There is allowance for some mismatches between the loaded state dict and the model's state dict. If there are keys in the loaded state dict that cannot be found in the model's state dict, or if there are keys whose shapes do not match, these keys will be ignored, and the corresponding parameters will not be loaded, but no error will be thrown.

Indeed, some keys will be ignored, and this will have some impact on fine-tuning, as some parameters are missing. However, I haven't found any other solutions.

[Sniper199999]

actually, I get this error on SD.Next. On A1111 (latest versions of dreambooth and A1111) there is no error. Might be some dependencies issue.

[github-actions[bot]]

This issue is stale because it has been open 5 days with no activity. Remove stale label or comment or this will be closed in 5 days

[uubuge]

@huan085128 I had this error as well;

I have solved this error:

in stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/sd_to_diff.py, in 1278 line. vae.load_state_dict(converted_vae_checkpoint, strict=False)

printi("Converting vae...")
# Convert the VAE model.
vae_config = create_vae_diffusers_config(original_config, image_size=image_size)
converted_vae_checkpoint = convert_ldm_vae_checkpoint(checkpoint, vae_config)

vae = AutoencoderKL(**vae_config)
vae.load_state_dict(converted_vae_checkpoint, strict=False)    # add strict=False
vae.save_pretrained(os.path.join(db_config.pretrained_model_name_or_path, "vae"), safe_serialization=True)
del vae

I had try it ,fix your network,it will work

[Diegolzgz]

@huan085128 I had this error as well;

I have solved this error:

in stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/sd_to_diff.py, in 1278 line. vae.load_state_dict(converted_vae_checkpoint, strict=False)

printi("Converting vae...")
# Convert the VAE model.
vae_config = create_vae_diffusers_config(original_config, image_size=image_size)
converted_vae_checkpoint = convert_ldm_vae_checkpoint(checkpoint, vae_config)

vae = AutoencoderKL(**vae_config)
vae.load_state_dict(converted_vae_checkpoint, strict=False)    # add strict=False
vae.save_pretrained(os.path.join(db_config.pretrained_model_name_or_path, "vae"), safe_serialization=True)
del vae

I tried this, but not working. The bug is: Exception setting up output: Error(s) in loading state_dict for AutoencoderKL: Missing key(s) in state_dict: "encoder.mid_block.attentions.0.to_q.weight", "encoder.mid_block.attentions.0.to_q.bias", "encoder.mid_block.attentions.0.to_k.weight", "encoder.mid_block.attentions.0.to_k.bias", "encoder.mid_block.attentions.0.to_v.weight", "encoder.mid_block.attentions.0.to_v.bias", "encoder.mid_block.attentions.0.to_out.0.weight", "encoder.mid_block.attentions.0.to_out.0.bias", "decoder.mid_block.attentions.0.to_q.weight", "decoder.mid_block.attentions.0.to_q.bias", "decoder.mid_block.attentions.0.to_k.weight", "decoder.mid_block.attentions.0.to_k.bias", "decoder.mid_block.attentions.0.to_v.weight", "decoder.mid_block.attentions.0.to_v.bias", "decoder.mid_block.attentions.0.to_out.0.weight", "decoder.mid_block.attentions.0.to_out.0.bias". Unexpected key(s) in state_dict: "encoder.mid_block.attentions.0.key.bias", "encoder.mid_block.attentions.0.key.weight", "encoder.mid_block.attentions.0.proj_attn.bias", "encoder.mid_block.attentions.0.proj_attn.weight", "encoder.mid_block.attentions.0.query.bias", "encoder.mid_block.attentions.0.query.weight", "encoder.mid_block.attentions.0.value.bias", "encoder.mid_block.attentions.0.value.weight", "decoder.mid_block.attentions.0.key.bias", "decoder.mid_block.attentions.0.key.weight", "decoder.mid_block.attentions.0.proj_attn.bias", "decoder.mid_block.attentions.0.proj_attn.weight", "decoder.mid_block.attentions.0.query.bias", "decoder.mid_block.attentions.0.query.weight", "decoder.mid_block.attentions.0.value.bias", "decoder.mid_block.attentions.0.value.weight". Traceback (most recent call last):

How can i solve this?

[darksinthe]

Same exact issue, since yesterday. Cannot create any new models. I tried adding the flag strict=False and it got it working, but after making a model i tried to train on the same model again and it completely broke the model, made SD unable to even create images in any model until i rebooted it. Want a real fix, is this just from a recent version update? i tried different versions of torch and it did not help

[severian42]

Same here, can't find a fix or the right spot in the code to pass it