`convert_diffusers_to_original_stable_diffusion.py` produces files that `convert_original_stable_diffusion_to_diffusers.py` cannot open

[damian0815]

Describe the bug

Original stable diffusion checkpoints that have been extracted from diffusers folders via convert_diffusers_to_original_stable_diffusion.py cause convert_original_stable_diffusion_to_diffusers.py to fail with AttributeError: 'Attention' object has no attribute 'to_to_k':

File "/workspace/diffusers/scripts/convert_original_stable_diffusion_to_diffusers.py", line 154, in <module>
    pipe = download_from_original_stable_diffusion_ckpt(
  File "/workspace/diffusers/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1425, in download_from_original_stable_diffusion_ckpt
    set_module_tensor_to_device(vae, param_name, "cpu", value=param)
  File "/workspace/venv/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 255, in set_module_tensor_to_device
    new_module = getattr(module, split)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1614, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Attention' object has no attribute 'to_to_k'

The cause is these lines in convert_from_ckpt.py:

        new_item = new_item.replace("q.weight", "to_q.weight")
        new_item = new_item.replace("q.bias", "to_q.bias")

        new_item = new_item.replace("k.weight", "to_k.weight")
        new_item = new_item.replace("k.bias", "to_k.bias")

        new_item = new_item.replace("v.weight", "to_v.weight")
        new_item = new_item.replace("v.bias", "to_v.bias")

Because the loaded checkpoint already has keys labelled to_q.weight etc, this method renames them to to_to_q.weight causing downstream logic to fail.

The replacement logic should not do this.

Reproduction

To reproduce, convert a .safetensors checkpoint to diffusers, then convert the diffusers folder back to .safetensors:

python scripts/convert_diffusers_to_original_stable_diffusion.py --model_path model-diffusers-folder --checkpoint_path model.safetensors --use_safetensors --half
python scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path model.safetensors --from_safetensors --to_safetensors --original_config config.yaml --dump_path model-diffusers-folder-loop --half

expected result: works actual result: AttributeError: 'Attention' object has no attribute 'to_to_k'

Logs

No response

System Info

diffusers main (0d81e543a2b048d7b1f558fb1006d10a062a6e9e)

- `diffusers` version: 0.21.0.dev0
- Platform: Linux-5.15.0-79-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- PyTorch version (GPU?): 2.0.1 (True)
- Huggingface_hub version: 0.16.4
- Transformers version: 4.31.0
- Accelerate version: 0.21.0
- xFormers version: 0.0.21
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

Who can help?

@patrickvonplaten

[damian0815]

Note that this scenario commonly arises because of finetuning and distribution: finetuner produces diffusers folder, which is converted to .safetensors for distribution on the (very bad) platform civitai. user downloads the checkpoint and then attempts to convert it back to .safetensors for use in eg InvokeAI, or for further finetuning

[patrickvonplaten]

That's unfortunate :-/ We don't really maintain convert_diffusers_to_original_stable_diffusion.py it was added by the community

[patrickvonplaten]

Any chance you could add a reproducible code snippet here maybe? :-)

Also, does it work when removing:

        new_item = new_item.replace("q.weight", "to_q.weight")
        new_item = new_item.replace("q.bias", "to_q.bias")

        new_item = new_item.replace("k.weight", "to_k.weight")
        new_item = new_item.replace("k.bias", "to_k.bias")

        new_item = new_item.replace("v.weight", "to_v.weight")
        new_item = new_item.replace("v.bias", "to_v.bias")

we indeed deprecated the old "q.weight" syntax - so these lines should not exist anymore I think

[ghunkins]

Adding this code snippet here, in case anyone needs a quick fix to this. This converts a corrupted safetensors file to one that can be loaded using the from_single_file method.

It appears that the incompatible keys are localized to the VAE.

from safetensors import safe_open
from safetensors.torch import save_file

def fix_diffusers_model_conversion(load_path: str, save_path: str):
    # load original
    tensors = {}
    with safe_open(load_path, framework="pt") as f:
        for key in f.keys():
            tensors[key] = f.get_tensor(key)

    # migrate
    new_tensors = {}
    for k, v in tensors.items():
        new_key = k
        # only fix the vae
        if 'first_stage_model.' in k:
            # migrate q, k, v keys
            new_key = new_key.replace('.to_q.weight', '.q.weight')
            new_key = new_key.replace('.to_q.bias', '.q.bias')
            new_key = new_key.replace('.to_k.weight', '.k.weight')
            new_key = new_key.replace('.to_k.bias', '.k.bias')
            new_key = new_key.replace('.to_v.weight', '.v.weight')
            new_key = new_key.replace('.to_v.bias', '.v.bias')
        new_tensors[new_key] = v

    # save
    save_file(new_tensors, save_path)

[nayan-dhabarde]

Getting the same issue when converting the ckpt file

First converted to ckpt using convert_diffusers_to_original_stable_diffusion.py

Then tried to convert back to diffusers using convert_original_stable_diffusion_to_diffusers.py

Doing this because the diffusers output is huge in size, where as ckpt is only 2gb with half option

It is not possible for me to zip and store diffusers it is huge in size

[damian0815]

Doing this because the diffusers output is huge in size, where as ckpt is only 2gb with half option

you can convert the pipeline to float16 before you save it:

pipe = StableDiffusionPipeline.from_pretrained(...)
pipe.save('blah_big') # 4GB
pipe.to('cpu', dtype=torch.float16)
pipe.save('blah_small') # 2GB

[nayan-dhabarde]

@damian0815 I am using dreambooth training to generate my model

Here is my accelerate config file, it is using mixed_precision fp16 is the above suggestion you made and this the same thing

compute_environment: LOCAL_MACHINE distributed_type: 'NO' downcast_bf16: 'no' gpu_ids: all machine_rank: 0 main_training_function: main mixed_precision: fp16 num_machines: 1 num_processes: 1 rdzv_backend: static same_network: true tpu_env: [] tpu_use_cluster: false tpu_use_sudo: false use_cpu: false

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[duskfallcrew]

I'm struggling to get one of my own to go, and it doesnt seem to be corrupted, but it gives me this error, i'm not very good at programming, and 90% of the time the convert script works, but ONE OF MY OWN merged models that's in safetensors won't convert...

Edit: https://github.com/huggingface/diffusers/issues/4790#issuecomment-1745398865. This works but if you're on colab i modified it using chat GPT:

from safetensors import safe_open from safetensors.torch import save_file

def fix_diffusers_model_conversion(load_path: str, save_path: str):

# Specify the path to the uploaded file in Colab
colab_load_path = '/content/models/theAgendaMix_v35.safetensors'  # Update this with the actual path

# Specify the path where you want to save the fixed file in Colab
colab_save_path = '/content/models/FixedModels/theAgendaMix_v35.safetensors'  # Update this with the actual path

# load original
tensors = {}
with safe_open(colab_load_path, framework="pt") as f:
    for key in f.keys():
        tensors[key] = f.get_tensor(key)

# migrate
new_tensors = {}
for k, v in tensors.items():
    new_key = k
    # only fix the vae
    if 'first_stage_model.' in k:
        # migrate q, k, v keys
        new_key = new_key.replace('.to_q.weight', '.q.weight')
        new_key = new_key.replace('.to_q.bias', '.q.bias')
        new_key = new_key.replace('.to_k.weight', '.k.weight')
        new_key = new_key.replace('.to_k.bias', '.k.bias')
        new_key = new_key.replace('.to_v.weight', '.v.weight')
        new_key = new_key.replace('.to_v.bias', '.v.bias')
    new_tensors[new_key] = v

# save
save_file(new_tensors, colab_save_path)

Example usage

fix_diffusers_model_conversion(load_path='/content/drive/path/to/your/original_file.pth', save_path='/content/drive/path/to/save/fixed_file.pth')

I don't program, i duct tape :3

[InB4DevOps]

Adding this code snippet here, in case anyone needs a quick fix to this. This converts a corrupted safetensors file to one that can be loaded using the from_single_file method.

It appears that the incompatible keys are localized to the VAE.

from safetensors import safe_open
from safetensors.torch import save_file

def fix_diffusers_model_conversion(load_path: str, save_path: str):
    # load original
    tensors = {}
    with safe_open(load_path, framework="pt") as f:
        for key in f.keys():
            tensors[key] = f.get_tensor(key)

    # migrate
    new_tensors = {}
    for k, v in tensors.items():
        new_key = k
        # only fix the vae
        if 'first_stage_model.' in k:
            # migrate q, k, v keys
            new_key = new_key.replace('.to_q.weight', '.q.weight')
            new_key = new_key.replace('.to_q.bias', '.q.bias')
            new_key = new_key.replace('.to_k.weight', '.k.weight')
            new_key = new_key.replace('.to_k.bias', '.k.bias')
            new_key = new_key.replace('.to_v.weight', '.v.weight')
            new_key = new_key.replace('.to_v.bias', '.v.bias')
        new_tensors[new_key] = v

    # save
    save_file(new_tensors, save_path)

Thanks a lot!