`scripts/convert_original_stable_diffusion_to_diffusers.py` tells me `HeaderTooLarge`.

[landmann]

Describe the bug

I'm trying to download a CivitAI model to use with diffusers. I'm following this tutorial https://medium.com/mlearning-ai/using-civitai-models-with-diffusers-package-45e0c475a67e, but it's giving me the following error:

Traceback (most recent call last):
  File "/home/ubuntu/notebooks/convert_original_stable_diffusion_to_diffusers.py", line 160, in <module>
    pipe = download_from_original_stable_diffusion_ckpt(
  File "/home/ubuntu/myenv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1257, in download_from_original_stable_diffusion_ckpt
    checkpoint = safe_load(checkpoint_path_or_dict, device="cpu")
  File "/home/ubuntu/myenv/lib/python3.10/site-packages/safetensors/torch.py", line 308, in load_file
    with safe_open(filename, framework="pt", device=device) as f:
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

I'm trying to download this model https://civitai.com/api/download/models/303786.

Any ideas?

Reproduction

from pathlib import Path
import requests
import requests, os, subprocess
import os

def get_civitai_model(name, url):
    print(url, name)
    response = requests.get(url, stream=True)

    if response.status_code == 200:
        # Define the name of the file you want to save the content to
        filename = f"{name}.safetensors"

        # Create a variable for the folder name
        folder_name = MODEL_CACHE_FOLDER / name

        # Create the directory if it does not exist
        os.makedirs(folder_name, exist_ok=True)

        # Save the content to the file in the created directory
        print("⏳ Saving safetensors...")

        file_path = f"{folder_name}/{filename}"
        if not os.path.exists(file_path):
            with open(file_path, 'wb') as file:
                for chunk in response.iter_content(chunk_size=8192):
                    file.write(chunk)

        print("👍 Saved safetensor. Converting original to diffusers...")

        process = [
            "python",
            "convert_original_stable_diffusion_to_diffusers.py",
            "--checkpoint_path",
            f"{folder_name}/{filename}",
            "--dump_path",
            folder_name,
            "--from_safetensors",
            "--num_in_channels", "9"
        ]
        try:
            result = subprocess.run(process, check=True, capture_output=True, text=True)
            print(result.stdout)
            print(result.stderr)
        except subprocess.CalledProcessError as e:
            print(e.stderr)

get_civitai_model(
    name="realDreamLCM",
    url="https://civitai.com/api/download/models/303786"
)

Logs

No response

System Info

ubuntu 🙂

Who can help?

No response

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

cc @DN6

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.