Runtime error with from_single_file

suzukimain commented 5 months ago

Describe the bug

Versioning diffusers to 0.29.0.dev0 did not solve the problem.

Reproduction

pip install diffusers from diffusers import StableDiffusionPipeline pipe = StableDiffusionPipeline.from_single_file("https://huggingface.co/gsdf/Counterfeit-V2.5/blob/main/Counterfeit-V2.5_fp16.safetensors")

Logs

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict, assign)
   2187 
   2188         if len(error_msgs) > 0:
-> 2189             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
   2190                                self.__class__.__name__, "\n\t".join(error_msgs)))
   2191         return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
    Unexpected key(s) in state_dict: "text_model.embeddings.position_ids".

System Info

🤗 Diffusers version: 0.28.2
Platform: Ubuntu 22.04.3 LTS - Linux-6.1.85+-x86_64-with-glibc2.35
Running on a notebook?: No
Running on Google Colab?: No
Python version: 3.10.12
PyTorch version (GPU?): 2.3.0+cu121 (True)
Flax version (CPU?/GPU?/TPU?): 0.8.4 (gpu)
Jax version: 0.4.26
JaxLib version: 0.4.26
Huggingface_hub version: 0.23.2
Transformers version: 4.41.2
Accelerate version: not installed
PEFT version: not installed
Bitsandbytes version: not installed
Safetensors version: 0.4.3
xFormers version: not installed
Accelerator: Tesla T4, 15360 MiB VRAM
Using GPU in script?:
Using distributed or parallel set-up in script?:

Who can help?

No response

sayakpaul commented 5 months ago

Cc: @DN6

asomoza commented 5 months ago

Hi, thanks for reporting. I tested this and It works with the 0.28.2 version. It's weird that it doesn't work for you. I suggest you try uninstalling diffusers and then install it again.

- 🤗 Diffusers version: 0.28.2

from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_single_file(
    "https://huggingface.co/gsdf/Counterfeit-V2.5/blob/main/Counterfeit-V2.5_fp16.safetensors"
).to("cuda")

image = pipe(prompt="a happy capybara jumping on the street").images[0]
image.save("result.png")

result

suzukimain commented 5 months ago

Thank you for your swift response. I tried again, but it fails.

https://github.com/huggingface/diffusers/assets/131413573/4bde0c67-dd98-448f-8617-dd710296fef0

Detailed log.

RuntimeError Traceback (most recent call last)

in () 4 5 ----> 6 pipe = StableDiffusionPipeline.from_single_file( 7 "https://huggingface.co/gsdf/Counterfeit-V2.5/blob/main/Counterfeit-V2.5_fp16.safetensors" 8 ).to("cuda") 4 frames /usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py in _inner_fn(*args, **kwargs) 112 kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs) 113 --> 114 return fn(*args, **kwargs) 115 116 return _inner_fn # type: ignore /usr/local/lib/python3.10/dist-packages/diffusers/loaders/single_file.py in from_single_file(cls, pretrained_model_link_or_path, **kwargs) 489 else: 490 try: --> 491 loaded_sub_model = load_single_file_sub_model( 492 library_name=library_name, 493 class_name=class_name, /usr/local/lib/python3.10/dist-packages/diffusers/loaders/single_file.py in load_single_file_sub_model(library_name, class_name, name, checkpoint, pipelines, is_pipeline_module, cached_model_config_path, original_config, local_files_only, torch_dtype, is_legacy_loading, **kwargs) 109 110 elif is_transformers_model and is_clip_model_in_single_file(class_obj, checkpoint): --> 111 loaded_sub_model = create_diffusers_clip_model_from_ldm( 112 class_obj, 113 checkpoint=checkpoint, /usr/local/lib/python3.10/dist-packages/diffusers/loaders/single_file_utils.py in create_diffusers_clip_model_from_ldm(cls, checkpoint, subfolder, config, torch_dtype, local_files_only, is_legacy_loading) 1380 1381 else: -> 1382 model.load_state_dict(diffusers_format_checkpoint) 1383 1384 if torch_dtype is not None: /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict, assign) 2187 2188 if len(error_msgs) > 0: -> 2189 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( 2190 self.__class__.__name__, "\n\t".join(error_msgs))) 2191 return _IncompatibleKeys(missing_keys, unexpected_keys) RuntimeError: Error(s) in loading state_dict for CLIPTextModel: Unexpected key(s) in state_dict: "text_model.embeddings.position_ids".

asomoza commented 5 months ago

oh yeah, you're right, in colab it gives an error but locally it's just a warning:

local:

Some weights of the model checkpoint were not used when initializing CLIPTextModel: 
 ['text_model.embeddings.position_ids']

colab:

RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
    Unexpected key(s) in state_dict: "text_model.embeddings.position_ids".

I have the same pytorch and cuda version as in colab, the only difference is the python version. We'll have to wait for @DN6 for a solution it seems.

asomoza commented 5 months ago

Interesting, I installed transformers from source !pip install git+https://github.com/huggingface/transformers.git in the colab and I got a different error:

SingleFileComponentError: Failed to load CLIPTextModel. Weights for this component appear to be missing in the checkpoint.
Please load the component before passing it in as an argument to `from_single_file`.

asomoza commented 5 months ago

in the meantime, since that repo has a version in the diffusers format, why not use it like this?

pipe = StableDiffusionPipeline.from_pretrained("gsdf/Counterfeit-V2.5").to("cuda")

suzukimain commented 5 months ago

The same issue was also discussed here.

suzukimain commented 5 months ago

However, I'm not used to making pull requests, so if I'm wrong somewhere, please point it out.

suzukimain commented 4 months ago

Fixed in #8462 and I'll close it. Thank you!

huggingface / diffusers