search

invoke-ai / InvokeAI

InvokeAI is a leading creative engine for Stable Diffusion models, empowering professionals, artists, and enthusiasts to generate and create visual media using the latest AI-driven technologies. The solution offers an industry leading WebUI, supports terminal use through a CLI, and serves as the foundation for multiple commercial products.
https://invoke-ai.github.io/InvokeAI/
Apache License 2.0
22.87k stars 2.37k forks source link

[bug]: Certain SDXL controlnets fail conversion from safetensors with `KeyError: 'time_embed.0.weight'` #6479

Open psychedelicious opened 3 months ago

psychedelicious commented 3 months ago

Is there an existing issue for this problem?

Operating system

Linux

GPU vendor

Nvidia (CUDA)

GPU model

No response

GPU VRAM

No response

Version number

32fa4a8d4509d8238e751c96b018d8b8f5ea74f3

Browser

FF

Python dependencies

No response

What happened

When downloading the safetensors version of certain controlnet models, installation is successful but conversion fails during generation.

Example model: https://huggingface.co/thibaud/controlnet-openpose-sdxl-1.0/blob/main/OpenPoseXL2.safetensors Error:

[2024-06-03 18:15:52,525]::[InvokeAI]::ERROR --> Error while invoking session 981b63b0-3873-4d8e-9f7d-1ff8f1a3f6fe, invocation 8fbe984f-607b-41e9-950f-eef1b900cac8 (denoise_latents): 'time_embed.0.weight'
[2024-06-03 18:15:52,525]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/home/bat/Documents/Code/InvokeAI/invokeai/app/services/session_processor/session_processor_default.py", line 129, in run_node
    output = invocation.invoke_internal(context=context, services=self._services)
  File "/home/bat/Documents/Code/InvokeAI/invokeai/app/invocations/baseinvocation.py", line 289, in invoke_internal
    output = self.invoke(context)
  File "/home/bat/Documents/Code/InvokeAI/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/bat/Documents/Code/InvokeAI/invokeai/app/invocations/latent.py", line 962, in invoke
    controlnet_data = self.prep_control_data(
  File "/home/bat/Documents/Code/InvokeAI/invokeai/app/invocations/latent.py", line 637, in prep_control_data
    control_model = exit_stack.enter_context(context.models.load(control_info.control_model))
  File "/home/bat/Documents/Code/InvokeAI/invokeai/app/services/shared/invocation_context.py", line 359, in load
    return self._services.model_manager.load.load_model(model, _submodel_type)
  File "/home/bat/Documents/Code/InvokeAI/invokeai/app/services/model_load/model_load_default.py", line 72, in load_model
    ).load_model(model_config, submodel_type)
  File "/home/bat/Documents/Code/InvokeAI/invokeai/backend/model_manager/load/load_default.py", line 62, in load_model
    locker = self._convert_and_load(model_config, model_path, submodel_type)
  File "/home/bat/Documents/Code/InvokeAI/invokeai/backend/model_manager/load/load_default.py", line 89, in _convert_and_load
    loaded_model = self._do_convert(config, model_path, cache_path, submodel_type)
  File "/home/bat/Documents/Code/InvokeAI/invokeai/backend/model_manager/load/load_default.py", line 121, in _do_convert
    pipeline = self._convert_model(config, model_path, cache_path if self.convert_cache.max_size > 0 else None)
  File "/home/bat/Documents/Code/InvokeAI/invokeai/backend/model_manager/load/model_loaders/controlnet.py", line 50, in _convert_model
    result = convert_controlnet_to_diffusers(
  File "/home/bat/Documents/Code/InvokeAI/invokeai/backend/model_manager/convert_ckpt_to_diffusers.py", line 77, in convert_controlnet_to_diffusers
    pipe = download_controlnet_from_original_ckpt(checkpoint_path.as_posix(), **kwargs)
  File "/home/bat/Documents/Code/InvokeAI/.venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1849, in download_controlnet_from_original_ckpt
    controlnet = convert_controlnet_checkpoint(
  File "/home/bat/Documents/Code/InvokeAI/.venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1120, in convert_controlnet_checkpoint
    converted_ctrl_checkpoint = convert_ldm_unet_checkpoint(
  File "/home/bat/Documents/Code/InvokeAI/.venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 431, in convert_ldm_unet_checkpoint
    new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]
KeyError: 'time_embed.0.weight'

If you install via repo ID, the diffusers version is downloaded and works fine, so it's a conversion issue.

What you expected to happen

Both safetensors and diffusers version of the cnet model work.

How to reproduce the problem

No response

Additional context

Appears to be the same issue as in #4050.

Discord username

No response

psychedelicious commented 3 months ago

@freshlesh3 I ran into the issue from #4050 and am creating this bug report to track it. In the meantime you can install the working diffusers version: