How to use the models from sd_control_collection hf repo in diffusers

[anilsathyan7]

How to load/convert the models at https://huggingface.co/lllyasviel/sd_control_collection/tree/main with diffusers?

>>> pipe = diffusers.StableDiffusionPipeline.from_single_file("diffusers_xl_canny_full.safetensors")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/loaders/single_file.py", line 261, in from_single_file
    pipe = download_from_original_stable_diffusion_ckpt(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1436, in download_from_original_stable_diffusion_ckpt
    converted_unet_checkpoint = convert_ldm_unet_checkpoint(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 426, in convert_ldm_unet_checkpoint
    new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]
KeyError: 'time_embed.0.weight'

Also not able to convert it via hf script: https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_controlnet_to_diffusers.py

How can it be used with diffusers?

[sdbds]

they are most sdxl controlnet, cant use https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_controlnet_to_diffusers.py