[Feature Request]: Add support for merging depth models with fine tuned models.

[simonhavelock]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Presumably similar to https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/3788 Currently, when using the Checkpoint Merger and loading in the 512-depth-ema.ckpt as base model and any other model you get an error: Error merging checkpoints: Bad dimensions for merged layer model.diffusion_model.input_blocks.0.0.weight: A=torch.Size([320, 5, 3, 3]), B=torch.Size([320, 4, 3, 3])

I assume the extra dimension (5 layers) for the depth model is the depth data itself - but even bypassing it when merging with another model should still give results better than without it. My case here is specific to the use of img2img in an attempt to achieve some temporal cohesion. Of course it would be no issue also merging a model with depth maps attached, but how to do that goes beyond me.

Proposed workflow

Modifying the code so that in case of tensor mismatch it asks if you're okay with it leaving the values empty or doing a median transfer of values to fill up the missing ones based on "average".

Additional information

No response

[ddday]

I've run into a similar error, Error merging checkpoints: Bad dimensions for merged layer model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn2.to_k.weight: A=torch.Size([320, 1024]), B=torch.Size([320, 768]) +1

[toyxyz]

I merged Waifu diffusion 1.4 and Depth model using this script. https://zenn.dev/discus0434/articles/ef418a8b0b3dc0

[satwiksunnam19]

I've run into a similar error, Error merging checkpoints: Bad dimensions for merged layer model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn2.to_k.weight: A=torch.Size([320, 1024]), B=torch.Size([320, 768]) +1, do anyone have an idea how to solve this @ddday have you solved this error?

[ddday]

I've run into a similar error, Error merging checkpoints: Bad dimensions for merged layer model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn2.to_k.weight: A=torch.Size([320, 1024]), B=torch.Size([320, 768]) +1, do anyone have an idea how to solve this @ddday have you solved this error?

Sorry I have not