search

xinsir6 / ControlNetPlus

ControlNet++: All-in-one ControlNet for image generations and editing!
Apache License 2.0
1.77k stars 43 forks source link

A problem encountered when loading the promax model #26

Open mm65zzz opened 4 months ago

mm65zzz commented 4 months ago

First of all, thank you for your outstanding contribution! However, I encountered a problem when using the promax model. There is a mismatch in the expected shape of the tensor weights when trying to load a pretrained model using the ControlNetModel_Union class from the diffusion_pytorch_model_promax.safetensors. Specifically, the control_add_embedding.linear_1.weight tensor has an expected shape of (1280, 1536) but the actual shape is (1280, 2048).

This might be because control_add_embedding was not initialized using the parameter num_control_type and is still using "6".

https://github.com/xinsir6/ControlNetPlus/blob/ba6c35b62e9df4c8f3b6429c4844ecc92685c8ec/models/controlnet_union.py#L409C1-L409C100

When I changed the code to self.control_add_embedding = TimestepEmbedding(addition_time_embed_dim * num_control_type, time_embed_dim) it loaded correctly.

xinsir6 commented 4 months ago

I update the scripts to fix it, please try it again, follow the instructions.

janrvdolf commented 4 months ago

Hi @xinsir6 and @mm65zzz,

I am still having similar / same problem in promax/controlnet_union_test_tile_deblur.py after this change:

File "/home/XXX/1337/ControlNetPlus/promax/../models/controlnet_union.py", line 846, in forward
  control_emb = self.control_add_embedding(control_embeds)
File "/home/XXX/miniconda3/envs/controlplus/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  return forward_call(*args, **kwargs)
File "/home/XXX/miniconda3/envs/controlplus/lib/python3.8/site-packages/diffusers/models/embeddings.py", line 227, in forward
  sample = self.linear_1(sample)
File "/home/XXX/miniconda3/envs/controlplus/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  return forward_call(*args, **kwargs)
File "/home/XXX/miniconda3/envs/controlplus/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
  return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x2048 and 1536x1280)

Do you know what could be an issue?