Promax Inapinting issue

[rudransh2004]

I'm facing the following error while running the code in Promax directory controlnet_union_test_inpainting.py

Traceback (most recent call last): File "/home/anuvadiniDev/workspace/ControlNetPlus/promax/controlnet_union_test_inpainting.py", line 76, in images = pipe(prompt=prompt, File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "/home/anuvadiniDev/workspace/ControlNetPlus/promax/../pipeline/pipeline_controlnet_union_inpaint_sd_xl.py", line 1708, in call down_block_res_samples, mid_block_res_sample = self.controlnet( File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/anuvadiniDev/workspace/ControlNetPlus/promax/../models/controlnet_union.py", line 846, in forward control_emb = self.control_add_embedding(control_embeds) File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/diffusers/models/embeddings.py", line 227, in forward sample = self.linear_1(sample) File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x2048 and 1536x1280)

[xinsir6]

If you want to solve it immediately, please change the num_control_type to 8, 8 for promax and 6 for base, this will be help and this is the your bug report reason. I will change it soon when I have completed my work today.

[rudransh2004]

@xinsir6

I tried to changing it to 8 as per your instructions but now I face another error

Traceback (most recent call last): File "/home/anuvadiniDev/workspace/ControlNetPlus/promax/controlnet_union_test_inpainting.py", line 23, in controlnet_model = ControlNetModel_Union.from_pretrained("xinsir/controlnet-union-sdxl-1.0", torch_dtype=torch.float16, use_safetensors=True) File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn return fn(*args, **kwargs) File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/diffusers/models/modeling_utils.py", line 741, in from_pretrained model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_pretrained_model( File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/diffusers/models/modeling_utils.py", line 832, in _load_pretrained_model raise RuntimeError(f"Error(s) in loading state_dict for {model.class.name}:\n\t{error_msg}") RuntimeError: Error(s) in loading state_dict for ControlNetModel_Union: size mismatch for task_embedding: copying a param with shape torch.Size([6, 320]) from checkpoint, the shape in current model is torch.Size([8, 320]). You may consider adding ignore_mismatched_sizes=True in the model from_pretrained method.

Please help me to resolve it

[fmigneault]

@xinsir6 I get the same RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x2048 and 1536x1280) for the outpainting case using the definition as provided in the example: https://github.com/xinsir6/ControlNetPlus/blob/9acd5ca2878ad3f2ad2576515988c4d389dbcb05/promax/controlnet_union_test_outpainting.py

Changing the value to 8 as suggested also resulted in:

ValueError: Cannot load xinsir/controlnet-union-sdxl-1.0 because task_embedding expected shape tensor(..., device='meta', size=(8, 320)), but got torch.Size([6, 320]). If you want to instead overwrite randomly initialized weights, please make sure to pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example.

[xinsir6]

I update the scripts to fix it, please try it again, sorry to bring your inconvenience.

[fmigneault]

@xinsir6 I had a look at the updated pipeline. The model reference was modified to "./controlnet-union-sdxl-1.0-promax" which cannot be resolved. I tried with "xinsir/controlnet-union-sdxl-1.0-promax" as well, but it is not present in your uploads https://huggingface.co/xinsir Is this model going to be published soon?

[fmigneault]

In the meantime, I'm using brad-twinkl/controlnet-union-sdxl-1.0-promax as alternative. The pipeline seems to be running correctly with it and latest code updates. Waiting on the pipeline to finish processing to inspect the result...

[fmigneault]

@xinsir6 I have tried the script with this sample image, and saved the mask that is now generated instead of being provided as input. As shown below, I get completely different masks each time I run the script, and none seem to match the areas of the original image that I want to outpaint. Also, as a result, the obtained image from running the pipeline is identical to the input image.

Can you verify that the script behaves as intended, and guide to fix it?

[muzammildafedar]

@xinsir6 I get the same RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x2048 and 1536x1280) for the outpainting case using the definition as provided in the example: https://github.com/xinsir6/ControlNetPlus/blob/9acd5ca2878ad3f2ad2576515988c4d389dbcb05/promax/controlnet_union_test_outpainting.py

Changing the value to 8 as suggested also resulted in:

ValueError: Cannot load xinsir/controlnet-union-sdxl-1.0 because task_embedding expected shape tensor(..., device='meta', size=(8, 320)), but got torch.Size([6, 320]). If you want to instead overwrite randomly initialized weights, please make sure to pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example.

Use this but consume too much memrory

controlnet_model = ControlNetModel_Union.from_pretrained("./controlnet-union-sdxl-1.0-promax", torch_dtype=torch.float16, use_safetensors=True, low_cpu_mem_usage=False,ignore_mismatched_sizes=True)