Open rudransh2004 opened 1 month ago
If you want to solve it immediately, please change the num_control_type to 8, 8 for promax and 6 for base, this will be help and this is the your bug report reason. I will change it soon when I have completed my work today.
@xinsir6
I tried to changing it to 8 as per your instructions but now I face another error
Traceback (most recent call last):
File "/home/anuvadiniDev/workspace/ControlNetPlus/promax/controlnet_union_test_inpainting.py", line 23, in ignore_mismatched_sizes=True
in the model from_pretrained
method.
Please help me to resolve it
@xinsir6
I get the same RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x2048 and 1536x1280)
for the outpainting case using the definition as provided in the example: https://github.com/xinsir6/ControlNetPlus/blob/9acd5ca2878ad3f2ad2576515988c4d389dbcb05/promax/controlnet_union_test_outpainting.py
Changing the value to 8
as suggested also resulted in:
ValueError: Cannot load xinsir/controlnet-union-sdxl-1.0 because task_embedding expected shape tensor(..., device='meta', size=(8, 320)), but got torch.Size([6, 320]). If you want to instead overwrite randomly initialized weights, please make sure to pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example.
I update the scripts to fix it, please try it again, sorry to bring your inconvenience.
@xinsir6
I had a look at the updated pipeline.
The model reference was modified to "./controlnet-union-sdxl-1.0-promax"
which cannot be resolved.
I tried with "xinsir/controlnet-union-sdxl-1.0-promax"
as well, but it is not present in your uploads https://huggingface.co/xinsir
Is this model going to be published soon?
In the meantime, I'm using brad-twinkl/controlnet-union-sdxl-1.0-promax
as alternative.
The pipeline seems to be running correctly with it and latest code updates.
Waiting on the pipeline to finish processing to inspect the result...
@xinsir6
I have tried the script with this sample image, and saved the mask
that is now generated instead of being provided as input. As shown below, I get completely different masks each time I run the script, and none seem to match the areas of the original image that I want to outpaint. Also, as a result, the obtained image from running the pipeline is identical to the input image.
Can you verify that the script behaves as intended, and guide to fix it?
I'm facing the following error while running the code in Promax directory controlnet_union_test_inpainting.py
Traceback (most recent call last): File "/home/anuvadiniDev/workspace/ControlNetPlus/promax/controlnet_union_test_inpainting.py", line 76, in
images = pipe(prompt=prompt,
File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, kwargs)
File "/home/anuvadiniDev/workspace/ControlNetPlus/promax/../pipeline/pipeline_controlnet_union_inpaint_sd_xl.py", line 1708, in call
down_block_res_samples, mid_block_res_sample = self.controlnet(
File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, *kwargs)
File "/home/anuvadiniDev/workspace/ControlNetPlus/promax/../models/controlnet_union.py", line 846, in forward
control_emb = self.control_add_embedding(control_embeds)
File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(args, kwargs)
File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/diffusers/models/embeddings.py", line 227, in forward
sample = self.linear_1(sample)
File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/anuvadiniDev/workspace/ControlNetPlus/controlnetplus/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x2048 and 1536x1280)