I2V Error when changing input image after running at least once: "Sizes of tensors must match except in dimension 1. Expected size 65 but got size 60 for tensor number 1 in the list".

genericgod commented 2 months ago

I get this error when I exchange the input image after a single or more generations with a different image. I used an image resize node so the input image should be the exact same size. Model: CogVideoX-5b-I2V Here's the detailed comfy error report:

ComfyUI Error Report

Error Details

Node Type: CogVideoSampler
Exception Type: RuntimeError

Exception Message: Sizes of tensors must match except in dimension 1. Expected size 65 but got size 60 for tensor number 1 in the list.

Stack Trace


File "D:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "D:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 198, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "D:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)

File "D:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "D:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CogVideoXWrapper\nodes.py", line 421, in process
latents = pipeline["pipe"](
          ^^^^^^^^^^^^^^^^^

File "D:\AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^

File "D:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CogVideoXWrapper\pipeline_cogvideox.py", line 458, in __call__
image_cond_latents = torch.cat([image_cond_latents, latent_padding], dim=1)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

### System Information
- **ComfyUI Version:** v0.2.2-50-g0bfc7cc
- **Arguments:** ComfyUI\main.py --listen --preview-method auto
- **OS:** nt
- **Python Version:** 3.11.8 (tags/v3.11.8:db85d51, Feb  6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)]
- **Embedded Python:** true
- **PyTorch Version:** 2.3.1+cu121
### Devices

- **Name:** cuda:0 NVIDIA GeForce RTX 3060 : cudaMallocAsync
  - **Type:** cuda
  - **VRAM Total:** 12884377600
  - **VRAM Free:** 5946926384
  - **Torch VRAM Total:** 5737807872
  - **Torch VRAM Free:** 11371824

### Logs

2024-09-19 15:53:01,239 - root - INFO - Total VRAM 12288 MB, total RAM 32689 MB 2024-09-19 15:53:01,240 - root - INFO - pytorch version: 2.3.1+cu121 2024-09-19 15:53:07,111 - root - INFO - xformers version: 0.0.26.post1 2024-09-19 15:53:07,111 - root - INFO - Set vram state to: NORMAL_VRAM 2024-09-19 15:53:07,111 - root - INFO - Device: cuda:0 NVIDIA GeForce RTX 3060 : cudaMallocAsync 2024-09-19 15:53:08,829 - root - INFO - Using xformers cross attention 2024-09-19 15:53:14,565 - root - INFO - [Prompt Server] web root: D:\AI\ComfyUI_windows_portable\ComfyUI\web

### Attached Workflow

Gyramuur commented 2 months ago

Same here unfortunately. Even after changing no settings and successfully genning once, the second gen throws the mentioned error.

kijai commented 2 months ago

I can't replicate this myself, I've tested lot of different images in different aspects and never gotten an error. Are you certain the output of the image resize node is exactly the same size?

henrique-galimberti commented 2 months ago

I get the same error everytime, after first generation.

First time I generate with Image A and it works.
I change to Image B and it fails with Expected size 65 but got size 60 for tensor number 1 in the list
I change back to Image A and still get Expected size 65 but got size 60 for tensor number 1 in the list

This only happens with CogVideoX5B I2V workflow. With T2V or Fun workflows I dont get the error.

henrique-galimberti commented 2 months ago

@kijai not sure if this helps:

nieli123456 commented 2 months ago

同样的报错

Gyramuur commented 2 months ago

It should be noted that for me, the only setting I changed was setting vae tiling to true, but without doing that I get OOM on my 24 GB of VRAM, so currently it's essential for me to use that.

kijai commented 2 months ago

It should be noted that for me, the only setting I changed was setting vae tiling to true, but without doing that I get OOM on my 24 GB of VRAM, so currently it's essential for me to use that.

Thanks, that helped me find the issue: The VAE tiling stayed enabled for the VAE as a whole once used once, making the encoding (which uses same VAE) run tiled too and cause this issue. Should be fixed now.

genericgod commented 2 months ago

Thanks, that helped me find the issue: The VAE tiling stayed enabled for the VAE as a whole once used once, making the encoding (which uses same VAE) run tiled too and cause this issue. Should be fixed now.

@kijai Looks like that fixed it. Thank you. Great work!

4lt3r3go commented 1 month ago

Despite a thousand attempts, I managed to get it to work half of a once.. not even once, more like half a time, because it finally started rendering, but at some point, it stopped and gave me the same error again, as always:

RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 60 but got size 96 for tensor number 1 in the list

i have no idea how you solved guys

kijai commented 1 month ago

Despite a thousand attempts, I managed to get it to work half of a once.. not even once, more like half a time, because it finally started rendering, but at some point, it stopped and gave me the same error again, as always:

RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 60 but got size 96 for tensor number 1 in the list

i have no idea how you solved guys

Are you saying the example workflow does not run for you as it is?

4lt3r3go commented 1 month ago

tested everything again, including all json in the example folder on this github. here errors i get in this cogvideox_2b_controlnet_example_01.json file:

error: The size of tensor a (3072) must match the size of tensor b (1920) at non-singleton dimension 2

there's also a workflow on civitAI link wich gave me: Sizes of tensors must match except in dimension 2. Expected size 60 but got size 96 for tensor number 1 in the list

i know this is not related to this github page but is really hard to find anything controlnet/lora related that is working for me. there's litterally no one sharing workflows about this right now

i think this 2 workflow have something in common, since both using controlnet but don't quote me on that 😁

kijai commented 1 month ago

tested everything again, including all json in the example folder on this github. here errors i get in this cogvideox_2b_controlnet_example_01.json file:

error: The size of tensor a (3072) must match the size of tensor b (1920) at non-singleton dimension 2

there's also a workflow on civitAI link wich gave me: Sizes of tensors must match except in dimension 2. Expected size 60 but got size 96 for tensor number 1 in the list

i know this is not related to this github page but is really hard to find anything controlnet/lora related that is working for me. there's litterally no one sharing workflows about this right now

i think this 2 workflow have something in common, since both using controlnet but don't quote me on that 😁

That's probably because there are so many different models, and they aren't compatible with each other. LoRa trained for 2b won't work on 5b and vice versa. And they won't work between the Fun -versions and original model either.

Currently there's only controlnet for the original 2b text2vid model, it won't work with any other. What we can mix and match is very limited.

4lt3r3go commented 1 month ago

tested everything again, including all json in the example folder on this github. here errors i get in this cogvideox_2b_controlnet_example_01.json file: error: The size of tensor a (3072) must match the size of tensor b (1920) at non-singleton dimension 2 there's also a workflow on civitAI link wich gave me: Sizes of tensors must match except in dimension 2. Expected size 60 but got size 96 for tensor number 1 in the list i know this is not related to this github page but is really hard to find anything controlnet/lora related that is working for me. there's litterally no one sharing workflows about this right now i think this 2 workflow have something in common, since both using controlnet but don't quote me on that 😁

That's probably because there are so many different models, and they aren't compatible with each other. LoRa trained for 2b won't work on 5b and vice versa. And they won't work between the Fun -versions and original model either.

Currently there's only controlnet for the original 2b text2vid model, it won't work with any other. What we can mix and match is very limited.

You solved my problem just by mentioning the compatibility issue between 2b and 5b, as well as between fun and original. i'm gonna download 2B (wich i wasnt considering till now.. always thought 5b was better) and see if works.

I would also like to know wich files all this cog have in common. do i really need to download everything or i can use files from other cog models folder? like the 2 text encoders files and the vae both looks kinda similar to other cogs models in those folders

kijai commented 1 month ago

tested everything again, including all json in the example folder on this github. here errors i get in this cogvideox_2b_controlnet_example_01.json file: error: The size of tensor a (3072) must match the size of tensor b (1920) at non-singleton dimension 2 there's also a workflow on civitAI link wich gave me: Sizes of tensors must match except in dimension 2. Expected size 60 but got size 96 for tensor number 1 in the list i know this is not related to this github page but is really hard to find anything controlnet/lora related that is working for me. there's litterally no one sharing workflows about this right now i think this 2 workflow have something in common, since both using controlnet but don't quote me on that 😁

That's probably because there are so many different models, and they aren't compatible with each other. LoRa trained for 2b won't work on 5b and vice versa. And they won't work between the Fun -versions and original model either. Currently there's only controlnet for the original 2b text2vid model, it won't work with any other. What we can mix and match is very limited.

You solved my problem just by mentioning the compatibility issue between 2b and 5b, as well as between fun and original. i'm gonna download 2B (wich i wasnt considering till now.. always thought 5b was better) and see if works.

I would also like to know wich files all this cog have in common. do i really need to download everything or i can use files from other cog models folder? like the 2 text encoders files and the vae both looks kinda similar to other cogs models in those folders

You don't need any of the text encoders or tokenizers as we are using the Comfy T5 encoding, those also are not downloaded by the autodownload node.

4lt3r3go commented 1 month ago

thank you for your precious time, you saved me extra hdd space.

last question 🙄

is it actually possible to do I2V + controlnet driven by a video? or is just one or the other?

the workflow i see shared on Civit has both options but is one or the other

kijai commented 1 month ago

thank you for your precious time, you saved me extra hdd space.

last question 🙄

is it actually possible to do I2V + controlnet driven by a video? or is just one or the other?

the workflow i see shared on Civit has both options but is one or the other

Currently it's not possible, as the only actual controlnets are trained for the 2b text2video model.

The Fun -versions "pose" model also allows control input, but it's a whole model and can't currently be used with image conditioning.

Both of these do allow images/video encoded as latents as additional input, but that's more like video2video rather than image2video.

4lt3r3go commented 1 month ago

thank you for your precious time, you saved me extra hdd space. last question 🙄 is it actually possible to do I2V + controlnet driven by a video? or is just one or the other? the workflow i see shared on Civit has both options but is one or the other

Currently it's not possible, as the only actual controlnets are trained for the 2b text2video model.

The Fun -versions "pose" model also allows control input, but it's a whole model and can't currently be used with image conditioning.

Both of these do allow images/video encoded as latents as additional input, but that's more like video2video rather than image2video.

the whole thing is clear now at 5th day i'm playing with all this cog models. are you referring to this? why this are connected like this and what all this supposed to do (btw this workflow is in the example folder on this github)

or maybe this one?

kijai commented 1 month ago

thank you for your precious time, you saved me extra hdd space. last question 🙄 is it actually possible to do I2V + controlnet driven by a video? or is just one or the other? the workflow i see shared on Civit has both options but is one or the other

Currently it's not possible, as the only actual controlnets are trained for the 2b text2video model. The Fun -versions "pose" model also allows control input, but it's a whole model and can't currently be used with image conditioning. Both of these do allow images/video encoded as latents as additional input, but that's more like video2video rather than image2video.

the whole thing is clear now at 5th day i'm playing with all this cog models. are you referring to this? why this are connected like this and what all this supposed to do (btw this workflow is in the example folder on this github)

or maybe this one?

That looks like a mismatch between the version of the nodes and the workflow...

Mario-Forero commented 3 weeks ago

The issue seems to be related to the order of width and height in the node.

Usually other nodes use width first and then height while "CogVideo Sampler" has height first.

kijai / ComfyUI-CogVideoXWrapper