search

kijai / ComfyUI-CogVideoXWrapper

1.05k stars 63 forks source link

Image2Video Support (CogVideo recent update) #54

Closed phr00t closed 2 months ago

phr00t commented 2 months ago

Looks like CogVideo recently got Image2Video support, as seen in the description is this commit:

https://github.com/THUDM/CogVideo/commit/87ad61b531c55ba5c5034a6e70f2706ab2331010#diff-8e545c8323cdc76902096086a16383b013deb53b8057089ad5121b659c135062

Looking to get this integrated into the ComfyUI wrapper! I may be able to help with this if someone else doesn't get to it faster.

lishiyuwhu commented 2 months ago

https://github.com/THUDM/CogVideo/issues/270#issuecomment-2345100292 In this month.

phr00t commented 2 months ago

THUDM/CogVideo#270 (comment) In this month.

It appears like code has been added to accept images in the demo that I linked. However, in deeper inspection, perhaps this is just code for improving the prompt for upcoming image2video support. I may have jumped the gun!

JimWang151 commented 2 months ago

Looks like CogVideo recently got Image2Video support, as seen in the description is this commit:

THUDM/CogVideo@87ad61b#diff-8e545c8323cdc76902096086a16383b013deb53b8057089ad5121b659c135062

Looking to get this integrated into the ComfyUI wrapper! I may be able to help with this if someone else doesn't get to it faster.

Have you finished the update for image2Video?This is great funtion for video generation.

kijai commented 2 months ago

The image2video model has not been released yet, they said it would be sometime this month.

Lesteriax commented 2 months ago

does it have to do with this commit released 6 hours ago? it says i2v finetube https://github.com/THUDM/CogVideo/commit/01f19dad11d94d22caf3f1e47fc689e39ac34611

phr00t commented 2 months ago

does it have to do with this commit released 6 hours ago? it says i2v finetube THUDM/CogVideo@01f19da

It sure looks like it! But I have been known to get a little excited prematurely :D

EDIT: I believe we are now waiting for a model finetune release "THUDM/CogVideoX-5b-I2V" mentioned here: https://github.com/THUDM/CogVideo/commit/01f19dad11d94d22caf3f1e47fc689e39ac34611#diff-88763113c324bbae5a9297d9d097c018f02934a7a9e5a5ca998803ec0f4f6a09R8

phr00t commented 2 months ago

The image2video model has not been released yet, they said it would be sometime this month.

I found the model! It has been released!

https://cloud.tsinghua.edu.cn/d/5cc62a2d6e7d45c0a2f6/?p=%2F1&mode=list

Found here:

https://github.com/THUDM/CogVideo/commit/b410841bcf074144864d9565d894159348bf4a4a#diff-ee0228130a3c123b20735249d7cc6272f138bf9fd77823ed870fae7e95a79addR39

kijai commented 2 months ago

The image2video model has not been released yet, they said it would be sometime this month.

I found the model! It has been released!

https://cloud.tsinghua.edu.cn/d/5cc62a2d6e7d45c0a2f6/?p=%2F1&mode=list

Found here:

THUDM/CogVideo@b410841#diff-ee0228130a3c123b20735249d7cc6272f138bf9fd77823ed870fae7e95a79addR39

That file seems corrupt, tried downloading twice and it can't be loaded.

phr00t commented 2 months ago

The image2video model has not been released yet, they said it would be sometime this month.

I found the model! It has been released! https://cloud.tsinghua.edu.cn/d/5cc62a2d6e7d45c0a2f6/?p=%2F1&mode=list Found here: THUDM/CogVideo@b410841#diff-ee0228130a3c123b20735249d7cc6272f138bf9fd77823ed870fae7e95a79addR39

That file seems corrupt, tried downloading twice and it can't be loaded.

Ugh, I'm getting the same... sadge. I'll post an issue over there, I suppose.

kijai commented 2 months ago

Well they already have uploaded it to hugginface, the repo is just set private.

phr00t commented 2 months ago

Well they already have uploaded it to hugginface, the repo is just set private.

It's up!

https://huggingface.co/THUDM/CogVideoX-5b-I2V

kijai commented 2 months ago

Well they already have uploaded it to hugginface, the repo is just set private.

It's up!

https://huggingface.co/THUDM/CogVideoX-5b-I2V

It's working now with the nodes, but I couldn't get the fp8 to work with it... gonna see about that some other time.

phr00t commented 2 months ago

Well they already have uploaded it to hugginface, the repo is just set private.

It's up! https://huggingface.co/THUDM/CogVideoX-5b-I2V

It's working now with the nodes, but I couldn't get the fp8 to work with it... gonna see about that some other time.

Apparently we have Image2Video support with the CogVideo-Fun model, so perhaps this "issue" has already been addressed. I wonder how CogVideo-Fun compares to the "official" I2V model.

Looks like CogVideo-Fun supports a final image too, so I don't even think I'll be using the "official" I2V...

kijai commented 2 months ago

Well they already have uploaded it to hugginface, the repo is just set private.

It's up! https://huggingface.co/THUDM/CogVideoX-5b-I2V

It's working now with the nodes, but I couldn't get the fp8 to work with it... gonna see about that some other time.

Apparently we have Image2Video support with the CogVideo-Fun model, so perhaps this "issue" has already been addressed. I wonder how CogVideo-Fun compares to the "official" I2V model.

Looks like CogVideo-Fun supports a final image too, so I don't even think I'll be using the "official" I2V...

Official one looks to be better quality, but it's still resolution locked to the same default as CogVideoX, while the "Fun" version isn't. Both are running with these nodes now to some extend, and the "Fun" version works with fp8, including fast mode, and the Comfy T5 so I'd say it's better than the official wrapper.

Ratinod commented 2 months ago

@kijai Could you please add resolutions from 512 to 256 (in 64 increments) in "CogVideoXFun Sampler"? On 8GB VRAM at 512x512 resolution the limit is 21 frames. 25 frames and more give OOM. But with 320x320 with 8GB VRAM you can generate 49 frames without OOM. Yeah... I know... I know... 8GB videocard... Eh...

There is no point in even trying the official I2V. 720x480 will be too much for the 8GB VRAM.

kijai commented 2 months ago

@kijai Could you please add resolutions from 512 to 256 (in 64 increments) in "CogVideoXFun Sampler"? On 8GB VRAM at 512x512 resolution the limit is 21 frames. 25 frames and more give OOM. But with 320x320 with 8GB VRAM you can generate 49 frames without OOM. Yeah... I know... I know... 8GB videocard... Eh...

There is no point in even trying the official I2V. 720x480 will be too much for the 8GB VRAM.

Sure, done.

KrakeyMTL commented 2 months ago

can anyone attach a workflow for this i2v new model? I just can't seem to get mine to work thanks!

Ratinod commented 2 months ago

can anyone attach a workflow for this i2v new model? I just can't seem to get mine to work thanks!

my 8gb vram CogVideoXFun workflow: workflow_cogvideoxfun_8gbvr.json

Just use "Load CLIP" instead of "CLIPLoader (GGUF)" if you did not use https://github.com/city96/ComfyUI-GGUF

image:

workflow_8gb

KrakeyMTL commented 2 months ago

You are a superstar @Ratinod !!! Thank you so much.

phr00t commented 2 months ago

can anyone attach a workflow for this i2v new model? I just can't seem to get mine to work thanks!

Remember that there are included workflows in the repo:

https://github.com/kijai/ComfyUI-CogVideoXWrapper/tree/main/examples

KrakeyMTL commented 2 months ago

can anyone attach a workflow for this i2v new model? I just can't seem to get mine to work thanks!

Remember that there are included workflows in the repo:

https://github.com/kijai/ComfyUI-CogVideoXWrapper/tree/main/examples

I am legit embarrassed that I missed those on the refresh. wow. thank you all so much. I think i need some sleep! lol

phr00t commented 2 months ago

Closing this since this is DONE! Well played everyone.