search

kijai / ComfyUI-CogVideoXWrapper

997 stars 61 forks source link

CogVideoX + FreeNoise #102

Open a-r-r-o-w opened 1 month ago

a-r-r-o-w commented 1 month ago

Hey there! :wave: I'm Aryan from the Diffusers team

Great work here and I love using this for experimentation!

I was looking into ways that allowed for longer video generation, and prompt interpolation to create smooth transition videos. To make this possible, I tried using FreeNoise and adding support in Diffusers here. My results for multi-prompt settings are poor for txt2vid, and even worse when trying img2vid. In the case of a single prompt, I've been able to successfully generate long and consistent videos upto 30 seconds (using FreeNoise) without abrupt context changes as we see in the temporal tiling demo (which is still great work btw!)

Due to limited time for experimentation and my involvement in other projects, I'm unable to dedicate more time on this for now, but would be happy to brainstorm and see if prompt/image interpolation could be achieved/has already been made possible by someone else. Let me know if you have ideas! Thanks for your time @kijai.

cc @tin2tin

kijai commented 1 month ago

This would be interesting for the new CogVideoX-Fun pose models (which btw seems to work with all kinds of control inputs, not just pose), don't exactly have a lot of time at my hands either but I'll look into it at some point, thank you!