search

dvlab-research / ControlNeXt

Controllable video and image Generation, SVD, Animate Anyone, ControlNet, ControlNeXt, LoRA
Apache License 2.0
1.39k stars 66 forks source link

Can we create videos using SD1.5 or SDXL instead of ControlNeXt-SVD-v2 #46

Closed nitinmukesh closed 1 month ago

nitinmukesh commented 2 months ago

I have low VRAM system and always get out of memory while using ControlNeXt-SVD-v2. As I understand the models are huge in size.

Can we create video the same way using SD1.5 or SDXL? I see only image creation code using DWPose in ControlNeXt-SD1.5 and ControlNeXt-SDXL.

[Generate the video controlled by the sequence of human poses using ControlNeXt-SD1.5 or ControlNeXt-SDXL]?

Pbihao commented 1 month ago

Hello, and thank you for your interest!

This might not be the best solution since models like SD1.5 or SDXL are designed primarily for image generation and lack essential modules for handling temporal information.

That said, you're welcome to give it a try!