search

jianzongwu / MotionBooth

[NeurIPS 2024 Spotlight] The official implement of research paper "MotionBooth: Motion-Aware Customized Text-to-Video Generation"
111 stars 6 forks source link

can this be extended to image to video? #4

Closed SkylerZheng closed 1 month ago

SkylerZheng commented 4 months ago

wondering if this can be extended to image to video generation with image condition, and controllable camera motion?

jianzongwu commented 4 months ago

Our camera motion module is highly scalable. For vanilla text-to-video generation, you can directly use it and control the camera motion.

Since Stable Diffusion Pipeline does not provide image-to-video itself, you may need to find some image-to-video methods, like Stable Video Diffusion, and apply our camera motion method.

Please let me know if you success!