search

showlab / MotionDirector

[ECCV 2024 Oral] MotionDirector: Motion Customization of Text-to-Video Diffusion Models.
https://showlab.github.io/MotionDirector/
Apache License 2.0
848 stars 53 forks source link

Use an image as the initial status to generate videos #41

Open sunnyHelen opened 2 months ago

sunnyHelen commented 2 months ago

Hi,

Thank you very much for sharing your great work and the code. Is it possible to take an initial image to animate it rather than randomly generate videos from the text prompt? Could you provide some suggestions on how to do this?

ruizhaocv commented 2 months ago

Hi. I think there are two ways to achieve your goal. The first one is that train spatial LoRAs on the initial image, combining with the temporal LoRAs learned from reference videos. The second one is that you can try to use the inversion noise from the initial image to denoise and generate videos.

xiueran commented 3 hours ago

Hi,

Thank you very much for sharing your great work and the code. I encountered some problems when using MotionDirector for Image Animation. I trained temporal LoRAs and spatial LoRAs according to the instructions. However, the noise_prior parameter in the inference is 0.1, and the result is still similar to my reference video. Not the motion of the initial image.