The paper proposes a framework to animate the existing personalized text-to-image diffusion models. i.e. text to short animation clip(<2s) like Video Diffusion Model[^1]. To do this, the paper inserts a Motion Modeling Module to the frozen text-to-image model and trains on the video clips. Once the model being trained, the same Motion Modeling Module can be applied to other personalized models.
Method
train on WebVid-10[^2], text-video pair dataset
The training target is similar to latent diffusion but with 5D tensor(batch x channels x frames x height x width) as input, so the noise of each batch became 4D(channels x frames x height x width). The self-attention module in Motion Module provides the temporal and spatial relation in every batch.
Highlight
the experiment shows model trained on 256x256 can be generalized well to higher resolution
no need fine-tuning
can be combined with LoRA or DreamBooth
Limitation
some personalized model might fail due to the data distribution gap between the training video and personalize model
Introduction
The paper proposes a framework to animate the existing personalized text-to-image diffusion models. i.e. text to short animation clip(<2s) like Video Diffusion Model[^1]. To do this, the paper inserts a Motion Modeling Module to the frozen text-to-image model and trains on the video clips. Once the model being trained, the same Motion Modeling Module can be applied to other personalized models.
Method
Highlight
Limitation
Comments
[^1]: Video Diffusion Models [^2]: