DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
[](https://openxlab.org.cn/apps/detail/JinboXING/DynamiCrafter)
_**[Jinbo Xing](https://doubiiu.github.io/), [Menghan Xia](https://menghanxia.github.io), [Yong Zhang](https://yzhang2016.github.io), [Haoxin Chen](), [Wangbo Yu](), [Hanyuan Liu](https://github.com/hyliu), [Gongye Liu](), [Xintao Wang](https://xinntao.github.io/), [Ying Shan](https://scholar.google.com/citations?hl=en&user=4oXBp9UAAAAJ&view_op=list_works&sortby=pubdate), [Tien-Tsin Wong](https://ttwong12.github.io/myself.html)**_
From CUHK and Tencent AI Lab. at European Conference on Computer Vision (ECCV) 2024, Oral
๐ Introduction
๐ฅ๐ฅ Training / Fine-tuning code is available NOW!!!
๐ฅ We 1024x576 version ranks 1st on the I2V benchmark list from VBench!
๐ฅ Generative frame interpolation / looping video generation model weights (320x512) have been released!
๐ฅ New Update Rolls Out for DynamiCrafter! Better Dynamic, Higher Resolution, and Stronger Coherence!
๐ค DynamiCrafter can animate open-domain still images based on text prompt by leveraging the pre-trained video diffusion priors. Please check our project page and paper for more information.
๐ Seeking comparisons with Stable Video Diffusion and PikaLabs? Click the image below.
1.1. Showcases (576x1024)
|
|
|
|
1.2. Showcases (320x512)
|
|
|
|
1.3. Showcases (256x256)
| "bear playing guitar happily, snowing" | "boy walking on the street" | ||
|
|
|
|
 |
| Input starting frame | Input ending frame | Generated video |
|
|
|
|
|
|
|
|
|
|
|
|
## ๐งฐ Models |Model|Resolution|GPU Mem. & Inference Time (A100, ddim 50steps)|Checkpoint| |:---------|:---------|:--------|:--------| |DynamiCrafter1024|576x1024|18.3GB & 75s (`perframe_ae=True`)|[Hugging Face](https://huggingface.co/Doubiiu/DynamiCrafter_1024/blob/main/model.ckpt)| |DynamiCrafter512|320x512|12.8GB & 20s (`perframe_ae=True`)|[Hugging Face](https://huggingface.co/Doubiiu/DynamiCrafter_512/blob/main/model.ckpt)| |DynamiCrafter256|256x256|11.9GB & 10s (`perframe_ae=False`)|[Hugging Face](https://huggingface.co/Doubiiu/DynamiCrafter/blob/main/model.ckpt)| |DynamiCrafter512_interp|320x512|12.8GB & 20s (`perframe_ae=True`)|[Hugging Face](https://huggingface.co/Doubiiu/DynamiCrafter_512_Interp/blob/main/model.ckpt)| Currently, our DynamiCrafter can support generating videos of up to 16 frames with a resolution of 576x1024. The inference time can be reduced by using fewer DDIM steps. GPU memory consumed on RTX 4090 reported by @noguchis in [Twitter](https://x.com/noguchis/status/1754488826016432341?s=20): 18.3GB (576x1024), 12.8GB (320x512), 11.9GB (256x256). ## โ๏ธ Setup ### Install Environment via Anaconda (Recommended) ```bash conda create -n dynamicrafter python=3.8.5 conda activate dynamicrafter pip install -r requirements.txt ``` ## ๐ซ Inference ### 1. Command line ### Image-to-Video Generation 1) Download pretrained models via Hugging Face, and put the `model.ckpt` with the required resolution in `checkpoints/dynamicrafter_[1024|512|256]_v1/model.ckpt`. 2) Run the commands based on your devices and needs in terminal. ```bash # Run on a single GPU: # Select the model based on required resolutions: i.e., 1024|512|320: sh scripts/run.sh 1024 # Run on multiple GPUs for parallel inference: sh scripts/run_mp.sh 1024 ``` ### Generative Frame Interpolation / Looping Video Generation Download pretrained model DynamiCrafter512_interp and put the `model.ckpt` in `checkpoints/dynamicrafter_512_interp_v1/model.ckpt`. ```bash sh scripts/run_application.sh interp # Generate frame interpolation sh scripts/run_application.sh loop # Looping video generation ``` ### 2. Local Gradio demo ### Image-to-Video Generation 1. Download the pretrained models and put them in the corresponding directory according to the previous guidelines. 2. Input the following commands in terminal (choose a model based on the required resolution: 1024, 512 or 256). ```bash python gradio_app.py --res 1024 ``` ### Generative Frame Interpolation / Looping Video Generation Download the pretrained model and put it in the corresponding directory according to the previous guidelines. ```bash python gradio_app_interp_and_loop.py ``` ## ๐ฅ Training / Fine-tuning ### Image-to-Video Generation 0. Download the WebVid Dataset, and important items in `.csv` are `page_dir`, `videoid`, and `name`. 1. Download the pretrained models and put them in the corresponding directory according to the previous guidelines. 2. Change `