โ totally no tuning โ less than 20% extra time โ support 512 frames
[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/MoonQiu/FreeNoise) [![Replicate](https://replicate.com/cjwbw/longercrafter/badge)](https://replicate.com/cjwbw/longercrafter) _**[Haonan Qiu](http://haonanqiu.com/), [Menghan Xia*](https://menghanxia.github.io), [Yong Zhang](https://yzhang2016.github.io), [Yingqing He](https://github.com/YingqingHe),Input: "A chihuahua in astronaut suit floating in space, cinematic lighting, glow effect";
Resolution: 1024 x 576; Frames: 64.
Input: "Campfire at night in a snowy forest with starry sky in the background";
Resolution: 1024 x 576; Frames: 64.
๐ค๐ค๐ค LongerCrafter (FreeNoise) is a tuning-free and time-efficient paradigm for longer video generation based on pretrained video diffusion models.
Longer single-prompt results. Resolution: 256 x 256; Frames: 512. (Compressed)
Longer multi-prompt results. Resolution: 256 x 256; Frames: 256. (Compressed)
Model | Resolution | Checkpoint | Description |
---|---|---|---|
VideoCrafter (Text2Video) | 576x1024 | Hugging Face | Support 64 frames on NVIDIA A100 (40GB) |
VideoCrafter (Text2Video) | 256x256 | Hugging Face | Support 512 frames on NVIDIA A100 (40GB) |
VideoCrafter2 (Text2Video) | 320x512 | Hugging Face | Support 128 frames on NVIDIA A100 (40GB) |
(Reduce the number of frames when you have smaller GPUs, e.g. 256x256 resolutions with 64 frames.)
conda create -n freenoise python=3.8.5
conda activate freenoise
pip install -r requirements.txt
1) Download pretrained T2V models via Hugging Face, and put the model.ckpt
in checkpoints/base_1024_v1/model.ckpt
.
2) Input the following commands in terminal.
sh scripts/run_text2video_freenoise_1024.sh
1) Download pretrained T2V models via Hugging Face, and put the model.ckpt
in checkpoints/base_256_v1/model.ckpt
.
2) Input the following commands in terminal.
sh scripts/run_text2video_freenoise_mp_256.sh
FreeNoise is supposed to work on other similar frameworks. An easy way to test compatibility is by shuffling the noise to see whether a new similar video can be generated (set eta to 0). If your have any questions about applying FreeNoise to other frameworks, feel free to contact Haonan Qiu.
Current official implementation: FreeNoise-VideoCrafter, FreeNoise-AnimateDiff, FreeNoise-LaVie
VideoCrafter: Framework for high-quality video generation.
ScaleCrafter: Tuning-free method for high-resolution image/video generation.
TaleCrafter: An interactive story visualization tool that supports multiple characters.
@misc{qiu2023freenoise,
title={FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling},
author={Haonan Qiu and Menghan Xia and Yong Zhang and Yingqing He and Xintao Wang and Ying Shan and Ziwei Liu},
year={2023},
eprint={2310.15169},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
We develop this repository for RESEARCH purposes, so it can only be used for personal/research/non-commercial purposes.