search

mayuelala / FollowYourEmoji

[Siggraph Asia 2024] Follow-Your-Emoji: This repo is the official implementation of "Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation"
327 stars 25 forks source link

readme

Follow-Your-Emoji
Fine-Controllable and Expressive Freestyle Portrait Animation

[Yue Ma*](https://mayuelala.github.io/), [Hongyu Liu*](https://kumapowerliu.github.io/), [Hongfa Wang*](https://github.com/mayuelala/FollowYourEmoji), [Heng Pan*](https://github.com/mayuelala/FollowYourEmoji), [Yingqing He](https://github.com/YingqingHe), [Junkun Yuan](https://0-scholar-google-com.brum.beds.ac.uk/citations?user=j3iFVPsAAAAJ&hl=zh-CN), [Ailing Zeng](https://ailingzeng.site/), [Chengfei Cai](https://github.com/mayuelala/FollowYourEmoji), [Heung-Yeung Shum](https://scholar.google.com.hk/citations?user=9akH-n8AAAAJ&hl=en), [Wei Liu](https://scholar.google.com/citations?user=AjxoEpIAAAAJ&hl=zh-CN) and [Qifeng Chen](https://cqf.io) is Accpeted by Siggraph Asia 2024 ![visitors](https://visitor-badge.laobi.icu/badge?page_id=mayuelala.FollowYourEmoji&left_color=green&right_color=red) [![Open in OpenXLab](https://cdn-static.openxlab.org.cn/app-center/openxlab_app.svg)](https://openxlab.org.cn/apps/detail/houshaowei/FollowYourEmoji) [![GitHub](https://img.shields.io/github/stars/mayuelala/FollowYourEmoji?style=social)](https://github.com/mayuelala/FollowYourEmoji,pko)

📣 Updates

🤪 Gallery

Image 1

We present Follow-Your-Emoji, a diffusion-based framework for portrait animation, which animates a reference portrait with target landmark sequences.

🤪 Getting Started

1. Clone the code and prepare the environment

pip install -r requirements.txt

2. Download pretrained weights

[FollowYourEmoji] We also provide our pretrained checkpoints in Huggingface. you could download them and put them into checkpoints folder to inference our model.

Finally, these pretrained models should be organized as follows:

pretrained_models
├── AnimateDiff
│   └── mm_sd_v15_v2.ckpt
├── follow-your-emoji
│   ├── lmk_guider.pth
│   ├── referencenet.pth
│   └── unet.pth
├── sd-image-variations-diffusers
│   ├── alias-montage.jpg
│   ├── default-montage.jpg
│   ├── earring.jpg
│   ├── feature_extractor
│   │   └── preprocessor_config.json
│   ├── image_encoder
│   │   ├── config.json
│   │   └── pytorch_model.bin
│   ├── inputs.jpg
│   ├── model_index.json
│   ├── README.md
│   ├── safety_checker
│   │   ├── config.json
│   │   └── pytorch_model.bin
│   ├── scheduler
│   │   └── scheduler_config.json
│   ├── unet
│   │   ├── config.json
│   │   └── diffusion_pytorch_model.bin
│   ├── v1-montage.jpg
│   ├── v2-montage.jpg
│   └── vae
│       ├── config.json
│       └── diffusion_pytorch_model.bin
└── sd-vae-ft-mse
    ├── config.json
    ├── diffusion_pytorch_model.bin
    ├── diffusion_pytorch_model.safetensors
    └── README.md

3. Inference 🚀

bash infer.sh
CUDA_VISIBLE_DEVICES=0 python3 -m torch.distributed.run \
    --nnodes 1 \
    --master_addr $LOCAL_IP \
    --master_port 12345 \
    --node_rank 0 \
    --nproc_per_node 1 \
    infer.py \
    --config ./configs/infer.yaml \
    --model_path /path/to/model \
    --input_path your_own_images_path \
    --lmk_path ./inference_temple/test_temple.npy  \
    --output_path your_own_output_path

🤪 Make Your Emoji

You can make your own emoji using our model. First, you need to make your emoji temple using MediaPipe. We provide the script in make_temple.ipynb. You can replace the video path with your own emoji video and generate the .npy file.

CUDA_VISIBLE_DEVICES=0 python3 -m torch.distributed.run \
    --nnodes 1 \
    --master_addr $LOCAL_IP \
    --master_port 12345 \
    --node_rank 0 \
    --nproc_per_node 1 \
    infer.py \
    --config ./configs/infer.yaml \
    --model_path /path/to/model \
    --input_path your_own_images_path \
    --lmk_path  your_own_temple_path \
    --output_path your_own_output_path

👨‍👩‍👧‍👦 Follow Family

Follow-Your-Pose: Pose-Guided text-to-Video Generation.

Follow-Your-Click: Open-domain Regional image animation via Short Prompts.

Follow-Your-Handle: Controllable Video Editing via Control Handle Transformations.

Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation.

Citation 💖

If you find Follow-Your-Emoji useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX:

@article{ma2024follow,
  title={Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation},
  author={Ma, Yue and Liu, Hongyu and Wang, Hongfa and Pan, Heng and He, Yingqing and Yuan, Junkun and Zeng, Ailing and Cai, Chengfei and Shum, Heung-Yeung and Liu, Wei and others},
  journal={arXiv preprint arXiv:2406.01900},
  year={2024}
}