Realistic one-shot mesh-based avatars

Paper | Project Page

This repository contains official inference code for ROME.

This code helps you to create personal avatar from just a single image. The resulted meshes can be animated and rendered with photorealistic quality.

Important disclaimer

To render ROME avatar with pretrained weights you need download FLAME model and DECA weights. DECA reconstructs a 3D head model with detailed facial geometry from a single input image for FLAME template. Also, it can be replaced by another parametric model.

Getting started

Initialise submodules and download DECA & MODNet weights

git submodule update --init --recursive

Install requirements and download ROME model: gDrive, y-disk.

Put model into data folder.

To verify the code with images run:

python3 infer.py --deca DECA --rome data

For linear basis download ROME model: gDrive (or camera model for voxceleb gDrive), yDrive

python3 infer.py --deca DECA --rome data  --use_distill True

License

This code and model are available for scientific research purposes as defined in the LICENSE file. By downloading and using the project you agree to the terms in the LICENSE and DECA LICENSE. Please note that the restriction on distributing this code for non-scientific purposes is limited.

Links

This work is based on the great project DECA. Also we acknowledge additional projects that were essential and speed up the developement.

• DECA for FLAME regressor and useful functions
• Pytorch3D for differentiable rendering,
• face-alignment for keypoints
• VGGFace2-pytorch for identity loss
• MODNet, FaceParsing, Graphonomy for fast segmentations
• H3DNet for evaluation geometry

Citation

If you found this code helpful, please consider citing:

@inproceedings{Khakhulin2022ROME,
  author    = {Khakhulin, Taras and Sklyarova,  Vanessa and Lempitsky, Victor and Zakharov, Egor},
  title     = {Realistic One-shot Mesh-based Head Avatars},
  booktitle = {European Conference of Computer vision (ECCV)},
  year      = {2022}
}