⚡️ Lightning Track ⚡️

🚀 Track with landmark 🚀 🚅 Track with synthesis 🚅

🧐 Landmarks (2D/3D) and bounding box 🧐

Description

Lightning Track is a monocular face tracker built on FLAME. It provides optimized FLAME parameters and camera parameters, along with the bounding box and landmarks used during optimization.

Our tracker operates at a remarkable speed 🚀, processing 250 frames in approximately 60 seconds under landmark mode and 250 frames in around 200 seconds under synthesis mode.

Installation

Build environment

Install step by step

``` conda create -n track python=3.9 conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia conda install -c fvcore -c iopath -c conda-forge fvcore iopath conda install pytorch3d -c pytorch3d pip3 install mediapipe tqdm rich lmdb einops colored ninja av opencv-python scikit-image onnxruntime-gpu onnx transformers pykalman ```

Install with environment.yml (recommend)

``` conda env create -f environment.yml ```

Run with Dockerfile

``` If your environment has unknown or unsolvable issues, use the Dockerfile as a final solution. ```

Build source

Check the build_resources.sh.

Fast start

Track on video:

python track_video.py -v demos/demo.mp4 --synthesis

or track all videos in a directory:

python track_video.py -v demos/ --no_vis

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{
    chu2024gpavatar,
    title={{GPA}vatar: Generalizable and Precise Head Avatar from Image(s)},
    author={Xuangeng Chu and Yu Li and Ailing Zeng and Tianyu Yang and Lijian Lin and Yunfei Liu and Tatsuya Harada},
    booktitle={The Twelfth International Conference on Learning Representations},
    year={2024},
    url={https://openreview.net/forum?id=hgehGq2bDv}
}

xg-chu / lightning_track