SPIGA: Shape Preserving Facial Landmarks with Graph Attention Networks.

This repository contains the source code of SPIGA, a face alignment and headpose estimator that takes advantage of the complementary benefits from CNN and GNN architectures producing plausible face shapes in presence of strong appearance changes.

It achieves top-performing results in:

Setup

The repository has been tested on Ubuntu 20.04 with CUDA 11.4, the latest version of cuDNN, Python 3.8 and Pytorch 1.12.1. To run the video analyzer demo or evaluate the algorithm, install the repository from the source code:

# Best practices: 
#  1. Create a virtual environment.
#  2. Install Pytorch according to your CUDA version.
#  3. Install SPIGA from source code:

git clone https://github.com/andresprados/SPIGA.git
cd spiga
pip install -e .  

# To run the video analyzer demo install the extra requirements.
pip install -e .[demo]

Models: By default, model weights are automatically downloaded on demand and stored at ./spiga/models/weights/. You can also download them from Hugging Face and Google Drive.

Note: All the callable files provide a detailed parser that describes the behaviour of the program and their inputs. Please, check the operational modes by using the extension --help.

Inference and Demo

We provide an inference framework for SPIGA available at ./spiga/inference. The models can be easily deployed in third-party projects by adding a few lines of code. Check out our inference and application tutorials for more information:

Face Video Analyzer Demo:

The demo application provides a general framework for tracking, detecting and extracting features of human faces in images or videos. You can use the following commands to run the demo:

python ./spiga/demo/app.py \
            [--input] \      # Webcam ID or Video Path. Dft: Webcam '0'.
            [--dataset] \    # SPIGA pretrained weights per dataset. Dft: 'wflw'.
            [--tracker] \    # Tracker name. Dft: 'RetinaSort'.
            [--show] \       # Select the attributes of the face to be displayed. Dft: ['fps', 'face_id', 'landmarks', 'headpose']
            [--save] \       # Save record.
            [--noview] \     # Do not visualize window.
            [--outpath] \    # Recorded output directory. Dft: './spiga/demo/outputs'
            [--fps] \        # Frames per second.
            [--shape] \      # Visualizer shape (W,H).

Note: For more information check the Demo Readme or call the app parser --help.

Dataloaders and Benchmarks

This repository provides general-use tools for the task of face alignment and headpose estimation:

• Dataloaders: Training and inference dataloaders are available at ./spiga/data. Including the data augmentation tools used for training SPIGA and data-visualizer to analyze the dataset images and features. For more information check the Data Readme .

• Benchmark: A common benchmark framework to test any algorithm in the task of face alignment and headpose estimation is available at ./spiga/eval/benchmark. For more information check the following Evaluation Section and the Benchmark Readme.

Datasets: To run the data visualizers or the evaluation benchmark please download the dataset images from the official websites (300W, AFLW, WFLW, COFW). By default they should be saved following the next folder structure:

./spiga/data/databases/   # Default path can be updated by modifying 'db_img_path' in ./spiga/data/loaders/dl_config.py
|
└───/300w               
│   └─── /images           
│        | /private     
│        | /test                   
|        └ /train             
|
└───/cofw                   
│   └─── /images
|  
└───/aflw                   
│   └─── /data
|        └ /flickr
|  
└───/wflw
    └─── /images

Annotations: We have stored for simplicity the datasets annotations directly in ./spiga/data/annotations. We strongly recommend to move them out of the repository if you plan to use it as a git directory.

Results: Similar to the annotations problem, we have stored the SPIGA results in ./spiga/eval/results/<dataset_name>. Remove them if need it.

Evaluation

The models evaluation is divided in two scripts:

Results generation: The script extracts the data alignments and headpose estimation from the desired <dataset_name> trained network. Generating a ./spiga/eval/results/results_<dataset_name>_test.json file which follows the same data structure defined by the dataset annotations.

python ./spiga/eval/results_gen.py <dataset_name>

Benchmark metrics: The script generates the desired landmark or headpose estimation metrics. We have implemented an useful benchmark which allows you to test any model using a results file as input.

python ./spiga/eval/benchmark/evaluator.py /path/to/<results_file.json> --eval lnd pose -s

Note: You will have to interactively select the NME_norm and other parameters in the terminal window.

Results Sum-up

BibTeX Citation

@inproceedings{Prados-Torreblanca_2022_BMVC,
  author    = {Andrés  Prados-Torreblanca and José M Buenaposada and Luis Baumela},
  title     = {Shape Preserving Facial Landmarks with Graph Attention Networks},
  booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
  publisher = {{BMVA} Press},
  year      = {2022},
  url       = {https://bmvc2022.mpi-inf.mpg.de/0155.pdf}
}