Extreme 3D Face Reconstruction: Seeing Through Occlusions

Please note that the main part of the code has been released, though we are still testing it to fix possible glitches. Thank you.

Python and C++ code for realistic 3D face modeling from single image using our shape and detail regression networks published in CVPR 2018 [1] (follow the link to our PDF which has many, many more reconstruction results.)

This page contains end-to-end demo code that estimates the 3D facial shape with realistic details directly from an unconstrained 2D face image. For a given input image, it produces standard ply files of the 3D face shape. It accompanies the deep networks described in our paper [1] and [2]. The occlusion recovery code, however, will be published in a future release. We also include demo code and data presented in [1].

Dependencies

Data requirements

Before compiling the code, please, make sure to have all the required data in the following specific folder:

• Download our Bump-CNN and move the CNN model (1 file: ckpt_109_grad.pth.tar) into the CNN folder
• Download our PyTorch CNN model and move the CNN model (3 files: shape_model.pth, shape_model.py, shape_mean.npz) into the CNN folder
• Download the Basel Face Model and move 01_MorphableModel.mat into the 3DMM_model folder
• Acquire 3DDFA Expression Model, run its code to generate Model_Expression.mat and move this file the 3DMM_model folder
• Go into 3DMM_model folder. Run the script python trimBaselFace.py. This should output 2 files BaselFaceModel_mod.mat and BaselFaceModel_mod.h5.
• Download dlib face prediction model and move the .dat file into the dlib_model folder.

Note that we modified the model files from the 3DMM-CNN paper. Therefore, if you generated these files before, you need to re-create them for this code.

Installation

There are 2 options below to compile our code:

Installation with Docker (recommended)

• Install Docker CE
• With Linux, manage Docker as non-root user
• Install nvidia-docker
• Build docker image:

  docker build -t extreme-3dmm-docker .

  Installation without Docker on Linux

The steps below have been tested on Ubuntu Linux only:

• Install Python2.7
• Install the required third-party packages:

  sudo apt-get install -y libhdf5-serial-dev libboost-all-dev cmake libosmesa6-dev freeglut3-dev

• Install Dlib C++ library. Sample code to comiple Dlib:

  wget http://dlib.net/files/dlib-19.6.tar.bz2
  tar xvf dlib-19.6.tar.bz2
  cd dlib-19.6/
  mkdir build
  cd build
  cmake ..
  cmake --build . --config Release
  sudo make install
  cd ..

• Install PyTorch
• Install other required third-party Python packages:

  pip install opencv-python torchvision scikit-image cvbase pandas mmdnn dlib

• Config Dlib and HDF5 path in CMakefiles.txt, if needed
• Build C++ code

  mkdir build;
  cd build; \
  cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=../demoCode ..;
  make;
  make install;
  cd ..

  This code should generate TestBump in demoCode folder

Usage

Start docker container

If you compile our code with Docker, you need to start a Docker container to run our code. You also need to set up a shared folder to transfer input/output data between the host computer and the container.

• Prepare the shared folder on the host computer. For example, /home/ubuntu/shared
• Copy input data (if needed) to the shared folder
• Start container:

  nvidia-docker run --rm -ti --ipc=host --privileged -v /home/ubuntu/shared:/shared extreme-3dmm-docker bash

  Now folder /home/ubuntu/shared on your host computer will be mounted to folder /shared inside the container

3D face modeling with realistic details from a set of input images

• Go into demoCode folder. The demo script can be used from the command line with the following syntax:

$ Usage: python testBatchModel.py <inputList> <outputDir>

where the parameters are the following:

• <inputList> is a text file containing the paths to each of the input images, one in each line.
• <outputDir> is the path to the output directory, where ply files are stored.

An example for <inputList> is demoCode/testImages.txt

../data/test/03f245cb652c103e1928b1b27028fadd--smith-glasses-too-faced.jpg
../data/test/20140420_011855_News1-Apr-25.jpg
....

The output 3D models will be <outputDir>/<imageName>_<postfix>.ply with <postfix> = <modelType>_<poseType>. <modelType> can be "foundation", "withBump" (before soft-symmetry),"sparseFull" (soft-symmetry on the sparse mesh), and "final". <poseType> can be "frontal" or "aligned" (based on the estimated pose). The final 3D shape has <postfix> as "final_frontal". You can config the output models in code before compiling.

The PLY files can be displayed using standard off-the-shelf 3D (ply file) visualization software such as MeshLab.

Sample command:

    python testBatchModel.py testImages.txt /shared

Note that our occlusion recovery code is not included in this release.

Demo code and data in our paper

• Go into demoCode folder. The demo script can be used from the command line with the following syntax:

$ Usage: ./testPaperResults.sh

Before exiting the docker container, remember to save your output data to the shared folder.

Citation

If you find this work useful, please cite our paper [1] with the following bibtex:

@inproceedings{tran2017extreme,
  title={Extreme {3D} Face Reconstruction: Seeing Through Occlusions},
  author={Tran, Anh Tuan and Hassner, Tal and Masi, Iacopo and Paz, Eran and Nirkin, Yuval and Medioni, G\'{e}rard},
  booktitle={IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
  year=2018
}

References

[1] A. Tran, T. Hassner, I. Masi, E. Paz, Y. Nirkin, G. Medioni, "Extreme 3D Face Reconstruction: Seeing Through Occlusions", IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, June 2018

[2] A. Tran, T. Hassner, I. Masi, G. Medioni, "Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network", CVPR 2017

Changelog

• Dec. 2018, Convert to Dockerfile
• Dec. 2017, First Release

License and Disclaimer

Please, see the LICENSE here

Contacts

If you have any questions, drop an email to anhttran@usc.edu , hassner@isi.edu and iacopoma@usc.edu or leave a message below with GitHub (log-in is needed).