HoloFace is a framework for facial augmented reality on HoloLens, it allows for augmenting the faces of people seen through HoloLens with various items and visual effects. It is described in detail in an article published at the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV'2018). A preprint of the article is available on arXiv.
Look the video below to see HoloFace in action.
In order to run HoloFace you will need Unity (we used Unity 2017.3.0f3) and Visual Studio 2017. To run the application on HoloLens take the following steps:
If everything worked fine the application should now be running on your HoloLens. Now, look at someone's face, if the application works correctly, you should see fire going out of that person's mouth. You can also look at an image of a face on a screen, but keep in mind that if the size of the face on the screen does not match a real life face size, the augmented elements will not be correctly aligned.
For more information about how to use the application, see the next paragraphs.
If you are experiencing issues, please let us know in the Issues section above, all feedback is very valuable.
When the app is running you can use the following commands:
As explained in the article, HoloFace has supports two methods for facial landmark localization (face alignment). The default method runs locally on the device. The alternative method runs on a remote backend but is significantly more accurate and stable.
To run the remote method, first clone the Deep Alignment Network respoitory and follow the setup instruction there. Once everything is configured you can run the remote backend by executing the following command:
python HoloFaceBackend.py IP_OF_THE_HOLOLENS
You can configure HoloFace to render any Game Object on top of a face seen through the headset. In order to add a new object take the following steps:
If you want your object to be activated by a face expression you need to do two things. First add a script containing a MonoBehavious that inherits from IAnimationTrigger to your object (see SparksTrigger for an example). Second, add your object to the appropriate list in HoloFace->ItemManager. For example, if you want the action to be activated when the person opens their mouth, add the object to the MouthOpenActivations list. Note that the mouth opening activation is more robust than the other two and usually gives the best results.
There are currently two known issues, if you know how to solve them please let us know.
FrameToOrigin
matrix that is a part of SensorFrame
returned by HoloLensForCV is off if the headset moves during app initialization.HoloFace uses the HoloLensForCV library from Microsoft. The library is however included in the HoloFace repo since we have made minor modifications to its code. Because of that there is no need to download HoloLensForCV separately.
The Unity project includes the following third party assets:
All of the assets above were included in the projection with permission from the authors
If you use this software in your research, then please cite the following paper:
Kowalski, M.; Nasarzewski Z.; Galinski G., Garbat P.: " HoloFace: Augmenting Human-to-Human Interactions on HoloLens ", WACV 2018
The landmarks used for training the face alignment models in HoloFace are part of the 300-W and Menpo datasets. Note that the licenses for those datasets exclude commercial use. You should contact s.zafeiriou@imperial.ac.uk to find out if it's OK for you to use the model files in a commercial product.
If you have any questions or suggestions feel free to contact me at m.kowalski@ire.pw.edu.pl.