A libOBS filter plugin that detects faces and draws masks with the detected data.
Clone this repository with submodules:
git clone --recursive https://github.com/stream-labs/facemask-plugin
Download cmake:
Get Visual Studio 2015 (vc14) or Visual Studio 2017. When you install it, make sure you include the C++ stuff.
Download our fork of OBS Studio:
Do not forget the submodules:
git clone --recursive https://github.com/stream-labs/obs-studio.git
Build obs-studio
Note: You do not need to install Qt5 as a dependency, just make sure ENABLE_UI in CMake GUI is not checked before Configure/Generate. You can also handle this by setting ENABLE_UI to OFF in CMakeLists in the obs-studio folder.
Run cmake in the facemasks folder. When you hit CONFIGURE
, you will get errors on fields you need to fill in:
PATH_OBS_STUDIO Path to the obs-studio folder.
BUILD_SLOBS - Distributes to slobs instead of OBS Studio
If you have the Intel Math Kernel Library installed on your system, you might have DLIB_USE_BLAS or DLIB_USE_LAPACK turned on. Keep in mind that dlib links dynamically with these libs, so the MKL and TBB dlls will need to be found by slobs when it runs (for instance, by copying them into the slobs-client folder). I don't reccommend using these libs for this reason.
Once you have successfully configured and generated your Visual Studio project with cmake, you can open the facemask-plugin.sln file in Visual Studio. You can now compile the plugin, which will give you a distribution folder structure that mimics the structure in slobs. For example, if you built your files in the build64 folder:
build64/distribute/slobs/RelWithDebInfo/obs-plugins
If you are going to debug facemask code make sure to build it as RelWithDebInfo configuration. You can set this in Visual Studio.
You can copy the files in manually, or set up symbolic links so you can easily hit F5 and debug from Visual Studio. You can also modify the CMakeLists to copy the files to a desired location with each build.
The plugin is an OBS filter plugin. It can be broken down into 2 main parts; the plugin, and the face detection. The face detection runs in its own thread, separate from OBS.
The plugin portion performs the following duties:
The face detection portion runs in its own thread. It consumes frame data from the circular buffer, does the face detection computation, then feeds the resulting face data to a circular buffer that is consumed by the rendering.
The process of face detection consists of four main operations:
Face Detection The faces are detected using the histogram of oriented gradients method (HOG) in Dlib. HOG's are trained feature descriptors used for detecting objects.
Tracking Once we have detected the faces we then use a cheaper method of object tracking to follow the face. This Dlib object tracking method takes an arbitrary rectangle in an image and follows it.
Facial Landmarks Given a rectangle that locates a face, we can then use Dlib's landmark detection algorithm which uses a trained regression tree solver to find 68 2D facial landmark points, corresponding to the Multi-PIE definition (see links below).
3D Pose Estimation A subset of key points are taken from the 2D facial landmark points, and using 3D points for an arbitrary rest pose, we use openCV's solvePnP method to obtain a 3D transformation. This transform can be used to render 3D objects in the scene that track the head movement.
Face Morphing The 68 landmark points are used to subdivide the video quad into a mesh. Another 11 points are calculated to form the head points, and then catmull rom smoothing is performed to smooth out the contours. Then the mesh is distorted to create face morphs.
The FaceDetect object manages these operations and a current state, so that it performs the face detection, then uses object tracking to follow the face, then does landmark/3d pose estimation, and then the mesh subdivision for face morphing.
3D Tracking of Facial Features for Augmented Reality Applications
Automatic filtering techniques for three-dimensional kinematics data using 3D motion capture system
A Collection of Useful C++ Classes for Digital Signal Processing
FaceWarehouse: a 3D Facial Expression Database for Visual Computing
3D Face Reconstruction with Geometry Details from a Single Image
Install FFmpeg: ffmpeg Add a Path of FFmpeg executable in the Environment Variables: The path should look like: %PATH_TO_YOUR_DIR%\ffmpeg-%VERSION%\bin
Install ImageMagick: ImageMagick Turn OFF the Install FFmpeg option during the instalation Turn ON the Install Legacy Utilities (e.g. convert) option during the instalation
Download and build giflossy: giflossy Note: Gifview is not needed Add a Path of the gifsicle executable in the Environment Variables: The path should look like: %PATH_TO_YOUR_DIR%\giflossy\src
Install SVN: SVN Turn ON the 'Command Line Clients Tool' option during the installation
Start/Restart Streamlabs-OBS
Add a Media Resource
Add a filter: Face Mask Plugin
Go to Setting of facemask-plugin:
Note: if you want to regenerate existing mask, restart the program and remove associated thumbnails from the demo folder
For Unit Testing CppUtest framework is used
Formatting video makes start/end of the looped video detectable by adding a red frame at the start
./format.bat /path/to/the/input/video /path/to/the/output/video
Example:
./format.bat inputfolder/myinput.mp4 outputfolder/myoutput.mp4
It will generate output with red frame appended
This is the best way to use this feature. It will align the two videos perfectly.
./sidebyside.bat /path/to/the/beforeVideo /path/to/the/afterVideo before-text after-text /path/to/output/file