Extracting FaceMask with the OpenFace 2.0

[Sharan123]

I have been using the previous version of OpenFace (1.0) to extract facial landmarks and extract the face (just the face) region with a method similar to AlignFaceMask (along with DetectLandmarksInVideos) but without the scaling, rotation, and translation (I do my own rotation of facial landmarks).

For this, I needed the FaceAnalyser's triangulation matrix, that I've obtained with GetTriangulation. Even though this method still exists I was wondering if there is a way to create a face mask without creating an instance of the FaceAnalysis class. I have seen that CLNF class has the triangulation file but a triangulation matrix is given for each view.

Is there a connection between the CLNF triangulation variable and the FaceAnalysis triangulation variable (that one gets with GetTriangulation)? Also, is there a way to create a FaceMask without instantiating the FaceAnalysis class for a sole purpose of detecting landmarks and extracting the face mask, since it seems to me that, to extract the face mask, only the triangulation matrix along with the landmarks that the CLNF class captured are needed? Thanks!

[Sharan123]

Here is how I've solved it - correct me if there is a better and a simpler way.

void loadTriangulationMatrix(std::string& model_loc, cv::Mat_<int>& triangulation)
{
    std::ifstream locations(model_loc.c_str(), std::ios_base::in);
    if (!locations.is_open())
    {
        std::cout << "Couldn't open the model file, aborting" << std::endl;
        return;
    }
    std::string line;

    // Other module locations should be defined as relative paths from the main model
    fs::path root = fs::path(model_loc).parent_path();

    while (!locations.eof())
    {
        getline(locations, line);

        std::stringstream lineStream(line);

        std::string module;
        std::string location;

        // figure out which module is to be read from which file
        lineStream >> module;

        lineStream >> location;

        // remove carriage return at the end for compatibility with unix systems
        if (location.size() > 0 && location.at(location.size() - 1) == '\r')
        {
            location = location.substr(0, location.size() - 1);
        }

        // append to root
        location = (root / location).string();
        if (module.compare("Triangulation") == 0)
        {
            std::cout << "Reading the triangulation from:" << location;
            // The triangulation used for masking out the non-face parts of aligned image
            std::ifstream triangulation_file(location);
            FaceAnalysis::ReadMat(triangulation_file, triangulation);
            std::cout << "... Done" << std::endl;
        }
    }

cv::Mat_<int> triang_mat;
FaceAnalysis::FaceAnalyserParameters fa_params(arguments);
loadTriangulationMatrix(fa_params.model_loc,triang_mat);

The loadTriangulationMatrix is adapted from the FaceAnalysis constructor (but in-general a huge chunk was reused)

[TadasBaltrusaitis]

Just for getting the triangulation you don't need to construct a FaceAnalyser object, you can do what you did in your example. You can also call directly:

std::cout << "Reading the triangulation from:" << location;
// The triangulation used for masking out the non-face parts of aligned image
std::ifstream triangulation_file(location);
FaceAnalysis::ReadMat(triangulation_file, triangulation);
std::cout << "... Done" << std::endl;

The triangulation file that gets read is lib\local\LandmarkDetector\model\tris_68_full.txt, there's no need to go through all the model definition reading.

Alternatively, you can have a look how the Paw.cpp class loads the triangulation (it uses a different triangulation and a triangulation file, but the method is very similar).