Face Detection

[realmosn]

Thank you for your awesome work and you contribution.

I suspect current face recognition is resource intensive and would like to know if I can replace it with iOS internal face recognition. In other words is there anyway to distinguish LandMark Detection from Face Detection and do you have any suggestion where to start.

Thanks

[TadasBaltrusaitis]

Hi,

This framework does not perform face recognition, but is rather able to perform facial behavior analysis (facial expression, head pose, gaze, etc.).

Did you mean to say that face detection is resource intensive? If so yes, but there are some cheaper and less accurate algorithms you could use to perform face detection, such as OpenCV's inbuilt Haar detector.

[realmosn]

Thanks for quick response. I can also do the dace detection using Cocoa's built-in API which give me the bounds of the detected face in a stream. If I understand correctly you are suggesting I can bypass Face Detection and pass in the face boundaries to perform facial behavior analysis.

If above is correct could you suggest a method that I can pass these coordinates to.

Sorry if it sounds dumb, I am not so good in C++

[TadasBaltrusaitis]

You could possibly use Cocoa's API for face detection (however, I do not know how fast/efficient it is). To perform the landmark detection given a face bounding box you will need to change the code in the function LandmarkDetector::DetectLandmarksInVideo and replace the way it is currently performing face detection. However, do note that the face bounding box expected by landmark detector might not be the one returned by Cocoa API, as it expects a tight box around the 68 landmarks.

[realmosn]

Dear Tadas,

as your suggestion I was able to switch to iOS internal Face Detection and I can confirm that initial Face Detection is much faster now, However I am still experiencing 97% CPU usage and it seems that following method is consuming most of the CPU

LandmarkDetector::crossCorr_m(cv::Mat_<float> const&, cv::Mat_<double>&, cv::Mat_<float> const&, std::__1::map<int, cv::Mat_<double>, std::__1::less<int>, std::__1::allocator<std::__1::pair<int const, cv::Mat_<double> > > >&, cv::Mat_<float>&)

Do you have any suggestion how I can improve this

[TadasBaltrusaitis]

There is not that many easy things you could do:

1. Try not detecting every landmark and only use a subset of them, or perform fewer iterations. Another option is not to perform the full computation every frame.
2. Move the crossCorr_m computation onto the GPU if the load of the CPU is very high (this would require some non trivial coding)

[realmosn]

Do you have any particular method in mind that I can modify to achieve detecting less landmarks ? which in my ideal case since I mostly need eyebrows landmarks.

[TadasBaltrusaitis]

You can try not fitting the face outline landmarks or only fitting a subset of them, also you can look at fitting only a subset of mouth ones as well. Easiest way to ignore those landmarks is to set the visibility flag for the corresponding patch experts to 0.

[realmosn]

Apologies if this sound stupid, I am not very familiar with C++ but by "set the visibility flag for the corresponding patch experts to 0" you mean to change values for corresponding landmarks in main_clnf_general.txt ?

for instance I tried to change LandmarkDetector_part model_inner/main_clnf_inner.txt inner to only 0 but that didn't change anything in the detection of landmarks.

[TadasBaltrusaitis]

You can do it two ways:

1. through writing out a patch expert model that has different visibilities set (using Matlab code)
2. Changing the C++ code directly and setting What I meant is changing the C++ code directly, specifically the face_model.patch_experts.visibilities variable, this controls which landmarks are being actively tracked at a particular orientation or scale. This is what I mean by the visibility flag. If you set for example the visibility of landmark 0 at all views and scales to 0, the algorithm will stop actively tracking it.