patrikhuber
commented
8 years ago Hi,
What you're trying to do with initialising using the previous points was also my first intuition, and similar to you, it didn't work for the reason you mentioned. I think what's happening is the following:
- Initialising with the mean landmarks in each frame acts as a kind of regularisation, we go back near the space of valid faces. If you're curious, what you could try is train a model and train it with a high regularisation value, so the training error would be quite large - and then see if the landmarks also get scattered in your testing. I guess this would help analyse this behaviour.
- If you're initialising with the previous frame, the landmarks will be fairly close to the true location already (depending on how much the subject moves obviously). This is something that the regressor (or at least the first) actually has not learned from the training data. In the training, we initialise from the face-boxes, and thus quite far away from the true location. So if we're already close, the regressor in a sense can't really know the right thing to do. What I would do here is re-train a model with a different initialisation strategy, i.e. perturb around the ground-truth location of the training landmarks, and not around the face box. I've started this because I think it's a neat idea but never finished it so far.
I hope this helps, let me know how it goes and I'm glad to help more, I'm quite curious as well. (It's unfortunately the stuff that never gets mentioned in papers, for example Xiong & De La Torre don't disclose this information in their paper. But I'm actually pretty sure my idea should work (and might also be exactly how they do it)).
sidc9
lmrshare
Hi Patrik,
I have been testing your code for some time now for facial landmark training and tracking.
I modified rcr-track to use the landmark points from the previous frame as initialization, instead of the mean. Face detection is performed in the first frame. After that, I use the landmark points from the previous frame. But on doing this, the landmarks get scattered after just a few frames. This happens even if the face remains more or less stationary. I am pasting the code snippet that I used below. Am I doing something wrong?
(current_landmarks holds the landmarks form the previous frame, as also mentioned in your comments in the code)
if(!have_face) { ... ...
have_face=true; } else{ cv::Mat prev_landmarks = to_row(current_landmarks);
current_landmarks = rcr_model.detect(image, prev_landmarks);
rcr::draw_landmarks(image, current_landmarks); have_face=true; // simply setting it to true for now for testing purposes. }
I would greatly appreciate any help or insight from you.
Cheers!