Open nitheeshas opened 8 years ago
Hi @nitheeshas,
do you want C++ example, MATLAB or Python?
Thanks for replying. I would like to see an example in C++.
Ok, I will skip the face detector part, since I now have code only using a commercial one. However, it is possible to use combination of OpenCV haarcascades for frontal and profile faces.
Lets assume that we have the bbox of the face in the image (the format of bbox is described e.g. here: #20 ). The function which jointly detects the discretized yaw angle and landmarks looks like this:
void jointmv_detector(Flandmark **flandmarkPool, int *bbox, int *viewID)
{
const int PHIS = 5;
fl_double_t scores[PHIS];
fl_double_t maximum = -INFINITY;
for (int phi=0; phi < PHIS; ++phi)
{
Flandmark *flandmark = flandmarkPool[phi];
flandmark->detect_optimizedFromPool(bbox);
// compute score
scores[phi] = flandmark->getScore();
if (scores[phi] > maximum)
{
maximum = scores[phi];
*viewID = phi;
}
}
}
the viewID
serves as a pointer to flandmarkPool
, so we can later extract landmarks and view label.
Now how to initialize the flandmarkPool
. Lets assume we have a following .txt file:
./models/PART_fixed_JOINTMV_-PROFILE.xml
./models/PART_fixed_JOINTMV_-HALF-PROFILE.xml
./models/PART_fixed_JOINTMV_FRONTAL.xml
./models/PART_fixed_JOINTMV_HALF-PROFILE.xml
./models/PART_fixed_JOINTMV_PROFILE.xml
Then we can use the following function to parse it
std::vector<std::string> readModelList(const char *file)
{
std::vector<std::string> out;
std::ifstream infile;
infile.open(file);
std::string line;
while (std::getline(infile, line))
{
out.push_back(line);
}
return out;
}
So, in the main
function we can use it as follows:
// read models from a text file
std::vector<std::string> models = readModelList(argv[3]);
Flandmark *flandmarkPool[models.size()]; // pool of Flandmark instances
for (int i=0; i < models.size(); ++i)
{
flandmarkPool[i] = Flandmark::getInstanceOf(models[i].c_str());
if (!flandmarkPool[i])
{
cerr << "Couldn't create instance of flandmark with model " << models[i] << endl;
return -1;
}
}
tim = timer.toc();
const int * bw_size = flandmarkPool[0]->getBaseWindowSize();
CFeaturePool * featuresPool = new CFeaturePool(bw_size[0], bw_size[1]);
featuresPool->addFeaturesToPool(
new CSparseLBPFeatures(featuresPool->getWidth(),
featuresPool->getHeight(),
featuresPool->getPyramidLevels(),
featuresPool->getCumulativeWidths()
)
);
for (unsigned int i=0; i < models.size(); ++i)
{
flandmarkPool[i]->setNFfeaturesPool(featuresPool);
}
this initializes flandmarkPool (view dependent instances of Flandmark with corresponding models loaded) and featuresPool (the helper structure, which shares precomputed features among Flandmark instances).
Prior calling jointmv_detector
function, do not forget to do this
featuresPool->updateNFmipmap(featuresPool->getWidth(), featuresPool->getHeight(), flandmarkPool[0]->getNF(frm_gray, &bbox[0])->data());
where cimg_library::CImg* frm_gray
is supposed to be filled with the grayscale input image. This initiates feature computation in featuresPool
a necessary step for the function jointmv_detector
to work properly.
Thanks a lot for the detailed explanation! I had doubt regarding the face detector too for multi view since opencv's profile face detector gave only an average result. I saw in the website that you were using Eyedea face detector. Their face detection seems to be almost perfect. Anyway, I'll try this out right away. Thanks again!
Yeah, Eyedea face detector is performing really well. It implements this paper, if you would like to re-implement it. I guess another option is to re-train the OpenCV profile detector.
Wow, waldboost? Its actually already implemented by someone. Its available in opencv-contrib. I'll try to train it and check how good it performs.
Hi @uricamic I was able to build the multi view landmark extraction using Dlib's face detector. I used the jointly learned landmarks pool. But the extracted landmarks are not that proper. Is it a known problem?
Hi @nitheeshas,
the models currently available are learned on a very limited training set. We are currently learning them on a bigger database. It is also possible that since the search spaces are shrinked (in order to get the detector as fast as possible), the dlib's face detector should be corrected to match the expectations for the face detector used in training. Hard to tell without seeing some examples, though.
I've uploaded an example demo video of the outputs I got. Please check. https://www.youtube.com/watch?v=25dbq7KSLsI
Sorry for the poor quality!
It seems that the face detection is really suffering a huge variance in scale and position. On the other hand, when it is as one would expect, it looks quite nice, I would say.
One quick suggestion which should improve the accuracy a lot is to stabilize the face detector output by e.g. Kalman filtering.
The new models should also improve the quality a lot, however, they are not yet fully learned.
Yes, I just started modifying the code for Kalman filter now. Will update how it works :)
@uricamic I was not able to add kalman filter since i got caught up with some other work. But the thing is, while testing the previous output, even when i was standing perfectly still, and the face detector output was also pretty much constant, the detected landmarks kept jumping a lot.
Maybe the best solution for this problem is to build fully learned models as you said. Are you still working on creating better models?
@nitheeshas, I think in such case the problem is with a noise in the webcam input. The new models should help a bit, but depending on how severe the noise is.
New version should be learned within few days, the biggest benefit should be the better yaw estimation precision and I hope to some extent also the landmark localization accuracy. However, the accuracy is limited due to relatively small normalized frame. The idea is to have this detector as an initial phase and then for precise landmark detection or tracking use better model (either with increased size of normalized frame or using regression, to remove the systematic error introduced by transforming landmarks from the normalized frame back to image).
In that case, it will be better to wait for the new learned model and if its still shaky, will add a Kalman filter and check again.
Hope you'll update soon.
Hi @uricamic Can you share the dataset which you are using to train the multi view landmark detection?
Hi @nitheeshas,
we are still working on that. Maybe, some smaller portion of examples could be published soon. Sorry for the delay.
@uricamic : A small question:
You suggest adding a call to updateNFmipmap
prior todetect_optimized
.
But the latter function already includes a call to updateNFmipmap
. And the static_input
example you supply does not have that independent call to updateNFmipmap
and yet it seems to give good results nonetheless.
Also, since the CSparseLBPFeatures
class inside the CFeaturePool
is protected, this cannot be done on an image-by-image basis, but only during the CFeaturePool
initialization. If this is indeed a critical stage, then you should add an init_CSparseLBPFeatures
function to the CFeaturePool
class.
Hi @mousomer,
I think there is some misunderstanding. The updateNFmipmap
method of CFeaturePool
is needed if you want to call the detection on multiple images. You simply exhange the image on which the detection is performed. Without costly re-initialization of the objects. Btw, the static_input
example is also using it (see here).
The features are computed automatically inside the CFeaturePool
class, when you call this updateNFmipmap
method (see here), user is not supposed to interfere in features computation anyhow.
Maybe the names of some methods are a bit confusing, I am sorry if it is the case. However, all the important functionality is there and working. Some methods are there also just because of the MATLAB interface, especially for the purpose of the model learning, where the speed is very important.
Well, you pointed into the detect_optimized
function, which I suppose is the main API for extracting the features. So, am I correct in understanding that I don't need an extra call to updateFmipmap
before I call detect_optimized
?
Hi @mousomer,
yes, for detect_optimized
you really do not need to do that extra call of updateFmipmap
.
However, check the post, where I was suggesting this call. It was for the jointmv_detector
, which is internally calling detect_optimizedFromPool
. Then, you have to call updateFmipmap
prior calling the detector, because in that case there is no other way how to update the image and let the features to be computed. The reason why it is so is simply because there are multiple detectors to run, and the landmarks of the detector which has the maximal response are returned. The features are computed just once per face image and used in all detectors.
I see. Thanks! Oh, and if I haven't mentioned it before - this package is really awesome.
@mousomer No problem, it is always good to ask questions ;-)
Thanks!
I re-run my sample set with optimizedFromPool
instead of detect_optimized
.
Got exactly the same results.
And the score is always biased towards the NegativeHalfProfile.
I've run a few thousand examples. This is the statistics I'm seeing:
Frontal | NegProf | NefHProf | PosHProf | PosProf | |
---|---|---|---|---|---|
Mean score | 1.076 | 1.363 | 2.995 | 1.818 | -1.523 |
StdDev score | 0.064 | 0.046 | 0.065 | 0.062 | 0.049 |
Hi @mousomer,
thank you for reporting this. The values you show seem to be a bit suspicious, I would expect highest score for the frontal views, since those have the highest number of landmarks. Maybe there is a bias term missing in the code sample. I will check it soon and come back with an answer.
@uricamic I've tested a few of the images. Seems that when translating scores to Z-score (subtracting means, dividing by standard deviation), the best z-score does yield the best model match. I need to verify this on a large batch of images.
@uricamic I did testing with NIST face set 18 - which has right and left face profiles: https://catalog.data.gov/dataset/nist-mugshot-identification-database-mid-nist-special-database-18
The scoring is still bad. Even when translating into z-scores or tail scores, the results are not good. So, basically, I need some external reference software to decide on the right model (frontal, R/L profile or half profile).
Hi @mousomer,
no z-score translation should be needed. I will try to check on the database you mention and share the code with you. I hope I can manage it within a week, cannot guarantee that though.
I'm adding a sniplet of the code I'm using – in case there's a problem with it.
From: Michal Uřičář [mailto:notifications@github.com] Sent: Tuesday, June 20, 2017 9:02 AM To: uricamic/clandmark clandmark@noreply.github.com Cc: Omer Moussaffi mousomer@gmail.com; Mention mention@noreply.github.com Subject: Re: [uricamic/clandmark] Multi-view face landmark extraction (#38)
Hi @mousomer https://github.com/mousomer ,
no z-score translation should be needed. I will try to check on the database you mention and share the code with you. I hope I can manage it within a week, cannot guarantee that though.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/uricamic/clandmark/issues/38#issuecomment-309653150 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABLqJHt3l9nIESklnN42M6oSQZ1W6NLhks5sF2BQgaJpZM4HTbTk . https://github.com/notifications/beacon/ABLqJGuSHu-k1SIT37KBnrOwkCJNe83Oks5sF2BQgaJpZM4HTbTk.gif
inline void run_model_fromPool(Flandmark model, CFeaturePool c_featuresPool, facial_features_clandmark &ff_landmark, cimg_library::CImg
// Run the Clandmark model detector c_featuresPool->updateNFmipmap(c_featuresPool->getWidth(), c_featuresPool->getHeight(), model->getNF(frameCImg, ff_landmark.bbox_init)->data()); model->detect_optimizedFromPool(ff_landmark.bbox_init); //model->detect_optimized(c_jpg->GetDataY(), c_jpg->Width(), c_jpg->Height(), ff_landmark.bbox_init);
// Get score
ff_landmark.score = model->getScore();
// Get detected landmarks
fl_double_t *c_landmarks = model->getLandmarks();
// Show landmarks
string outCName = baseName + "." + c_extension;
fprintLandmarks(outCName, c_landmarks, model->getLandmarksCount(), false);
// crop faces
if (crop_faces)
{
outCName = baseName + "_" + c_extension;
draw_face(outCName, ff_landmark.index, c_landmarks, model->getLandmarksCount(), ff_landmark.bbox_init);
/ rectangle(frame_gray, Point(bbox[0], bbox[1]), Point(bbox[2], bbox[5]), Scalar(255, 0, 0)); circle(frame_gray, Point(int(landmarks[0]), int(landmarks[1])), 2, Scalar(255, 0, 0), -1); for (int i=2; i < 2flandmark->getLandmarksCount(); i+=2) circle(frame_gray, Point(int(landmarks[i]), int(landmarks[i+1])), 2, Scalar(0, 0, 255), -1); */
}
// Cleanup
if (allocate_model)
delete featuresPoolModel;
}
int main (int argc, char argv[]) { Flandmark CDPM = init_model("CDPM.xml"); Flandmark FDPM = init_model("FDPM.xml"); string FRONT_MODEL = GetRunPath () + "../../" +modelDir + "PART_fixed_JOINTMV_FRONTAL.xml"; Flandmark AFLW_F = Flandmark::getInstanceOf (FRONT_MODEL.c_str ()); CFeaturePool* AFLW_featuresPool = init_feature_pool(AFLW_F); AFLW_F->setNFfeaturesPool(AFLW_featuresPool); cout<<"initialize "<<FRONT_MODEL<<" model "<< AFLW_featuresPool->getWidth() <<" x "<< AFLW_featuresPool->getHeight() <<endl;
Flandmark AFLW_PHP = init_model(AFLW_featuresPool, "PART_fixed_JOINTMV_HALF-PROFILE.xml"); Flandmark AFLW_NHP = init_model(AFLW_featuresPool, "PART_fixedJOINTMV-HALF-PROFILE.xml"); Flandmark AFLW_PP = init_model(AFLW_featuresPool, "PART_fixed_JOINTMV_PROFILE.xml"); Flandmark AFLW_NP = init_model(AFLW_featuresPool, "PART_fixedJOINTMV-PROFILE.xml");
string AFLW_extentions[5] = { "Frontal", "PosHProf", "NegHProf", "PosProf", "NegProf" }; Flandmark FL_JA[5] = {AFLW_F, AFLW_PHP, AFLW_NHP, AFLW_PP, AFLW_NP}; string JDPM_extentions[5] = { "CDPM", "FDPM"}; Flandmark C2F_DPM[2] = {CDPM, FDPM}; c_prof.Add(pf_classesF, 7);
if (!CDPM || !FDPM || !AFLW_F || !AFLW_PHP || !AFLW_NHP || !AFLW_PP || !AFLW_NP ) { cerr << "Couldn't create instance of models. check model path "<<modelDir<< endl; return -1; }
// get input image, crop faces run_faces(AFLW_featuresPool, FL_JA, 5, input_basename, AFLW_extentions, true); // for CDPM // run_faces(CDPM, FDPM, c_org.FoldersProcess[0] + "/" + c_org.c_TaskName);
delete AFLW_featuresPool; delete AFLW_F; delete AFLW_NHP; delete AFLW_NP; delete AFLW_PHP; delete AFLW_PP; delete CDPM; delete FDPM; delete c_jpg;
std::cout<<"job summary:"<<std::endl; return 0;
}
int run_faces(CFeaturePool c_featuresPool, Flandmark land_array[], int land_number, string baseName, string c_extensions[], bool crop_faces = false)
{
cimg_library::CImg
for (int i_land=0; i_land<land_number; ++i_land)
{
for (size_t i_f = 0; i_f < ff_landmarks.size(); ++i_f)
run_model_fromPool(land_array[i_land], c_featuresPool, ff_landmarks[i_f], frameCImg, baseName, c_extensions[i_land], crop_faces, false);
printoutFaces(baseName + "_" + c_extensions[i_land], ff_landmarks);
}
delete frameCImg;
return 0;
}
inline void run_model_fromPool(Flandmark model, CFeaturePool c_featuresPool, facial_features_clandmark &ff_landmark, cimg_library::CImg
// Run the Clandmark model detector c_featuresPool->updateNFmipmap(c_featuresPool->getWidth(), c_featuresPool->getHeight(), model->getNF(frameCImg, ff_landmark.bbox_init)->data()); model->detect_optimizedFromPool(ff_landmark.bbox_init);
// Get score
ff_landmark.score = model->getScore();
// Get detected landmarks
fl_double_t *c_landmarks = model->getLandmarks();
// Show landmarks
string outCName = baseName + "." + c_extension;
fprintLandmarks(outCName, c_landmarks, model->getLandmarksCount(), false);
// crop faces
if (crop_faces)
{
outCName = baseName + "_" + c_extension;
draw_face(outCName, ff_landmark.index, c_landmarks, model->getLandmarksCount(), ff_landmark.bbox_init);
}
// Cleanup
if (allocate_model)
delete featuresPoolModel;
}
The output screenshots seem impressive, especially the multi view landmark extraction, but I can't figure out how to run it from the code you have provided. Please help.