Using dlib with opencv to make "real time" face recognition: conflict between the two librairies?

Toquet commented 6 years ago

Hello, I am currently working on a project (in c++) that aims at doing face recognition on images coming from a webcam. In order to do so, I use the opencv library which seems to cause issues during the dlib face encoding part:

Expected Behavior

The algorithm is as follow:

Detect a face thanks to the dlib face_detector.
Crop the image thanks to the shape predictor (shape_predictor_5_face_landmarks.dat).
Use dlib_face_recognition_resnet_model_v1.dat to transform this image into an array.
Compare this array to ones obtained with known faces in order to find the identity of the person detected.

As mentioned I need to store the arrays of the known persons before doing the recognition with the webcam. This is where strange things happen:

Current Behavior

When I do not open the camera by calling cv::VideoCapture(), everything is working (known faces get always the same array, test pictures are well recognized).
As soon as I declare the cv::VideoCapture() variable, it does not work anymore: known faces get always different arrays and the recognition does not work anymore on the test pictures. I checked everything and I figured out that in both cases the same picture was send to the neural network dlib_face_recognition_resnet_model_v1.dat, but different results are given. Is it possible that the face encoder is not compatible with opencv (especially the cv::VideoCapture function) ?

It may not be really clear with text so I hope the following code will be more explicit:

Steps to Reproduce

Main, Create a recognizer class that I defined and then call the function that should save the arrays of the known faces:

Recognizer *m_recognizer = RecognizerFactory::Get()->CreateRecognizer("facenet");
m_recognizer->training_function("../modified");

Then as soon as I add the following line, the result of the encoding seems to be random: cv::VideoCapture video(0);

training_function (Loop on images present in a folder):


struct dirent * dp;
// Trying to open the folder.
if((dirp = opendir(name.c_str())) == NULL) {
    std::cout << "Error opening " << name << endl;
    return;
}
// Loop on the files that are in the folder.
while ((dp = readdir(dirp)) != NULL) {
    // Do not take into account files that are smaller than 3 characters.
    if(std::string(dp->d_name).size() > 3) {
        // Load the image.
        dlib::matrix<rgb_pixel> image_dlib;
        dlib::load_image(image_dlib, name + std::string("/") + std::string(dp->d_name));
        // Save the embedding.
        ROI location_face(0,0,image_dlib.nr(),image_dlib.nc());
        this->known_embeddings->push_back(face_encoding(image_dlib,location_face));
        cout << this->known_embeddings->at(this->known_embeddings->size()-1)(0) << endl;
        this->known_names->push_back(std::string(dp->d_name).substr(0,std::string(dp->d_name).find(".")));
    }
}
closedir(dirp);

face_encoding function:


deserialize("dlib_face_recognition_resnet_model_v1.dat") >> face_encoder;
std::vector<dlib::matrix<rgb_pixel>> faces;
// Landmarks.
frontal_face_detector detector = get_frontal_face_detector();
for (auto face : detector(img)) {
    auto shape = this->sp(img, face);
    matrix<rgb_pixel> face_chip;
    extract_image_chip(img, get_face_chip_details(shape,150,0.1), face_chip);
    faces.push_back(move(face_chip));
}
// Embedding.
std::vector<dlib::matrix<float,0,1>> face_descriptors = face_encoder(faces); 
// Display (Check if the images are the same everytime I launch the program)
dlib::image_window win(faces.at(0));
win.wait_until_closed();
I just have to print the first value of face_descriptors.at(0) to see that it is always the same for the same image when cv::VideoCapture video(0); is not present in the main function. Otherwise it seems random... I hope I made my issue understable !

Version: dlib: 19.9 or 19.10, opencv: 3.3.1 or 3.4.0
Where did you get dlib: Github repo
Platform: Ubuntu 16.04 LTS
Compiler: gcc 5.4.0

davisking commented 6 years ago

These things (the dlib stuff and openCV tools) have nothing to do with each other. You just have a bug in your code somewhere and it's corrupting the outputs. Maybe you have some kind of memory error. The code you posted has a bunch of raw pointers so that's not helping you. C++ isn't C. You shouldn't have raw pointers or be doing manual memory management.

Toquet commented 6 years ago

I thought it was a memory leak also. However I found an interesting thing: when I run your example program dnn_face_recognition_ex.cpp, everything works fine, the outputs are always the same for the same inputs. However, when I include opencv and I declare a cv::VideoCapture in it, the outputs become random:

cv::VideoCapture video(0);
video.release();

I really believe that there is something strange between the two librairies but I can't find what.

davisking commented 6 years ago

Maybe there is a bug in opencv or you compiled it in some bad way. I don't know. There are no interactions between dlib and opencv though.

Toquet commented 6 years ago

Ok thank you for the answers !

Toquet commented 6 years ago

Just to let you know the end of the story: The issue was caused by the fact that my python libraries were interacting with the c++ ones. I just uninstalled them and reinstalled only the c++ lib and it solved the issue. So you were right! Thank you.

davisking / dlib