No rectagle is drawn around eyes

[Zumbalamambo]

Though I have turned on Eyes detection, it is not displaying any rectangle around my eyes and also it is not displaying right and left eye texts

[sladomic]

@Zumbalamambo good point. This is not shown using rectangles. If you enable eye detection, then it tries to detect the left eye in the left half of the detected face rectangle and the right eye in the right half. If both eyes could have been detected, the rectangle around the face is drawn. So if you see a rectangle it means both eyes were detected.

[Zumbalamambo]

So Eye detection will improve the accuracy of training?

[sladomic]

@Zumbalamambo It depends. If you use eye detection, then images where not both eyes are detected are ignored. This increases the probability of having "good" images where the face faces towards the camera.

If you want to improve further you need to use EyeAlignment preprocessing. But I only use this for Eigenfaces and Image Reshaping. If you use neural networks, I wouldn't use any preprocessing, since the neural networks are trained to use a raw color image as input

[Zumbalamambo]

ok wonderful. Is there any ways to get confidence score after face recognition?

[sladomic]

See here https://github.com/Qualeams/Android-Face-Recognition-with-Deep-Learning-Test-Framework/issues/14

[Zumbalamambo]

Thanks

[Zumbalamambo]

I tried it in following way

private synchronized double getConfidenceScore(Mat featureVectorToRecognize, Mat defaultFeature) {

        double dotProduct                   = defaultFeature.dot(featureVectorToRecognize);
        double normFeatureVector            = Core.norm(defaultFeature, Core.NORM_L2);
        double normFeatureVectorToRecognize = Core.norm(featureVectorToRecognize, Core.NORM_L2);
        double cosineSimilarity             = dotProduct / (normFeatureVector * normFeatureVectorToRecognize);
        double absoluteCosineSimilarity     = Math.abs(cosineSimilarity);
        Log.i(getClass().getName(), "getMostSimilarStudentIfInThreshold: absoluteCosineSimilarity: " + absoluteCosineSimilarity);
        return absoluteCosineSimilarity;
    }

I called it as follows,

public String recognize(Mat img, String expectedLabel) {
        // Ignore
        img = img.reshape(1, 1);
        // Subtract mean
        img.convertTo(img, CvType.CV_32F);
        Core.subtract(img, Psi, img);
        // Project to subspace
        Mat projected = getFeatureVector(img);
        // Save all points of image for tSNE
        img.convertTo(img, CvType.CV_8U);
        addImage(projected, expectedLabel, true);
        //addImage(projected, expectedLabel);
        Log.d(TAG, "Crashes here " + getConfidenceScore(img, projected));
        Mat distance = new Mat(Omega.rows(), 1, CvType.CV_64FC1);
        for (int i = 0; i < Omega.rows(); i++) {
            double dist = Core.norm(projected.row(0), Omega.row(i), Core.NORM_L2);
            distance.put(i, 0, dist);
        }

        Mat sortedDist = new Mat(Omega.rows(), 1, CvType.CV_8UC1);
        Core.sortIdx(distance, sortedDist, Core.SORT_EVERY_COLUMN + Core.SORT_ASCENDING);
        // Give back the name of the found person
        int index = (int) (sortedDist.get(0, 0)[0]);
        return labelMap.getKey(labelList.get(index));
    }

But it throws the following error,

E/cv::error(): OpenCV Error: Assertion failed (mat.type() == type() && mat.size == size && func != 0) in double cv::Mat::dot(cv::InputArray) const, file /build/master_pack-android/opencv/modules/core/src/matmul.cpp, line 3402 E/org.opencv.core.Mat: Mat::n_1dot() caught cv::Exception: /build/master_pack-android/opencv/modules/core/src/matmul.cpp:3402: error: (-215) mat.type() == type() && mat.size == size && func != 0 in function double cv::Mat::dot(cv::InputArray) const E/AndroidRuntime: FATAL EXCEPTION: Thread-3608 CvException [org.opencv.core.CvException: cv::Exception: /

what am i missing?

[sladomic]

I don't see any call in the code. Which 2 Mats are you passing?

[Zumbalamambo]

Please check the updated code

[sladomic]

Each row in Omega is one feature vector of a training image.

Either you need to get the cosine similarity with each row of Omega and return the highest score

or you need to first get the average feature vector per label (person) and then get the cosine similarity between the input image and each of the average feature vectors.

In the code here https://github.com/literacyapp-org/literacyapp-android/blob/master/app/src/main/java/org/literacyapp/authentication/thread/RecognitionThread.java I also need to iterate through all students and get the cosine similarity between the student mean feature vector and the input image.

[Zumbalamambo]

Being an university student, I have learnt a lot from your code. This code worked like a charm. Thank you so much. However the confidence score varies steeply.

for (int i = 0; i < Omega.rows(); i++) {
            double dist = Core.norm(projected.row(0), Omega.row(i), Core.NORM_L2);
            Log.d(TAG, "Confidence Percentage " + getConfidenceScore(projected, Omega.row(i)));
            distance.put(i, 0, dist);
        }

Also I have commented my understanding for image recognition in the following code. please check if my understanding is proper. Correct me if Im wrong. This is the only thing that Im stuck

  public String recognize(Mat img, String expectedLabel) {
        Log.d(TAG, "Before Reshaping " + img.size() + " - > " + img.dump());
        // Convert the 25 x 25 matrix to 625x1 matrix
        img = img.reshape(1, 1);

        // Subtract mean
        img.convertTo(img, CvType.CV_32F);
        Core.subtract(img, Psi, img);

        // Project the resultant 625x1 matrix to subspace
        Mat projected = getFeatureVector(img);
        // Save all points of image for tSNE
        img.convertTo(img, CvType.CV_8U);

        addImage(projected, expectedLabel, true);

        //Omega is the difference between mean and the individual image being inputted. CV_64FC1 converts vector to matrix .
        Mat distance = new Mat(Omega.rows(), 1, CvType.CV_64FC1);

        //Calculate the eucledian distance between projected image and each row of omega. Finally add it to distance Mat
        for (int i = 0; i < Omega.rows(); i++) {
            double dist = Core.norm(projected.row(0), Omega.row(i), Core.NORM_L2);
            distance.put(i, 0, dist);
        }

        //why do we perform sortIdx and what it does because?

        Mat sortedDist = new Mat(Omega.rows(), 1, CvType.CV_8UC1);
        Core.sortIdx(distance, sortedDist, Core.SORT_EVERY_COLUMN + Core.SORT_ASCENDING);

        //How it is correctly being mapped to the corresponding person?
        // Give back the name of the found person
        int index = (int) (sortedDist.get(0, 0)[0]);
        return labelMap.getKey(labelList.get(index));
    }  

[sladomic]

Hi @Zumbalamambo yes, I guess Eigenfaces is just too bad :) I generally only use the neural networks if I want to get a good accuracy/confidence. But for demo purposes you want to have a faster experience and then you need to use Eigenfaces or Image Reshaping with SVM.

I put some comments into your comments (indicated by >>):

  public String recognize(Mat img, String expectedLabel) {
        Log.d(TAG, "Before Reshaping " + img.size() + " - > " + img.dump());
        // Convert the 25 x 25 matrix to 625x1 matrix
        img = img.reshape(1, 1);

        // Subtract mean
        img.convertTo(img, CvType.CV_32F);
        Core.subtract(img, Psi, img);

        // Project the resultant 625x1 matrix to subspace
        Mat projected = getFeatureVector(img);
        // Save all points of image for tSNE
        img.convertTo(img, CvType.CV_8U);

        addImage(projected, expectedLabel, true);

        //Omega is the difference between mean and the individual image being inputted. CV_64FC1 converts vector to matrix . >> that means, that each image/vector in Omega is the difference to each other image, since the mean image (the similarity to all other images) is subtracted. >> CV_64CFC1 is just the type of the matrix (64 bit float matrix with 1 channel). Here we just allocate memory for a new matrix (distance), which has Omega.rows() rows and 1 column (therefore distance is a vector).
        Mat distance = new Mat(Omega.rows(), 1, CvType.CV_64FC1);

        //Calculate the eucledian distance between projected image and each row of omega. Finally add it to distance Mat >> Each row of Omega is a projected training image. Therefore we are calculating the eucledian distance between the projected input image and all projected training images
        for (int i = 0; i < Omega.rows(); i++) {
            double dist = Core.norm(projected.row(0), Omega.row(i), Core.NORM_L2);
            distance.put(i, 0, dist);
        }

        //why do we perform sortIdx and what it does because?  >> You want to find the nearest distance (which means, that this is the most similar looking face/projected image). With sortIdx you sort the matrix and get back the IDs (row in the Matrix) in the sorted order.

        Mat sortedDist = new Mat(Omega.rows(), 1, CvType.CV_8UC1);
        Core.sortIdx(distance, sortedDist, Core.SORT_EVERY_COLUMN + Core.SORT_ASCENDING);

        //How it is correctly being mapped to the corresponding person? >> You take the nearest face/projected image (sortedDist.get(0, 0)[0]) and in the file label_train you have each label per Omega image. So if you have for example 2 persons, then you have 2 labels (1 and 2). Then with labelMap.getKey(...) you get the Name (for example labelMap.getKey(1) will get you the name of person 1). It know, it could have been done a lot easier, but I just didn't put more effort into Eigenfaces, since it's outdated and only good for learning, but not "production" :)
        // Give back the name of the found person
        int index = (int) (sortedDist.get(0, 0)[0]);
        return labelMap.getKey(labelList.get(index));
    }  

[Zumbalamambo]

ok Thank you so much. I will try with tensorflow/caffe since neural networks might output better results