search

cmusatyalab / openface

Face recognition with deep neural networks.
http://cmusatyalab.github.io/openface/
Apache License 2.0
15.09k stars 3.6k forks source link

Classifier cannot distinguish unknown people #167

Closed luonay closed 8 years ago

luonay commented 8 years ago

I was expecting that a trained network shall be able to distinguish an unseen person from the ones who have occurred in the training set in the form of outputting a low confidence, as demonstrated by the Lennon example in Demo3.

However when I run the script with a picture of my friend who looks very different from any of the celebrities included in the training set: ./demos/classifier.py infer ./models/openface/celeb-classifier.nn4.small2.v1.pkl images/examples/PICTURE_OF_MY_FRIEND The classifier outputs a very high confidence with a wrong classification: Predict AmericaFerrera with 0.902 confidence.

This inability to effectively distinguish unseen people is a usual behaviour of the classifier than a coincidence. I have trained my own classifier with new training sets using the scripts provided in demo3 and it very often predicts an unseen person with very high confidence.

I think this problem might be caused by the ineffectiveness of SVM classifier in detecting novelty. In the event of binary classification of A vs B, if we input a sample from class C, which is on the B side of the decision boundary but far away from B in the direction opposite to the decision boundary, the classifier will still predict B with a very high confidence.

I tried to solve this problem by using the OneClassSVM provided by sklearn.svm, hoping that it could detect novelty in the testing samples. However the result is not very satisfactory.

Any idea on how to solve this problem? Thanks!

escouflaire commented 8 years ago

Hi @Luna86,

You can find some ideas and current progress to solve face recognition for unknown people at #144

bamos commented 8 years ago

Hi @Luna86 - we're on the brink of solving this in #144! Let's continue this discussion there. (And thanks for the response @escouflaire)

I think Open Set Recognition is a great way to model unknown person detection and I think it will perform well. There is some open set recognition source code that you can try if you're interested. All of us in that issue will be very interested in hearing your results. @Vijayenthiran has written some benchmark code for unknown people detection that we'll merge into OpenFace's master branch soon that you can use to try out techniques: https://github.com/Vijayenthiran/openface/blob/classification-unknown/evaluation/lfw-classification-unknown.py

-Brandon.