Unknown people detection

[524c]

Hello @bamos, how are you?

Congratulations on the project.

I wonder if you have plans to implement the detection of people still untrained and display as unknown. I would like to collaborate with the project in this area, but do not know where to start.

Thank you.

[musicfish1973]

I'm also interested in this work, and I've tried using decision function value of SVM in scikit-learn. But I found this SVM implementation can't output (n_samples, n_classes) values for one-vs-others mode. Origianl LIBLINER implementation does work.

[bamos]

Hi, sorry for the delayed response, I'm not actively working on unknown people detection but I'd be excited to look over and give some feedback on code and results so we can add this to the classification and web demos. I think the best starting point would be to create an "unknown" classification benchmark on the LFW that we can use to compare techniques. Let me know if you're still interested and I can provide more details about the baseline and benchmark.

Also @mbektimirov and @dista have asked about classifying unknown users in the past. Do you all have any results on this?

This is technically a duplicate of #67, but let's keep the discussion here and close that one when we find a good technique and add it to the web demo.

-Brandon.

[musicfish1973]

@bamos, I agree with you, I would like to take part in the exciting OpenFace and begin with this work.

[FRCassarino]

@bamos, This functionality is vital for the project I'm working on. As of today I've been working around this problems in various ways, but its increasingly clear that I'm gonna need some at least relatively reliable way to determine if a face is unknown.

So I welcome any suggestion you have as to how to approach this. If I find a decent solution I'll be sure to share it here afterwards.

[Vijayenthiran]

I was playing around with the classifier demo and I am interested in unknown people detection. I have trained the classifier with 25 different people with 13 images of each person. The problem statement is to detect the person who is not among the 25 people (sort of intruder recognition).

When there is a known person standing in front of the camera, the person prediction is correct with the confidence level of around 0.25 - 0.50. So I was expecting that if there is a unknown person standing, the confidence level would drop below 0.25. But it didn't happen that way. The classifier tries to predict the unknown person as one among the 25 people with the confidence level always greater than 0.35.

I tried the different classifiers (Linear SVM, GMM, Radial SVM, DecisionTree; GMM and Decision Tree performed worst) and no fruitful result. I am a beginner in Machine Learning. So any advice would be highly appreciated.

[FedericoRuiz1]

@Vijayenthiran the confidence score is simply not very reliable. The workaround I've been using is to take the last 15 frames and do some math with all the predictions in them. It works relatively well, but not nearly as well as I need.

So I'm now looking for any alternative approaches.

[bamos]

Hi all, apologies for the delays, I'm traveling now and have been for a few weeks. I think a great starting point is to create a benchmark based on the LFW dataset since the LFW contains realistic images that the neural network isn't trained with. This should include the accuracies on known and unknown people.

1. Sort the LFW people by the number of images per person.
2. Set knownAccuracies = [] (empty list), unknownAccuracies = []
3. for exp in 1, ... , 5
   • Training Set: Randomly (but deterministically) sample N images from the first M people
   • Testing Set (of known people): Remaining images from the first M people.
   • Testing Set (of unknown people): Sample P images from the remaining people that aren't in the training set
   • Append accuracies from training and testing on the sets.
4. Output: mean and stdev of the accuracies.

What do you all think of this?

---

Hi @musicfish1973 - would you still like to do the initial implementation of this? You can use my LFW classification experiment as a starting point. See our tech report for a description of the classification experiment.

---

On unknown detection techniques (\cc @Vijayenthiran), my intuition is that a probabilistic technique like GMMs should work well, but I haven't had much success. I've tried most of the techniques @Vijayenthiran mentioned as well as using PCA/LDA before classification to change the feature space. There are some interesting ideas in open set recognition that might work better here. Is anybody interested in further exploring open set recognition?

Also, another idea I've tried that didn't work well but might be worth further exploring is explicitly adding an "unknown" class.

-Brandon.

[musicfish1973]

I benefit much from the discussions above. I'm still focusing on statistical analysis of the output values of LIBLINEAR in one-vs-the-rest mode. I believe softmax may be a better solution although it has a much more expensive training procedure.
Particularly @bamos, begin to learn the things you provided.

[Vijayenthiran]

@bamos Thanks for the update. Your message was helpful in understanding the basics of open set recognition.

I tried creating benchmark with lfw. Please let me know the procedure I followed is correct or not:

I took the following people from the lfw dataset ['Alejandro_Toledo', 'Ariel_Sharon', 'Arnold_Schwarzenegger', 'Colin_Powell', 'Donald_Rumsfeld', 'George_W_Bush', 'Gerhard_Schroeder', 'Gloria_Macapagal_Arroyo', 'Hugo_Chavez', 'Jacques_Chirac', 'Jean_Chretien', 'Jennifer_Capriati', 'John_Ashcroft', 'Junichiro_Koizumi', 'Laura_Bush', 'Lleyton_Hewitt', 'Luiz_Inacio_Lula_da_Silva'] which had large number of images. Among these people I split the first 90% of the image as known people train data and remaining 10% of the image as known people test data.

I trained the different classifiers with the train dataset. Then I tested the classifier with known people test data. For single iterations, following were the accuracies for different classifiers:

Linear SVM: knownAccuracies = [0.975757575758]

GMM: knownAccuracies = [0.127272727273]

Radial SVM: knownAccuracies = [0.975757575758]

Decision Tree: knownAccuracies = [0.818181818182]

Then I took person names starting with alphabet 'A' in the lfw (except Alejandro_Toledo, Ariel_Sharon, Arnold_Schwarzenegger - since they were already in the train set) and used it as unknown people test data.

For single iteration here are the following accuracies for different classifiers:

Linear SVM: unknownAccuracies = [0.0]

GMM: unknownAccuracies = [0.0]

Radial SVM: unknownAccuracies = [0.0]

Decision Tree: unknownAccuracies = [0.0]

unknownAccuracies were zero since the unknown people test image is not in the train image and there is not class as "unknown".

Accuracy was calculated with the following logic: no_of_correct_predictions/total_no_of_predicitions

Is the procedure I am following is correct. If so I will continue doing it for 5 different set of known people data set and share the results here. Also, I tried creating a unknown class by creating a unknown folder with random set of actor/actress images (of about 2000 images). If I run the classifier (for my test data which was mentioned in my previous comment, not the lfw data) with unknown class, for any image input the classifier predict it as unknown. I guess may be due to the large number of images in the unknown class?

[luinstra]

@bamos I was looking over the open set project you linked, and it does sounds interesting, but I think the license agreement of their project would be prohibitive to most use cases that are not strictly research.

I have been considering the 'unknown' class as a possibility as well. In your implementation did you randomly pick an image from many different people to create a sort of average? Or did you have a different approach?

[luinstra]

@Vijayenthiran I have found that the GaussianNB also works well as a classifier. It's biggest down side (for the idea I have been pursuing anyway) is that the prediction probabilities are garbage, according to their docs.

[bamos]

Hi @Vijayenthiran - great! This is a good starting point. A few comments:

I took the following people from the lfw dataset

This works for now, but I prefer sorting the people by the number of images so it's easy to change the number of known people to use.

90% of the image as known people train data and remaining 10% of the image as known people test data

Using a percent instead of a fixed amount here (as I previous said) makes sense.

GMM: knownAccuracies = [0.127272727273]

I'm surprised this is so low, I think this can be improved.

Then I took person names starting with alphabet 'A' in the lfw (except Alejandro_Toledo, Ariel_Sharon, Arnold_Schwarzenegger - since they were already in the train set) and used it as unknown people test data.

I think it's cleaner if you sort the LFW identities and then sample uniformly from the remaining images.

unknownAccuracies were zero since the unknown people test image is not in the train image and there is not class as "unknown".

If you use a probabilistic variants (of SVM and GMM), we could threshold the highest probability to identify unknown users. I think this is a reasonable approach and I wonder if we should also include some "unknown" images as part of the training set.

Accuracy was calculated with the following logic: no_of_correct_predictions/total_no_of_predicitions

This is correct.

Also, I tried creating a unknown class by creating a unknown folder with random set of actor/actress images (of about 2000 images)

So it's easier to modify and share experiments, I don't think we should modify the LFW directory structure and add an known folder. Instead we should be able to easily do it in code.

If I run the classifier (for my test data which was mentioned in my previous comment, not the lfw data) with unknown class, for any image input the classifier predict it as unknown. I guess may be due to the large number of images in the unknown class?

Yes, this happened to me too, but I think some classifiers should be able to overcome this problem. What classifier are you using? I'd try a neural network with 2-3 (small) fully-connected hidden layers first, then potentially a RBF SVM.

[bamos]

Hi @luinstra -

I was looking over the open set project you linked, and it does sounds interesting, but I think the license agreement of their project would be prohibitive to most use cases that are not strictly research.

We can use their code as a quick prototype to see if the technique works well. If it works well, we can implement our own version based on the paper and release it under the Apache 2 license.

I have been considering the 'unknown' class as a possibility as well. In your implementation did you randomly pick an image from many different people to create a sort of average? Or did you have a different approach?

Yes, I randomly sampled from a lot of unknown people. Classifiers will quickly collapse to labeling everything as unknown, but I think we can overcome this with a little more analysis.

[Vijayenthiran]

Thanks for the feedback @bamos. I will work it on this weekend.

Meanwhile I worked on the following comment:

If you use a probabilistic variants (of SVM and GMM), we could threshold the highest probability to identify unknown users.

Instead of thresholding the probability to the highest (since in the lfw dataset, the highest prob was 1.0), I took the average of the all confidences and it came to around 0.9594 with nolearn-DBN classifier. Then I kept this threshold for the unknown people dataset and the unknownAccuracies was 0.929372197309. So I applied this technique to my 25 people dataset (With DBN classifier) and it worked well (reasonable accuracy) in predicting the unknown people.

[FedericoRuiz1]

@Vijayenthiran I'm not sure if I understand. You switched the GMM classifier to the nolearn-DBN classifier, and its able to detect unknown people with 0.92% accuracy?

How did you do this exactly?

clf = DBN( learn_rates = 0.3, learn_rate_decays = 0.9, epochs = 10, verbose = 1)

Is that what you wrote on the classifier.py file?

[Vijayenthiran]

@FedericoRuiz1 Intially I calcuated the average confidence from the known people data set. The avg. confidence came aroung 0.95. So I kept it as threshold for the unknown people data set. Means, if the confidence is less than 0.95, then consider the person as unknown (correct prediction). If the confidence is greater than 0.95, then check if the person name matches with the predicted person name. If so it is a correct prediction (which would not happen) and if the names didn't match then it would be incorrect prediction.

[luinstra]

@Vijayenthiran Are you testing this with the same size data set each time? From what I have seen the confidence value will be reduced as the number of classes grows, because each class gets a small percentage of the total probability. When I run tests on a data set with ~100 classes the highly confident predictions rarely get over the 0.5-0.6 range, IIRC.

[Vijayenthiran]

@luinstra For lfw I used 17 classes of same size for known train and test. In lfw unknown test dataset I used 429 classes. In my own data set there were 25 classes, in which case the average confidence came around 0.55. I haven't tested with 100 classes. If I try, I will let you know my results.

[musicfish1973]

@bamos Through learning the open set recognition I found that rejecting uncertainty is what we should care at first. So I run the code of Meta-recognition. The example data works, but my result data of SVM decision values not. See below: (1)example data in wbl-test.c: double tail[] = {0.74580, 0.67048, 0.65088, 0.64507, 0.64500, 0.64402, 0.64295, 0.64279, 0.64244, 0.64079}; double tail_fit[] = {0.67048, 0.65088, 0.64507, 0.64500, 0.64402, 0.64295, 0.64279, 0.64244, 0.64079}; =>the first item 0.74580 can be correctly regarded as an outlier through Weibull distribution. (2)my data: //FR SVM:-0.9134 0.3909 -0.7918 -0.8268 -0.7458 -0.7586 -->original values //add 0.92 and sorted,values in [0,1] double tail[] = {1.0, 0.1742, 0.1614, 0.1282, 0.0932, 0.0066}; double tail_fit[] = {0.1742, 0.1614, 0.1282, 0.0932, 0.0066}; =>the outlier 0.3909 can't be identified!! What mistakes I have made? Can you and anybody else give me some suggestions?

[FedericoRuiz1]

@Vijayenthiran Why did having more people in the database increase the confidence, rather than decrease it? Am I getting this right? When you had 25 people in the database,avg confidence was around 0.55, but when you had around 500 people the avg confidence was 0.95?

With the default classifiers confidence goes down as you add people to the database.

[Vijayenthiran]

@FedericoRuiz1 I didn't test it with 500 people. I tested with 25 people (each of around 100-200 photos) in lfw. In that case the average confidence was around 0.95 (since it had lot of photos per person). When I tested with my own dataset of 25 people (with around 10 photos per person) the average confidence was close to 0.55. So I guess increasing the no. of photos per person will increase the avg. confidence. I am yet to clean up the code for benchmarking. Will send a PR once it is clean.

[luinstra]

I actually have a method for determining unknown people that is showing some promise. It's not quite where I want it to be yet, but it's a start. The approach I took to get a measure for the uncertainty was to create an ensemble of classifiers, each of which is trained on a random subset of the data for each person. Effectively combining the random forrest concept with classifiers that are not trees. I settled on using the GaussianNB classifier as my base type after testing out several of the others because it is very fast and accurate. Using the _predictproba method is normally not useful for this classifier, but when you average these values in the ensemble you effectively get a voting system that gives you a decent measure of uncertainty. So when the confidence for the top prediction was below 0.95, I give the data some extra scrutiny, and if it's below 0.6 I flag it as unknown.

Using an ensemble size of 15 and a sample ratio of 0.7 I was able to get some decent results. I used the LFW data methods baked into sklearn and set _min_faces_perperson=15. This ends up giving me 96 classes in my data set, and I randomly pick 10 to use as my unknown set.

The table below shows my results of running tests on the test data set (all were in the training data) and the unknown data set (none were in the training data). I ran several iterations of this test and the numbers all come out in a similar range to the ones below. The numbers shown are all percentages of the respective set size. The 'baseline' method represents using highest confidence match from the classifier with no extra verification. The 'filtered' method uses the confidence thresholds I mentioned above and an attempt at some extra verification.

data set	method	Unknown	False Positives	True Positives
test	baseline	0.0	7.42	92.58
test	filtered	8.83	3.18	87.99
unknown	baseline	0.0	100.0	0.0
unknown	filtered	73.83	26.17	0.0

[FedericoRuiz1]

@luinstra So have you got any news on this front? I'd like @bamos or anyone else to pitch in? What does everyone think of this method? Anybody up to implementing it in a more general manner?

[bamos]

I'm still very interested to see how well the open set classification techniques work. My intuition is that it models unknown people detection very well.

@Vijayenthiran can you send your code implementing the unknown people detection benchmark with your baselines in a PR?

-Brandon.

[luinstra]

I have continued to work at this a quite bit and have thoroughly convinced myself that the numbers above are as good as it will get with the current type of classifier being used. I agree with @bamos that an open set classifier is ultimately the way to go here. I will put some work into trying to get that setup and tested.

[luinstra]

I wanted to throw a link to this paper on here for reference. The results seem promising for an open set classifier implementation.

[bamos]

Hi all, I've merged @Vijayenthiran's PR implementing an unknown user detection benchmark into the master branch. We should start using and building on it so that we have a reliable way to compare results and evaluate techniques. @Vijayenthiran sketched the usage information in the PR, #171.

@Vijayenthiran - thanks for implementing this and making some modifications for the PR! I noticed that your example uses the LFW funneled images. I think we should instead be using the raw LFW images on input so the accuracies are comparable to normal images that aren't funneled. Since your code does an alignment, I don't think any changes are necessary here other than using a different input directory.

Also, I've added a section to the OpenFace FAQ on unknown person detection. For now I've linked to this page.

-Brandon.

[Vijayenthiran]

Thanks @bamos, I ran my example on raw LFW images and it worked without any issue. Here's a quick test I ran for known people benchmarking to test the classifiers accuracy on raw LFW and funneled LFW images.

Classifier	Raw LFW Accuracy	Funneled LFW Accuracy
LinearSvm	0.73	0.97
GMM	0.0	0.075
RadialSvm	0.73	0.98
DecisionTree	0.67	0.81
GaussianNB	0.71	0.975
nolearn-DBN	0.715	0.96

To 20 classes with maximum number of images were used for this test.

[bamos]

Right, these are the results that I'd expect. Funneling the LFW images gives better performance since it's a better alignment technique than what we're currently doing. The results on the raw images have nearly the same relative accuracies compared to the funneled images. Improvements here should carry through to future OpenFace versions that potentially use an improved alignment technique.

If anybody's interested, I've tried this code that I think is similar to deep funneling and it doesn't work well in general and it performed much worse than the technique we're currently using. The differences between raw vs funneled here indicate that there may be some cases where it performs better, but I haven't looked into it. The running time will also be longer than what we're currently using.

-Brandon.

[Vijayenthiran]

Hi @bamos, I just compared the aligned images output from raw lfw and funneled lfw images side by side. It looks exactly the same. Which means the alignment technique we are using produces same result for both raw and funneled images. But why the accuracies vary? Am I misunderstanding something here?

P.S.: Shall we move this discussion to another thread and concentrate on unknown classification in this one?

[bamos]

The images look similar, but they're not identical. Compare the OpenFace features between the two images. Minor differences are causing differences in the representation space. You're running the raw and deep funneled images through OpenFace alignment, which is normalizing and cropping them in the same way. However the funneled ones are subtly transformed in a "better" way.

[Vijayenthiran]

I was wondering if we can use One Class SVM, in addition to the exisiting classifier. Have anyone tried this already?

[luinstra]

@Vijayenthiran I did some testing with that classifier a couple of months back and did not have good results with it.

[FedericoRuiz1]

@luinstra Any new insights or any progress regarding unknown people detection using an Open Set Classifier?

[luinstra]

@FedericoRuiz1 Sorry, some things came up and I haven't been able to work on this for a couple of weeks. I will pick it back up in the next few days and hopefully have something to share soon.

[FedericoRuiz1]

@luinstra , just saw your reply. Great, keep us on the loop :)

[ElivisC]

Is it possible to merge the unknown classifier into the WebDemo?

[drravenstone]

Hello, I would like to add the existing unknown people detection functionality to the web demo, since I need it in the openface docker container for a project I am currently doing. Because I am pretty new to github, however, I would like to know what would be the correct procedure to contribute to your project, so as not to mess anything up. Should I fork your repository and then create a new branch, make changes and issue a pull request in the end? Thanks a lot

[bamos]

Hi @drravenstone - thank you for your interest in contributing to OpenFace. Forking the repo and sending in a PR for a new branch with your modifications sounds great! I'll comment with anything else that needs to be changed on your PR. By the way, it's not clear to me yet the best way of recognizing unknown people, what technique do you plan on implementing or do the rest of you suggest here? DBNs?

-Brandon.

[wangg12]

Maybe the recent idea of open set recognition could address the unknown people problem? I'm not sure about it. Just put it here for reference.

[tuna123]

@drravenstone which is the 'existing' unknown people detection functionality that you are referring to? Do you mean the benchmark that @Vijayenthiran did or are you referring to something that you worked on and are planning on adding it to the web demo?

[Vijayenthiran]

@tuna123 I haven't worked on the web demo. I tried One Class SVM, but it produced bad results. Haven't focused on unknown classifier for few weeks as I am busy with other works. Will post updates if I have any progress.

[pythonanonuser]

Is there a crude way to do this via scaling the probability threshold depending on how many classes we've trained. I know people have said the confidence from the SVM varies depending on how many classes you have. So perhaps a confidence 90% distinguishes unknown from trained if you've only trained 2 people, and scale it down to 50% if you have 100+ people trained? Just a a thought.

[dmandreev]

Hello! I use simpler way, without SVM. I started with empty list of descriptors, add first face, then compare descriptors of new faces by cosine distance. If value of similarity is lower than threshold, I assume that this face is new, and add it to list. Threshold is calculated statistically from LFW, and for my network it is 0,36 from [-1.0;1.0]. Face is tracked from frame to frame by detector, and I simply calculate centroid from accumulated descriptors of person, waiting for 7 photos. If number of accumulated descriptors lower than 5, I did not add centroid descriptor as unique person. It seems that this approach can achieve decent results, with real-time performance on I7/GTX980. Error rate is around 15%. Check this video https://www.youtube.com/watch?v=H5PUbPhdnVM for example, there I’m trying to count all unique faces.

[lilac]

Hi, I was working on this problem as well. I came up with a naive solution, though I am not sure it is the right way to go. The solution is easy, let the classifier decides a class, then compute the distance of image (feature) to a all images of that class. If the average distance is larger than a threshold, then mark it as unknown. The accuracy evaluation result shows that 0.99 is a good threshold for positive match, so it could be the threshold for unknown class as well. What do you think?

[pythonanonuser]

@lilac I've thought about this solution but haven't gotten around to implementing it. How accurate is it? I was wondering if it might be too slow to run in real time as well since you have to compare against every rep of each person.

[dmandreev]

@lilac in such case, network need good discriminative properties, one of ideas - to approximate center loss during training http://ydwen.github.io/papers/WenECCV16.pdf @pythonanonuser to my experience, comparing one million of R256 float vectors by cosine distance is not an issue. Hardware like NVidia 820M can do it in one second. GTX980 can crunch 25 millions at same time.

[AdamMiltonBarker]

Actually quite happy in a way to have found this issue. I originally developed a version of Inception V3 on a Raspberry Pi which was doing great until I realised it was identifying unknowns as knowns. I started to look at other alternatives and came across OpenFace. I decided to build a server using MQTT as communication between our devices, the server and our broker, but also hit the same issue with OpenFace, it feels a lot better to know this is a known issue rather than something I had repeatedly done wrong ;)

Will now continue working on both of the implementations, some good suggestions here that will test out and share my findings here. Has anyone made any good progress on this yet, I have noticed that most unknowns come below 0.94 so I am going to set a threshold but I know this will not be a permanent solution as the classes grow etc.

[pythonanonuser]

@AdamMiltonBarker I haven't found the euclidean distance or confidence score outputted by SVM to be a reliable metric for unknown face detection. It varies a lot depending on many factors such as how many people you have trained, how many training images you have per person, etc. Keep us posted on your progress.

[AdamMiltonBarker]

Hi @pythonanonuser thanks for the reply so far have trained with 15 images on two classes, that part is fine and the threshold for now is working but like you say it will change with the amount of classes and our solution will allow people to train as many people as they like so a solution will definitely be something I will work with you guys on.