guswnsxodlf commented 7 years ago

I set the environment with docker.

And I followed the demo command at Minimal Working Example to Extract Features.

It went well, and I trained features and tried it.
./demos/classifier.py train classify-test/features
./demos/classifier.py infer classify-test/features/classifier.pkl classify-test/raw/lennon/lennon-1.jpg

But the result was

/usr/local/lib/python2.7/dist-packages/sklearn/lda.py:4: DeprecationWarning: lda.LDA has been moved to discriminant_analysis.LinearDiscriminantAnalysis in 0.17 and will be removed in 0.19
  "in 0.17 and will be removed in 0.19", DeprecationWarning)

=== classify-test/raw/lennon/lennon-1.jpg ===
Predict clapton with 1.00 confidence.

Why the result is clapton? Is it right?

bamos commented 7 years ago

Can you try running our tests with run-tests.sh and send the output? (It will give you an error the first time you run it with some instructions of files you need to download.) I have a test in there that checks if the classifier demo works as expected.

-Brandon.

guswnsxodlf commented 7 years ago

Oh, there are 3 fails..

root@5560bfc87a3b:~/openface# ./run-tests.sh
tests.openface_api_tests.test_pipeline ... ok
tests.openface_batch_represent_tests.test_batch_represent ... FAIL
tests.openface_demo_tests.test_compare_demo ... ok
tests.openface_demo_tests.test_classification_demo_pretrained ... ok
tests.openface_demo_tests.test_classification_demo_pretrained_multi ... ok
tests.openface_demo_tests.test_classification_demo_training ... FAIL
tests.openface_neural_net_training_tests.test_dnn_training ... FAIL

======================================================================
FAIL: tests.openface_batch_represent_tests.test_batch_represent
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/root/openface/tests/openface_batch_represent_tests.py", line 39, in test_batch_represent
    lfwSubset), "Get lfw-subset by running ./data/download-lfw-subset.sh"
AssertionError: Get lfw-subset by running ./data/download-lfw-subset.sh

======================================================================
FAIL: tests.openface_demo_tests.test_classification_demo_training
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/root/openface/tests/openface_demo_tests.py", line 69, in test_classification_demo_training
    assert os.path.isdir(lfwSubset), "Get lfw-subset by running ./data/download-lfw-subset.sh"
AssertionError: Get lfw-subset by running ./data/download-lfw-subset.sh

======================================================================
FAIL: tests.openface_neural_net_training_tests.test_dnn_training
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/root/openface/tests/openface_neural_net_training_tests.py", line 37, in test_dnn_training
    lfwSubset), "Get lfw-subset by running ./data/download-lfw-subset.sh"
AssertionError: Get lfw-subset by running ./data/download-lfw-subset.sh

----------------------------------------------------------------------
Ran 7 tests in 10.550s

FAILED (failures=3)

Thank you for your answer!

guswnsxodlf commented 7 years ago

I commanded ./data/download-lfw-subset.sh as the assertionError and re-tested.

root@5560bfc87a3b:~/openface# ./run-tests.sh
tests.openface_api_tests.test_pipeline ... ok
tests.openface_batch_represent_tests.test_batch_represent ... ok
tests.openface_demo_tests.test_compare_demo ... ok
tests.openface_demo_tests.test_classification_demo_pretrained ... ok
tests.openface_demo_tests.test_classification_demo_pretrained_multi ... ok
tests.openface_demo_tests.test_classification_demo_training ... ok
tests.openface_neural_net_training_tests.test_dnn_training ... ok

----------------------------------------------------------------------
Ran 7 tests in 224.863s

OK

All passed and I trained images again, but the result is still wrong..

bamos commented 7 years ago

Thanks! At least that looks good, I'm not sure what's going on in your case. Let me know if you find out anything else.

-Brandon.

guswnsxodlf commented 7 years ago

Thanks :)

RodrigoPetter commented 7 years ago

I did the same as @guswnsxodlf and here are my result:

=== images/examples/carell.jpg ===
Predict lennon with 0.98 confidence.

=== images/examples/adams.jpg ===
Predict lennon with 0.95 confidence.

=== images/examples/lennon-1.jpg ===
Predict clapton with 1.00 confidence.

=== images/examples/lennon-2.jpg ===
Predict clapton with 1.00 confidence.

As said in http://cmusatyalab.github.io/openface/demo-3-classifier/#classifying-new-images the "celeb-classifier.nn4.small2.v1.pkl" was trained with 6000 images from 10 persons (600 per person).

Maybe the problem is the lower number of images used in the training process?

guswnsxodlf commented 7 years ago

Yep I think so :/

RodrigoPetter commented 7 years ago

Just an update... I was doing some tests and got the following result:

Using one image from 6 different people to generate the "classifier.pkl" the ./demos/classifier.py infer ... was not able to identify correctly anyone in the images.
I increased the number of image to generate the "classifier.pkl" one by one and I started to get good predictions around 6 images per person.
But now comes the tricky part that I don't know why it happened: I just copied and pasted the exacly same 6 images per person, getting a total of 12 per person but with duplicated images, and the result of the ./demos/classifier.py infer ... increassed about 15% in the confidence.

Another worrisome behaivor that i noticed is that running the ./demos/classifier.py train ... on the same features folder(without change the images), several times, without erasing the previous results, the "classifier.pkl" generated by the training produce a prediction with a different confidence every time you use the new classifier.

guswnsxodlf commented 7 years ago

That's weird ..

jaskaran1 commented 7 years ago

@guswnsxodlf @bamos

The problem seems to be with scikit learn's predict_proba. Probabilities are predicted from SVM using Platt Scaling. For less number of samples per class it gives wrong results because of platt scaling.

250

Please look into this issue for more details about the problem.

shivani-vemula commented 7 years ago

while i was running this

./demos/classifier.py train ./generated-embeddings/

ian getting this error

Traceback (most recent call last): File “./demos/classifier.py”, line 294, in train(args) File “./demos/classifier.py”, line 114, in train le = LabelEncoder().fit(labels) File “/usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/label.py”, line 108, in fit y = column_or_1d(y, warn=True) File “/usr/local/lib/python3.5/dist-packages/sklearn/utils/validation.py”, line 551, in column_or_1d raise ValueError(“bad input shape {0}”.format(shape)) ValueError: bad input shape ()

please help me in debugging it

cmusatyalab / openface

Creating a Classification Model example seems wrong #234

250