search

cmusatyalab / openface

Face recognition with deep neural networks.
http://cmusatyalab.github.io/openface/
Apache License 2.0
15.1k stars 3.6k forks source link

All test cases fail, when running 'run-tests.sh' after installation. Test cases dont match. #248

Closed purudpd closed 6 years ago

purudpd commented 7 years ago

All of my test cases fail after successfully installing open-face and all of its dependencies as given http://cmusatyalab.github.io/openface/setup/

I had installed the latest version of both Open-CV and DLIB. OpenCV -> ver_3.2.0-dev DLIB -> ver_19.4.0

Please see this line in output below. UserWarning: Trying to unpickle estimator SVC from version pre-0.18 when using version 0.18.1. This might lead to breaking code or invalid results. Use at your own risk. (Could the issue be related to the sklearn library version?)

Here is my output from the tests.

purusharth@purusharth-Lenovo-G40-70:~/openface$ ./run-tests.sh 
tests.openface_api_tests.test_pipeline ... FAIL
tests.openface_batch_represent_tests.test_batch_represent ... FAIL
tests.openface_demo_tests.test_compare_demo ... FAIL
tests.openface_demo_tests.test_classification_demo_pretrained ... FAIL
tests.openface_demo_tests.test_classification_demo_pretrained_multi ... FAIL
tests.openface_demo_tests.test_classification_demo_training ... ok
tests.openface_neural_net_training_tests.test_dnn_training ... ok

======================================================================
FAIL: tests.openface_api_tests.test_pipeline
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/purusharth/openface/tests/openface_api_tests.py", line 55, in test_pipeline
    assert bb.left() == 341
AssertionError

======================================================================
FAIL: tests.openface_batch_represent_tests.test_batch_represent
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/purusharth/openface/tests/openface_batch_represent_tests.py", line 89, in test_batch_represent
    assert np.isclose(cosDist, 0.1568, atol=1e-4)
AssertionError: 
-------------------- >> begin captured stdout << ---------------------
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0001.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0003.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0003.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0010.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0005.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0004.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0002.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0011.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0005.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0009.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0007.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0012.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0001.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0004.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0011.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0010.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0002.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0007.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0008.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0009.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0012.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0006.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0010.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0008.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0009.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0005.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0004.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0003.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0002.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0008.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0011.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0007.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0006.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0006.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0001.jpg ===

=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0010.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0001.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0003.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0004.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0006.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0001.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0004.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0011.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0009.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0008.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0005.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0009.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0012.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0010.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0011.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0001.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0010.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0002.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0005.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0005.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0007.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0012.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0008.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0002.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0007.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0011.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0002.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0004.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0009.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0006.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0003.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Adrien_Brody/Adrien_Brody_0008.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0006.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Ann_Veneman/Ann_Veneman_0007.jpg ===
=== /home/purusharth/openface/data/lfw-subset/raw/Anna_Kournikova/Anna_Kournikova_0003.jpg ===

{
  data : "/tmp/OpenFaceBatchRep-0eY_Qv/aligned"
  imgDim : 96
  model : "/home/purusharth/openface/models/openface/nn4.small2.v1.t7"
  device : 1
  outDir : "/tmp/OpenFaceBatchRep-0eY_Qv/reps"
  cache : false
  cuda : false
  batchSize : 50
}
/tmp/OpenFaceBatchRep-0eY_Qv/aligned    
cache lotation:     /tmp/OpenFaceBatchRep-0eY_Qv/aligned/cache.t7   
Creating metadata for cache.    
{
  sampleSize : 
    {
      1 : 3
      2 : 96
      3 : 96
    }
  split : 0
  verbose : true
  paths : 
    {
      1 : "/tmp/OpenFaceBatchRep-0eY_Qv/aligned"
    }
  samplingMode : "balanced"
  loadSize : 
    {
      1 : 3
      2 : 96
      3 : 96
    }
}
running "find" on each class directory, and concatenate all those filenames into a single file containing all image paths for a given class 
now combine all the files to a single large file    
load the large concatenated list of sample paths to self.imagePath  
34 samples found..... 0/34 .....................] ETA: 0ms | Step: 0ms          
Updating classList and imageClass appropriately 
 [=================== 3/3 =====================>] Tot: 51ms | Step: 17ms        
Cleaning up temporary files 
Splitting training and test sets to a ratio of 0/100    
nImgs:  34  
Represent: 34/34    

('brody1:', array([-0.02, -0.11, -0.13,  0.06,  0.05,  0.15,  0.01, -0.05,  0.04,
        0.02,  0.08,  0.06,  0.02,  0.1 ,  0.02,  0.03, -0.02,  0.1 ,
        0.08, -0.12,  0.09, -0.02,  0.07, -0.06,  0.09,  0.03, -0.  ,
       -0.15, -0.06,  0.05,  0.16,  0.06,  0.01, -0.09,  0.02,  0.12,
       -0.15, -0.06, -0.  ,  0.05,  0.05,  0.1 , -0.1 ,  0.18, -0.16,
       -0.16, -0.01, -0.09,  0.11, -0.17,  0.05, -0.05, -0.01, -0.06,
       -0.03,  0.12,  0.01,  0.16,  0.05,  0.05, -0.14, -0.11,  0.14,
       -0.09,  0.17, -0.05,  0.  ,  0.04,  0.04,  0.19,  0.09,  0.18,
        0.02, -0.15, -0.12, -0.01, -0.09, -0.04, -0.2 , -0.04, -0.07,
        0.03,  0.04, -0.01, -0.14, -0.05,  0.14, -0.05, -0.09, -0.15,
       -0.08, -0.01, -0.08, -0.11,  0.09, -0.02, -0.04,  0.05, -0.02,
        0.15, -0.05,  0.  ,  0.07, -0.07, -0.01,  0.01, -0.03, -0.1 ,
        0.01, -0.01, -0.09, -0.05,  0.  , -0.09,  0.11,  0.03,  0.07,
       -0.06,  0.07,  0.06,  0.1 , -0.11, -0.02, -0.04,  0.01,  0.06,
        0.01,  0.21]))
('brody2:', array([ 0.02, -0.09, -0.19,  0.08,  0.01,  0.07, -0.01, -0.04, -0.01,
        0.04,  0.14,  0.03,  0.05,  0.15, -0.01,  0.03,  0.1 ,  0.08,
        0.18, -0.05,  0.1 , -0.05, -0.01, -0.06,  0.08,  0.07,  0.01,
       -0.12, -0.02,  0.12,  0.09, -0.01,  0.02, -0.17,  0.02,  0.13,
       -0.12, -0.04,  0.04,  0.02, -0.06,  0.17, -0.09,  0.18, -0.1 ,
       -0.11,  0.03, -0.1 ,  0.11, -0.21,  0.05, -0.  , -0.04, -0.05,
       -0.01,  0.09, -0.04,  0.13,  0.05,  0.05, -0.07, -0.15,  0.15,
       -0.09,  0.15, -0.1 ,  0.04,  0.02,  0.1 ,  0.08,  0.13,  0.15,
        0.04, -0.19, -0.14, -0.01, -0.07,  0.04, -0.15, -0.02, -0.07,
        0.03, -0.  , -0.02, -0.08, -0.04,  0.09, -0.07, -0.02, -0.19,
       -0.13,  0.01, -0.07, -0.07,  0.11, -0.1 ,  0.03,  0.  , -0.04,
        0.11, -0.08,  0.07,  0.09, -0.11, -0.1 ,  0.07,  0.04, -0.11,
        0.04, -0.  , -0.08, -0.13, -0.02, -0.06,  0.08,  0.03,  0.05,
       -0.1 , -0.01, -0.03, -0.03, -0.09, -0.01, -0.02,  0.04,  0.07,
        0.03,  0.17]))
('cosDist:', 0.13904788999160334)

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: tests.openface_demo_tests.test_compare_demo
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/purusharth/openface/tests/openface_demo_tests.py", line 38, in test_compare_demo
    assert "0.763" in out
AssertionError: 
-------------------- >> begin captured stdout << ---------------------
Comparing /home/purusharth/openface/images/examples/lennon-1.jpg with /home/purusharth/openface/images/examples/lennon-2.jpg.
  + Squared l2 distance between representations: 0.782

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: tests.openface_demo_tests.test_classification_demo_pretrained
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/purusharth/openface/tests/openface_demo_tests.py", line 51, in test_classification_demo_pretrained
    assert "Predict SteveCarell with 0.97 confidence." in out
AssertionError: 
-------------------- >> begin captured stdout << ---------------------

=== /home/purusharth/openface/images/examples/carell.jpg ===
Predict SteveCarell with 0.99 confidence.

/usr/local/lib/python2.7/dist-packages/sklearn/lda.py:6: DeprecationWarning: lda.LDA has been moved to discriminant_analysis.LinearDiscriminantAnalysis in 0.17 and will be removed in 0.19
  "in 0.17 and will be removed in 0.19", DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.py:43: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.
  DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator LabelEncoder from version pre-0.18 when using version 0.18.1. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator SVC from version pre-0.18 when using version 0.18.1. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator GridSearchCV from version pre-0.18 when using version 0.18.1. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: tests.openface_demo_tests.test_classification_demo_pretrained_multi
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/purusharth/openface/tests/openface_demo_tests.py", line 64, in test_classification_demo_pretrained_multi
    assert "Predict EvaLongoria @ x=91 with 0.99 confidence." in out
AssertionError: 
-------------------- >> begin captured stdout << ---------------------

=== /home/purusharth/openface/images/examples/longoria-cooper.jpg ===
List of faces in image from left to right
Predict EvaLongoria @ x=92 with 0.99 confidence.
Predict BradleyCooper @ x=192 with 0.99 confidence.

/usr/local/lib/python2.7/dist-packages/sklearn/lda.py:6: DeprecationWarning: lda.LDA has been moved to discriminant_analysis.LinearDiscriminantAnalysis in 0.17 and will be removed in 0.19
  "in 0.17 and will be removed in 0.19", DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.py:43: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.
  DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator LabelEncoder from version pre-0.18 when using version 0.18.1. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator SVC from version pre-0.18 when using version 0.18.1. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator GridSearchCV from version pre-0.18 when using version 0.18.1. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 7 tests in 120.861s

FAILED (failures=5)
qacollective commented 7 years ago

Hi @purudpd , I got very similar test output to yours and have concluded that the expected results for test cases probably need to be adjusted because the updated underlying libraries (probably dlib?) are producing slightly different results. I didn't feel comfortable doing this in my pull request so have left @bamos to look at it, being his project - that seems to be his prerogative.

See #172 and associated pull request for more detail.