test_run_kfold_cross_validation_with_list_input fails on Fedora 23

[tdaede]

FAIL: test_run_kfold_cross_validation_with_list_input (cross_validation_test.FeatureCrossValidationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thomas/sandbox/vmaf/python/test/cross_validation_test.py", line 92, in test_run_kfold_cross_validation_with_list_input
    self.assertAlmostEquals(output['aggr_stats']['SRCC'], 0.90636761259756715, places=4)
AssertionError: 0.90647090523758211 != 0.9063676125975672 within 4 places

The value is off just a little too much for places=4. How much variance is expected here?

[li-zhi]

Hi Tdaede, there could be several possibilities: 1) If the numpy/scipy/sklearn are not up-to-date, there could be numeric differences. You can follow the Prerequiste/Troubleshooting section for this. 2) I also found that there could be numeric difference across platforms. I have not tried on Fedora. I could definitely look into a Fedora to see if the result makes sense or not. Is this the only failed test case?

[haiqianw]

Hi Zhi,

I am Haiqiang Wang from USC. Currently I work with Ioannis on the project.

I pulled it from Git yesterday. All the prerequisite (numpy, scipy, sklearn) were installed with correct version, but I encountered the same FAIL as Thomas, it is the only failure that happened. My platform is Mac pro El Capitan 10.11.

Best, Haiqiang

On Tue, Jun 7, 2016 at 12:51 PM, Zhi Li notifications@github.com wrote:

Hi Tdaede, there could be several possibilities: 1) If the numpy/scipy/sklearn are not up-to-date, there could be numeric differences. You can follow the Prerequiste/Troubleshooting section for this. 2) I also found that there could be numeric difference across platforms. I have not tried on Fedora. I could definitely look into a Fedora to see if the result makes sense or not. Is this the only failed test case?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Netflix_vmaf_issues_1-23issuecomment-2D224394049&d=DQMCaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=6qctLIgM3vp75AehAaBacw&m=0D2L7TG714unv2aRMQ1mksIhrhdZSkaCoeQn1Soa3VM&s=kNUWKEaKIY_snd2-PEmxKb3rUQk54dQQwjiXldrVZxc&e=, or mute the thread https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe_APd647kdkDuC8gDqMe5m9nUevQrxev9dks5qJcvZgaJpZM4IwRqm&d=DQMCaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=6qctLIgM3vp75AehAaBacw&m=0D2L7TG714unv2aRMQ1mksIhrhdZSkaCoeQn1Soa3VM&s=WXXvR3fI2XRujoWs-nkLP9kwCmE2ibmUMR_4UVPWSOE&e= .

[tdaede]

I realized that Fedora 23's versions of all the packages were out of date from those listed. I made a virtualenv and installed the versions listed, but still got the exact same result.

Maybe it's a difference in BLAS implementation or something? I'm using atlas 3.10.2.

[li-zhi]

Hi Haiqiang, just want to make sure you were directly building in Mac OS, not on a Fedora VM, right?

[haiqianw]

Yes, directly in Mac OS.

[li-zhi]

@tdaede: I was able to reproduce your issue on Fedora 23 (I noticed the atlas version was atlas-3.10.2-6.fc23.x86_64 when installing numpy). After upgrading numpy/scipy/sklearn/matplotlib/pandas (pip install --upgrade xxx), the issue went away.

@haiqianw: I tried it on El Capitan, but was unable to reproduce it.

I suspect it is still issue of package out of date. When you check the package version, please try to use:

python -c 'import numpy as pkg; print pkg.version; print pkg.file' python -c 'import scipy as pkg; print pkg.version; print pkg.file' python -c 'import matplotlib as pkg; print pkg.version; print pkg.file' python -c 'import pandas as pkg; print pkg.version; print pkg.file' python -c 'import sklearn as pkg; print pkg.version; print pkg.file'

instead of 'pip freeze'. Sometimes they can give you different results.

[haiqianw]

python -c 'import numpy as pkg; print pkg.version; print pkg.file'

<module 'numpy.version' from '/Users/Haiqiang/anaconda/lib/python2.7/site-packages/numpy/version.pyc'> Traceback (most recent call last): File "<string>", line 1, in <module> AttributeError: 'module' object has no attribute 'file'

If I check the package version following README 1.10.4 /Users/Haiqiang/anaconda/lib/python2.7/site-packages/numpy/__init__.pyc 0.17.0 /Users/Haiqiang/anaconda/lib/python2.7/site-packages/scipy/__init__.pyc 1.5.1 /Users/Haiqiang/anaconda/lib/python2.7/site-packages/matplotlib/__init__.pyc 0.18.0 /Users/Haiqiang/anaconda/lib/python2.7/site-packages/pandas/__init__.pyc 0.17.1 /Users/Haiqiang/anaconda/lib/python2.7/site-packages/sklearn/__init__.pyc

The only difference with README is about pandas, which required >=0.17.1

[li-zhi]

@haiqianw: from your output, I realize that you are using the anaconda installation. It's an known issue to me that some test cases will fail under anaconda (it seems to be an artifact with their bake). I would recommend not to use anaconda but simply use pip.

[tdaede]

In a Fedora 23 VM with blas and lapack but no atlas, all unit tests pass. So it seems like this is due to one of the dependencies using atlas instead. I'll walk through each one and see if I can find the offending package.

[li-zhi]

Thanks! Let me know if you find anything and I'll put some notes in the troubleshoot section.