failing Python tests (after fixing some problems)

boegel commented 5 years ago

While running the Essentia Python tests with python ./waf run_python_tests, I ran into a couple of problems.

Context:

GCC 8.2 (compilation flags: -O2 -ftree-vectorize -march=native -fno-math-errno)
Essentia 2.1_beta5
Python 2.7.15 + numpy 1.16.2
FFTW 3.3.8
FFmpeg 4.1.3
libsamplerate 0.1.9
TagLib 1.11.1
Chromaprint 1.4.3
Gaia 2.4.5

First, I had apply a small patch to ensure that $PYTHONPATH doesn't get wiped when running the Python tests (we install numpy in a separate location, and set $PYTHONPATH to tell Python where to find it):

--- essentia-2.1_beta5/wscript.orig 2019-10-01 20:26:06.136279484 +0200
+++ essentia-2.1_beta5/wscript  2019-10-01 20:26:57.286223056 +0200
@@ -338,7 +338,7 @@
     os.system('cp -r src/python/essentia build/python/')
     os.system('cp build/src/python/_essentia*.so build/python/essentia')

-    ret = os.system('PYTHONPATH=build/python %s test/src/unittests/all_tests.py' % sys.executable)
+    ret = os.system('PYTHONPATH=build/python:$PYTHONPATH %s test/src/unittests/all_tests.py' % sys.executable)
     if ret:
         ctx.fatal('failed to run python tests. Check test output')

Then, I had to rename test/src/unittests/io/ to test/src/unittests/iotest/ to avoid this problem with the io.open in numpy getting broken (numpy 1.16.2):

Unable to import NumPy C API from Essentia module. Error code = -1
Traceback (most recent call last):
  File "test/src/unittests/all_tests.py", line 27, in <module>
    import essentia
  File "/tmp/essentia-2.1_beta5/build/python/essentia/__init__.py", line 1, in <module>
    from . import _essentia
  File "/software/SciPy-bundle/2019.03-foss-2019a/lib/python2.7/site-packages/numpy/__init__.py", line 145, in <module>
    from . import lib
  File "/software/SciPy-bundle/2019.03-foss-2019a/lib/python2.7/site-packages/numpy/lib/__init__.py", line 24, in <module>
    from .npyio import *
  File "/software/SciPy-bundle/2019.03-foss-2019a/lib/python2.7/site-packages/numpy/lib/npyio.py", line 14, in <module>
    from ._datasource import DataSource
  File "/software/SciPy-bundle/2019.03-foss-2019a/lib/python2.7/site-packages/numpy/lib/_datasource.py", line 226, in <module>
    _file_openers = _FileOpeners()
  File "/software/SciPy-bundle/2019.03-foss-2019a/lib/python2.7/site-packages/numpy/lib/_datasource.py", line 168, in __init__
    self._file_openers = {None: io.open}
AttributeError: 'module' object has no attribute 'open'

I also had to add a missing __init__.py file in test/src/unittests/audioproblems/ to fix this issue:

Traceback (most recent call last):
  File "test/src/unittests/all_tests.py", line 243, in <module>
    result1 = runTests(getTests(testList, exclude=testExclude))
  File "test/src/unittests/all_tests.py", line 92, in getTests
    importTest(test, strategy)
  File "test/src/unittests/all_tests.py", line 69, in importTest                                                                                                                                                exec(cmd)
  File "<string>", line 1, in <module>
ImportError: No module named audioproblems.test_saturationdetector

With those fixes done, I'm able to run the Python tests, but several of them fail (see detailed output below).

Are these known issues? Are the Python tests supposed to pass? The problems I had to fix seem to suggest these tests aren't run on a regular basis?

======================================================================
ERROR: testAll (unittests.highlevel.test_gaiatransform.TestGaiaTransform)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 70, in testAll
    self.dotest('all')
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 57, in dotest
    transfo, gt = loadTestData(type)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 28, in loadTestData
    transfo = GaiaTransform(history = 'highlevel/svm/test_svm_%s.history' % type)
  File "/tmp/essentia-2.1_beta5/build/python/essentia/standard.py", line 44, in __init__
    self.configure(**kwargs)
  File "/tmp/essentia-2.1_beta5/build/python/essentia/standard.py", line 64, in configure
    self.__configure__(**kwargs)
RuntimeError: Error while configuring GaiaTransform: GaiaTransform: error loading gaia history: Could not open file: highlevel/svm/test_svm_all.history

======================================================================
ERROR: testMultiDimDesc (unittests.highlevel.test_gaiatransform.TestGaiaTransform)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 67, in testMultiDimDesc
    self.dotest('multidimdesc')
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 57, in dotest
    transfo, gt = loadTestData(type)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 28, in loadTestData
    transfo = GaiaTransform(history = 'highlevel/svm/test_svm_%s.history' % type)
  File "/tmp/essentia-2.1_beta5/build/python/essentia/standard.py", line 44, in __init__
    self.configure(**kwargs)
  File "/tmp/essentia-2.1_beta5/build/python/essentia/standard.py", line 64, in configure
    self.__configure__(**kwargs)
RuntimeError: Error while configuring GaiaTransform: GaiaTransform: error loading gaia history: Could not open file: highlevel/svm/test_svm_multidimdesc.history

======================================================================
ERROR: testProbability (unittests.highlevel.test_gaiatransform.TestGaiaTransform)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 77, in testProbability
    transfo, gt = loadTestData('probability')
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 28, in loadTestData
    transfo = GaiaTransform(history = 'highlevel/svm/test_svm_%s.history' % type)
  File "/tmp/essentia-2.1_beta5/build/python/essentia/standard.py", line 44, in __init__
    self.configure(**kwargs)
  File "/tmp/essentia-2.1_beta5/build/python/essentia/standard.py", line 64, in configure
    self.__configure__(**kwargs)
RuntimeError: Error while configuring GaiaTransform: GaiaTransform: error loading gaia history: Could not open file: highlevel/svm/test_svm_probability.history

======================================================================
ERROR: testSingleDesc (unittests.highlevel.test_gaiatransform.TestGaiaTransform)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 64, in testSingleDesc
    self.dotest('singledesc')
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 57, in dotest
    transfo, gt = loadTestData(type)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/highlevel/test_gaiatransform.py", line 28, in loadTestData
    transfo = GaiaTransform(history = 'highlevel/svm/test_svm_%s.history' % type)
  File "/tmp/essentia-2.1_beta5/build/python/essentia/standard.py", line 44, in __init__
    self.configure(**kwargs)
  File "/tmp/essentia-2.1_beta5/build/python/essentia/standard.py", line 64, in configure
    self.__configure__(**kwargs)
RuntimeError: Error while configuring GaiaTransform: GaiaTransform: error loading gaia history: Could not open file: highlevel/svm/test_svm_singledesc.history

======================================================================
ERROR: testRegressionStartAtZero (unittests.temporal.test_loudnessebur128.TestLoudnessEBUR128)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/temporal/test_loudnessebur128.py", line 145, in testRegressionStartAtZero
    m = m[fade_size_m:-fade_size_m]
TypeError: slice indices must be integers or None or have an __index__ method

======================================================================
ERROR: testOgg (unittests.iotest.test_audiowriter_streaming.TestAudioWriter_Streaming)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/iotest/test_audiowriter_streaming.py", line 243, in testOgg
    self.encoderTest('audiowritertest.ogg', precision=5e-6)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/iotest/test_audiowriter_streaming.py", line 186, in encoderTest
    run(gen)
  File "test/src/unittests/all_tests.py", line 200, in runResetRun
  File "/tmp/essentia-2.1_beta5/build/python/essentia/__init__.py", line 148, in run
    return _essentia.run(gen)
RuntimeError: Error while writing delayed audio frame

======================================================================
FAIL: testImpulse (unittests.rhythm.test_superfluxextractor.TestSuperFluxExtractor)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/rhythm/test_superfluxextractor.py", line 56, in testImpulse
    self.assertAlmostEqualVectorAbs(result, expected, precission)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 175, in assertAlmostEqualVectorAbs
    self.assertAlmostEqualAbs(val1, val2, precision)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 170, in assertAlmostEqualAbs
    self.assert_(diff <= precision, 'Difference is %e while allowed absolute error is %e' % (diff, precision))
AssertionError: Difference is 2.979591e-02 while allowed absolute error is 0.000000e+00

======================================================================
FAIL: testFixSeed (unittests.standard.test_noiseadder.TestNoiseAdder)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/standard/test_noiseadder.py", line 61, in testFixSeed
    self.assertNotEqual(a[i], b[i])
AssertionError: -2.8747369e-11 == -2.8747369e-11

======================================================================
FAIL: testRegression (unittests.standard.test_constantq.TestConstantQ)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/standard/test_constantq.py", line 39, in testRegression
    self.assertAlmostEqualVector(numpy.mean(predicted, axis=0), expected, 1e-7)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 145, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 1.641926e-06 while allowed deviation from the expected value is 1.000000e-05 %

======================================================================
FAIL: testRegressionNoZeroPhase (unittests.standard.test_constantq.TestConstantQ)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/standard/test_constantq.py", line 55, in testRegressionNoZeroPhase
    self.assertAlmostEqualVector(numpy.mean(predicted, axis=0), expected, 1e-7)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 145, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 1.641926e-06 while allowed deviation from the expected value is 1.000000e-05 %

======================================================================
FAIL: testRegression (unittests.standard.test_spectrumcq.TestSpectrumCQ)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/standard/test_spectrumcq.py", line 39, in testRegression
    self.assertAlmostEqualVector(numpy.mean(predicted, axis=0), expected, 1e-7)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 141, in assertAlmostEqualVector
    return self.assertTrue(almostEqualArray(found, expected, precision))
AssertionError: False is not true

======================================================================
FAIL: testRegressionNoZeroPhase (unittests.standard.test_spectrumcq.TestSpectrumCQ)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/standard/test_spectrumcq.py", line 55, in testRegressionNoZeroPhase
    self.assertAlmostEqualVector(numpy.mean(predicted, axis=0), expected, 1e-7)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 141, in assertAlmostEqualVector
    return self.assertTrue(almostEqualArray(found, expected, precision))
AssertionError: False is not true

======================================================================
FAIL: testRegression (unittests.standard.test_panning.TestPanning)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/standard/test_panning.py", line 83, in testRegression
    self.assertAlmostEqualVectorFixedPrecision(expected[i], output[0], 2)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 123, in assertAlmostEqualVectorFixedPrecision
    self.assertAlmostEqualFixedPrecision(val1, val2, digits)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 118, in assertAlmostEqualFixedPrecision
    BaseTestCase.assertAlmostEqual(self, found, expected, digits)
AssertionError: -149.452 != -149.87401 within 2 places

======================================================================
FAIL: testRegression (unittests.standard.test_chromagram.TestChromagram)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/standard/test_chromagram.py", line 42, in testRegression
    self.assertAlmostEqualVector(numpy.mean(predicted, axis=0), expected, 1e-7)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 141, in assertAlmostEqualVector
    return self.assertTrue(almostEqualArray(found, expected, precision))
AssertionError: False is not true

======================================================================
FAIL: testRegressionNoZeroPhase (unittests.standard.test_chromagram.TestChromagram)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/standard/test_chromagram.py", line 58, in testRegressionNoZeroPhase
    self.assertAlmostEqualVector(numpy.mean(predicted, axis=0), expected, 1e-7)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 141, in assertAlmostEqualVector
    return self.assertTrue(almostEqualArray(found, expected, precision))
AssertionError: False is not true

======================================================================
FAIL: testAggregateReal (unittests.stats.test_poolaggregator.TestPoolAggregator)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/stats/test_poolaggregator.py", line 44, in testAggregateReal
    self.assertAlmostEqual(results['foo.kurt'], 0.34124279022216797)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 6.986773e-07 while allowed deviation from the expected value is 1.000000e-05 %

======================================================================
FAIL: testMatrixRealAggregation (unittests.stats.test_poolaggregator.TestPoolAggregator)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/stats/test_poolaggregator.py", line 99, in testMatrixRealAggregation
    self.assertAlmostEqualVector(results['foo.var'], [7.26]*3)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 145, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 1.655137e-07 while allowed deviation from the expected value is 1.000000e-05 %

======================================================================
FAIL: testRealCase (unittests.spectral.test_gfcc.TestGFCC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/spectral/test_gfcc.py", line 154, in testRealCase
    self.assertAlmostEqualVector(mean(pool['gfcc'], 0), expected, 1.0e-5)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 145, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 2.226567e-05 while allowed deviation from the expected value is 1.000000e-03 %

======================================================================
FAIL: testRealCase (unittests.spectral.test_mfcc.TestMFCC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/spectral/test_mfcc.py", line 278, in testRealCase
    self.assertAlmostEqualVector(mean(pool['mfcc'], 0), expected, 1.e-8)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 145, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 2.335875e-07 while allowed deviation from the expected value is 1.000000e-06 %

======================================================================
FAIL: testRegression (unittests.spectral.test_mfcc.TestMFCC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/spectral/test_mfcc.py", line 67, in testRegression
    self.assertAlmostEqualVector(mfccs, expected, 1.e-6)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 145, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 1.635205e-06 while allowed deviation from the expected value is 1.000000e-04 %

======================================================================
FAIL: testARealCase (unittests.tonal.test_pitchyinprobabilistic.TestPitchYinProbabilistic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/tonal/test_pitchyinprobabilistic.py", line 117, in testARealCase
    self.assertAlmostEqualVector(pitch, expected_pitch)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 145, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 4.728411e-07 while allowed deviation from the expected value is 1.000000e-05 %

======================================================================
FAIL: testARealCase (unittests.tonal.test_pitchyinfft.TestPitchYinFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/essentia-2.1_beta5/test/src/unittests/tonal/test_pitchyinfft.py", line 140, in testARealCase
    self.assertAlmostEqualVector(pitch, expected_pitch)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 145, in assertAlmostEqualVector
    self.assertAlmostEqual(val1, val2, precision)
  File "/tmp/essentia-2.1_beta5/test/src/unittests/essentia_test.py", line 132, in assertAlmostEqual
    self.assert_(diff <= precision, 'Difference is %e while allowed deviation from the expected value is %e %%' % (diff, precision*100))
AssertionError: Difference is 4.027879e-07 while allowed deviation from the expected value is 1.000000e-05 %

----------------------------------------------------------------------
Ran 1286 tests in 607.999s

FAILED (failures=16, errors=6)
failed to run python tests. Check test output
Running python unit tests using /software/Python/2.7.15-GCCcore-8.2.0/bin/python

yyf commented 4 years ago

Ran into the same error ("ImportError: No module named audioproblems.test_saturationdetector") and wondering if you have any luck resolving this?

I was able to configure, compile, and install essentia version 2.1_beta5 on my macOS 10.15.4. Unfortunately it failed all three tests (./waf run_tests, ./waf run_python_tests, python3 ./waf run_python_tests).

boegel commented 4 years ago

@yyf As mentioned, I fixed that by creating an empty __init__.py file in test/src/unittests/audioproblems/:

touch test/src/unittests/audioproblems/__init__.py

The tests still failed though, I was hoping to hear back from the Essentia developers on this...

palonso commented 4 years ago

Hi @boegel thank you for reporting those problems.

Your patch to prevent $PYTHONPATH from being wiped looks fine to me. I think we can merge it. On the other hand, the rest of the problems you found are related to Python 2 which we are no longer supporting. In Python 3, the __init__.py file is not required thanks to the implicit namespace packages. Also, for the same reason there is no need to rename io/ as it is not imported as a separate module.

Regarding the results, those are all known issues and there should not be any problems with them.

Different architectures or dependency versions are known to produce small numeric differences that can make some Regression or RealCase tests fail. These differences are generally not important unless reproducing the exact values from an older Essentia version is crucial for your application. In my case, for example, I'm building Essentia from the master on Ubuntu 20.04 with Python 3.8 and all those tests are passing.
The GaiaTransform tests are outdated and ERROR is expected.
TestLoudnessEBUR128 ERRORs because it uses float values as indices in Python 2 (round() returns float values)
TestAudioWriter_Streaming ERRORs when writing Vorbis files. This problem is related to your particular FFmpeg installation. If you need to write audio files a quick solution would be to use any other format (.wav, .mp3...)

MTG / essentia

failing Python tests (after fixing some problems) #914