search

meinardmueller / synctoolbox

Sync Toolbox - Python package with reference implementations for efficient, robust, and accurate music synchronization based on dynamic time warping (DTW)
https://meinardmueller.github.io/synctoolbox
Other
108 stars 11 forks source link

Unit tests fail #7

Closed faroit closed 3 years ago

faroit commented 3 years ago

running the unit tests on python 3.8 using the proposed procedure seem to fail:

base38 ❯ py.test tests                                                                                                (base38)
===================================================== test session starts =====================================================
platform darwin -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/faro/repositories/synctoolbox
collected 13 items

tests/test_dtw.py ....                                                                                                  [ 30%]
tests/test_features.py ....FF...                                                                                        [100%]

========================================================== FAILURES ===========================================================
_____________________________________________________ test_pitch_features _____________________________________________________

    def test_pitch_features():
        audio_1, _ = librosa.load('data_music/Schubert_D911-03_HU33.wav', 22050)
        audio_2, _ = librosa.load('data_music/Schubert_D911-03_SC06.wav', 22050)
        f_pitch_1_gt = np.load('tests/data/f_pitch_1.npy')
        f_pitch_2_gt = np.load('tests/data/f_pitch_2.npy')

        f_pitch_1 = audio_to_pitch_features(f_audio=audio_1,
                                            Fs=22050,
                                            tuning_offset=1,
                                            feature_rate=50,
                                            verbose=False)

        f_pitch_2 = audio_to_pitch_features(f_audio=audio_2,
                                            Fs=22050,
                                            tuning_offset=7,
                                            feature_rate=50,
                                            verbose=False)

>       assert np.allclose(f_pitch_1, f_pitch_1_gt, atol=1e-5)
E       assert False
E        +  where False = <function allclose at 0x7f85d8b23670>(array([[0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       ...,\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.]]), array([[0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       ...,\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.]]), atol=1e-05)
E        +    where <function allclose at 0x7f85d8b23670> = np.allclose

tests/test_features.py:81: AssertionError
---------------------------------------------------- Captured stdout call -----------------------------------------------------
................................................................................................................................................................................
__________________________________________________ test_pitch_onset_features __________________________________________________

    def test_pitch_onset_features():
        audio_1, _ = librosa.load('data_music/Schubert_D911-03_HU33.wav', 22050)
        audio_2, _ = librosa.load('data_music/Schubert_D911-03_SC06.wav', 22050)
        f_pitch_onset_1_gt = load_dict('tests/data/f_pitch_onset_1.pickle')
        f_pitch_onset_2_gt = load_dict('tests/data/f_pitch_onset_2.pickle')

        f_pitch_onset_1 = audio_to_pitch_onset_features(f_audio=audio_1,
                                                        Fs=22050,
                                                        tuning_offset=1,
                                                        verbose=False)

        f_pitch_onset_2 = audio_to_pitch_onset_features(f_audio=audio_2,
                                                        Fs=22050,
                                                        tuning_offset=7,
                                                        verbose=False)

>       dict_allclose(f_pitch_onset_1, f_pitch_onset_1_gt, atol=1e-5)

tests/test_features.py:101:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/utils.py:26: in dict_allclose
    assert np.allclose(dict1[k], dict2[k], atol)
<__array_function__ internals>:5: in allclose
    ???
/opt/homebrew/Caskroom/miniconda/base/envs/base38/lib/python3.8/site-packages/numpy/core/numeric.py:2171: in allclose
    res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
<__array_function__ internals>:5: in isclose
    ???
/opt/homebrew/Caskroom/miniconda/base/envs/base38/lib/python3.8/site-packages/numpy/core/numeric.py:2272: in isclose
    return within_tol(x, y, atol, rtol)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

x = array([[[3.10917800e+04],
        [3.11484694e+04],
        [3.11824830e+04],
        [3.12164966e+04],
        [3.123...9819230e-09],
        [1.69793182e-09],
        [1.56447331e-09],
        [1.20539503e-09],
        [9.61932239e-10]]])
y = array([[[3.10917800e+04],
        [3.11484694e+04],
        [3.11824830e+04],
        [3.12164966e+04],
        [3.123...7001423e-09],
        [1.67276090e-09],
        [1.54219300e-09],
        [1.18766368e-09],
        [9.48867734e-10]]])
atol = 1e-08, rtol = 1e-05

    def within_tol(x, y, atol, rtol):
        with errstate(invalid='ignore'):
>           return less_equal(abs(x-y), atol + rtol * abs(y))
E           ValueError: operands could not be broadcast together with shapes (2,138,1) (2,136,1)

/opt/homebrew/Caskroom/miniconda/base/envs/base38/lib/python3.8/site-packages/numpy/core/numeric.py:2258: ValueError
---------------------------------------------------- Captured stdout call -----------------------------------------------------
................................................................................................................................................................................21 (2, 138, 1) (2, 136, 1)
=================================================== short test summary info ===================================================
FAILED tests/test_features.py::test_pitch_features - assert False
FAILED tests/test_features.py::test_pitch_onset_features - ValueError: operands could not be broadcast together with shapes ...
================================================ 2 failed, 11 passed in 56.14s ================================================

related #5 https://github.com/openjournals/joss-reviews/issues/3434

yiitozer commented 3 years ago

Dear @faroit, thank you for pointing this out.

I fixed this problem, it was due to the scipy version. Could you try it out again?

faroit commented 3 years ago

thanks @yiitozer. I think I would leave this open until #5 is addressed so I wouldn't have to try it manually

faroit commented 3 years ago

@yiitozer i guess this can be closed? Maybe you could add github action unit test status badges to the main readme?

yiitozer commented 3 years ago

Done! I'm closing this issue.

magdalenafuentes commented 3 years ago

These two tests keep failing for me locally after pulling latest changes, but they are passing remotely. Any idea what might be going on?

yiitozer commented 3 years ago

I found out that the scipy version was the cause of this issue. Could you tell me what you scipy version is? It should be greater than or equal to 1.7.0.

We'll update the package on the PyPI in short: https://pypi.org/project/synctoolbox/

magdalenafuentes commented 3 years ago

I have scipy 1.7.1, here my full conda environment:

Package             Version
------------------- -------------------
appdirs             1.4.4
appnope             0.1.2
argon2-cffi         20.1.0
async-generator     1.10
attrs               21.2.0
audioread           2.1.9
backcall            0.2.0
bleach              4.0.0
certifi             2021.5.30
cffi                1.14.6
charset-normalizer  2.0.4
cycler              0.10.0
debugpy             1.4.1
decorator           5.0.9
defusedxml          0.7.1
entrypoints         0.3
idna                3.2
iniconfig           1.1.1
ipykernel           6.0.3
ipython             7.26.0
ipython-genutils    0.2.0
ipywidgets          7.6.3
jedi                0.18.0
Jinja2              3.0.1
joblib              1.0.1
jsonschema          3.2.0
jupyter             1.0.0
jupyter-client      6.1.12
jupyter-console     6.4.0
jupyter-core        4.7.1
jupyterlab-pygments 0.1.2
jupyterlab-widgets  1.0.0
kiwisolver          1.3.1
libfmp              1.2.1
librosa             0.8.1
llvmlite            0.36.0
MarkupSafe          2.0.1
matplotlib          3.4.2
matplotlib-inline   0.1.2
mido                1.2.10
mistune             0.8.4
music21             5.7.2
nbclient            0.5.3
nbconvert           6.1.0
nbformat            5.1.3
nest-asyncio        1.5.1
notebook            6.4.0
numba               0.53.1
numpy               1.21.1
packaging           21.0
pandas              1.3.1
pandocfilters       1.4.3
parso               0.8.2
pexpect             4.8.0
pickleshare         0.7.5
Pillow              8.3.1
pip                 21.2.2
pluggy              0.13.1
pooch               1.4.0
pretty-midi         0.2.9
prometheus-client   0.11.0
prompt-toolkit      3.0.19
ptyprocess          0.7.0
py                  1.10.0
pycparser           2.20
Pygments            2.9.0
pyparsing           2.4.7
pyrsistent          0.18.0
pyrubberband        0.3.0
PySoundFile         0.9.0.post1
pytest              6.2.4
python-dateutil     2.8.2
pytz                2021.1
pyzmq               22.2.0
qtconsole           5.1.1
QtPy                1.9.0
requests            2.26.0
resampy             0.2.2
scikit-learn        0.24.2
scipy               1.7.1
Send2Trash          1.7.1
setuptools          52.0.0.post20210125
six                 1.16.0
SoundFile           0.10.3.post1
synctoolbox         1.0.0
terminado           0.10.1
testpath            0.5.0
threadpoolctl       2.2.0
toml                0.10.2
tornado             6.1
traitlets           5.0.5
urllib3             1.26.6
wcwidth             0.2.5
webencodings        0.5.1
wheel               0.36.2
widgetsnbextension  3.5.1
magdalenafuentes commented 3 years ago

@yiitozer I tried with scipy==1.7.0 as well and I still see the errors. Let me know how to proceed.

yiitozer commented 3 years ago

@magdalenafuentes could you send me the error message? Is that identical to the error message above?

magdalenafuentes commented 3 years ago

Yes, it's the same:

tests/test_dtw.py ....                                                                                                       [ 30%]
tests/test_features.py ....FF...                                                                                             [100%]

============================================================= FAILURES =============================================================
_______________________________________________________ test_pitch_features ________________________________________________________

    def test_pitch_features():
        audio_1, _ = librosa.load('data_music/Schubert_D911-03_HU33.wav', 22050)
        audio_2, _ = librosa.load('data_music/Schubert_D911-03_SC06.wav', 22050)
        f_pitch_1_gt = np.load('tests/data/f_pitch_1.npy')
        f_pitch_2_gt = np.load('tests/data/f_pitch_2.npy')

        f_pitch_1 = audio_to_pitch_features(f_audio=audio_1,
                                            Fs=22050,
                                            tuning_offset=1,
                                            feature_rate=50,
                                            verbose=False)

        f_pitch_2 = audio_to_pitch_features(f_audio=audio_2,
                                            Fs=22050,
                                            tuning_offset=7,
                                            feature_rate=50,
                                            verbose=False)

        assert np.allclose(f_pitch_1, f_pitch_1_gt, atol=1e-5)
>       assert np.allclose(f_pitch_2, f_pitch_2_gt, atol=1e-5)
E       assert False
E        +  where False = <function allclose at 0x7fc7c81ea430>(array([[0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       ...,\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.]]), array([[0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       ...,\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.]]), atol=1e-05)
E        +    where <function allclose at 0x7fc7c81ea430> = np.allclose

tests/test_features.py:82: AssertionError
------------------------------------------------------- Captured stdout call -------------------------------------------------------
................................................................................................................................................................................
____________________________________________________ test_pitch_onset_features _____________________________________________________

    def test_pitch_onset_features():
        audio_1, _ = librosa.load('data_music/Schubert_D911-03_HU33.wav', 22050)
        audio_2, _ = librosa.load('data_music/Schubert_D911-03_SC06.wav', 22050)
        f_pitch_onset_1_gt = load_dict('tests/data/f_pitch_onset_1.pickle')
        f_pitch_onset_2_gt = load_dict('tests/data/f_pitch_onset_2.pickle')

        f_pitch_onset_1 = audio_to_pitch_onset_features(f_audio=audio_1,
                                                        Fs=22050,
                                                        tuning_offset=1,
                                                        verbose=False)

        f_pitch_onset_2 = audio_to_pitch_onset_features(f_audio=audio_2,
                                                        Fs=22050,
                                                        tuning_offset=7,
                                                        verbose=False)

>       dict_allclose(f_pitch_onset_1, f_pitch_onset_1_gt, atol=1e-5)

tests/test_features.py:101: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/utils.py:26: in dict_allclose
    assert np.allclose(dict1[k], dict2[k], atol)
<__array_function__ internals>:5: in allclose
    ???
/Users/mf3734/miniconda3/envs/joss_review_2/lib/python3.8/site-packages/numpy/core/numeric.py:2249: in allclose
    res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
<__array_function__ internals>:5: in isclose
    ???
/Users/mf3734/miniconda3/envs/joss_review_2/lib/python3.8/site-packages/numpy/core/numeric.py:2358: in isclose
    return within_tol(x, y, atol, rtol)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = array([[[3.10917800e+04],
        [3.11484694e+04],
        [3.11824830e+04],
        [3.12164966e+04],
        [3.123...0950938e-09],
        [1.70858722e-09],
        [1.57374780e-09],
        [1.21270285e-09],
        [9.67767082e-10]]])
y = array([[[3.10917800e+04],
        [3.11484694e+04],
        [3.11824830e+04],
        [3.12164966e+04],
        [3.123...8774396e-09],
        [1.68822468e-09],
        [1.55594432e-09],
        [1.19865935e-09],
        [9.56583904e-10]]])
atol = 1e-08, rtol = 1e-05

    def within_tol(x, y, atol, rtol):
        with errstate(invalid='ignore'):
>           return less_equal(abs(x-y), atol + rtol * abs(y))
E           ValueError: operands could not be broadcast together with shapes (2,138,1) (2,136,1)

/Users/mf3734/miniconda3/envs/joss_review_2/lib/python3.8/site-packages/numpy/core/numeric.py:2339: ValueError
------------------------------------------------------- Captured stdout call -------------------------------------------------------
................................................................................................................................................................................21 (2, 138, 1) (2, 136, 1)

Then a lot of warnings I skip for clarity and

-- Docs: https://docs.pytest.org/en/stable/warnings.html
===================================================== short test summary info ======================================================
FAILED tests/test_features.py::test_pitch_features - assert False
FAILED tests/test_features.py::test_pitch_onset_features - ValueError: operands could not be broadcast together with shapes (2,13...
=========================================== 2 failed, 11 passed, 1482 warnings in 42.34s ===========================================
yiitozer commented 3 years ago

@magdalenafuentes thank you, I could reproduce it. I'll work on it.

yiitozer commented 3 years ago

@magdalenafuentes, I noticed that the problem is related to the filtfilt function (scipy.signal.filtfilt).

Somehow its result is not deterministic, although the scipy version is the same for each python version. I have replaced it with sosfiltfilt following the scipy.signal.filfilt docstring:

The function `sosfiltfilt` (and filter design using ``output='sos'``)
should be preferred over `filtfilt` for most filtering tasks, as
second-order sections have fewer numerical problems.

The workflows are running successfully with this change. https://github.com/meinardmueller/synctoolbox/actions/runs/1105179972 https://github.com/meinardmueller/synctoolbox/actions/runs/1105179973

yiitozer commented 3 years ago

I solved the issue with the pull request #15.

@magdalenafuentes, can you please run the tests with the up-to-date version of the master branch on your local machine? If that also works on your local machine, I will create a new release in pip. This would close issue #12 as well.

magdalenafuentes commented 3 years ago

Hey @yiitozer I pulled the latest changes and this is what I'm getting when running the testes (pytest test):

(joss_review_2) cusp-vmn244:synctoolbox mf3734$ git pull origin master
remote: Enumerating objects: 73, done.
remote: Counting objects: 100% (73/73), done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 48 (delta 24), reused 39 (delta 20), pack-reused 0
Unpacking objects: 100% (48/48), done.
From https://github.com/meinardmueller/synctoolbox
 * branch            master     -> FETCH_HEAD
   cb52c9c..e5fba5c  master     -> origin/master
Updating cb52c9c..e5fba5c
Fast-forward
 .github/workflows/test_conda.yml    |   8 ++++++--
 synctoolbox/feature/filterbank.py   |  12 +-----------
 synctoolbox/feature/pitch.py        |   7 +++----
 synctoolbox/feature/pitch_onset.py  |  15 ++++++++-------
 tests/data/f_CENS_1.npy             | Bin 14624 -> 14624 bytes
 tests/data/f_DLNCO_1.npy            | Bin 721856 -> 721856 bytes
 tests/data/f_DLNCO_2.npy            | Bin 658688 -> 658688 bytes
 tests/data/f_chroma_1.npy           | Bin 721856 -> 721856 bytes
 tests/data/f_chroma_2.npy           | Bin 658688 -> 658688 bytes
 tests/data/f_chroma_quantized_1.npy | Bin 721856 -> 721856 bytes
 tests/data/f_filtfilt.npy           | Bin 0 -> 1060832 bytes
 tests/data/f_onset.npy              | Bin 0 -> 106184 bytes
 tests/data/f_pitch_1.npy            | Bin 7698560 -> 7698560 bytes
 tests/data/f_pitch_2.npy            | Bin 7024768 -> 7024768 bytes
 tests/data/f_pitch_onset_1.pickle   | Bin 160542 -> 160558 bytes
 tests/data/f_pitch_onset_2.pickle   | Bin 168946 -> 168562 bytes
 tests/data/fb.pickle                | Bin 0 -> 22817 bytes
 tests/data/peaks.npy                | Bin 0 -> 1008 bytes
 tests/data/thresh.npy               | Bin 0 -> 106184 bytes
 tests/data/wp_high_res.npy          | Bin 125616 -> 125600 bytes
 tests/data/wp_simple.npy            | Bin 125072 -> 125072 bytes
 tests/test_features.py              |  19 +++++++++++++++++--
 tests/utils.py                      |  25 +++++++++++++++++++++----
 23 files changed, 56 insertions(+), 30 deletions(-)
 create mode 100644 tests/data/f_filtfilt.npy
 create mode 100644 tests/data/f_onset.npy
 create mode 100644 tests/data/fb.pickle
 create mode 100644 tests/data/peaks.npy
 create mode 100644 tests/data/thresh.npy
(joss_review_2) cusp-vmn244:synctoolbox mf3734$ pytest tests
======================================================= test session starts ========================================================
platform darwin -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/mf3734/Documents/repos/review/2021/JOSS/synctoolbox
collected 15 items                                                                                                                 

tests/test_dtw.py ....                                                                                                       [ 26%]
tests/test_features.py ....F.FF...                                                                                           [100%]

============================================================= FAILURES =============================================================
_________________________________________________________ test_filterbank __________________________________________________________

    def test_filterbank():
        fb_gt = load_dict('tests/data/fb.pickle')
        fb = generate_filterbank(semitone_offset_cents=7)
>       filterbank_equal(fb, fb_gt)

tests/test_features.py:67: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fb = {21: {'a': array([  1.        ,  -7.8384458 ,  27.03303336, -53.5719055 ,
        66.71929192, -53.47244784,  26.93275... -0.0245074 ,  0.08410023, -0.16623302,  0.20698032,
       -0.16623302,  0.08410023, -0.0245074 ,  0.00315005])}, ...}
fb_gt = {21: array([[ 0.00315176, -0.00615725,  0.00315176,  1.        , -1.95801479,
         0.99720306],
       [ 1.       ...     0.99886093],
       [ 1.        , -1.94980891,  1.        ,  1.        , -1.94659207,
         0.99890433]]), ...}

    def filterbank_equal(fb, fb_gt):
        """Checks whether the entries of two filterbanks are equal to each other."""
        assert fb.keys() == fb_gt.keys(), 'The MIDI indices of two filterbanks are not identical.'

        for k in fb:
>           assert np.array_equal(fb[k], fb_gt[k])
E           AssertionError

tests/utils.py:43: AssertionError
_______________________________________________________ test_pitch_features ________________________________________________________

    def test_pitch_features():
        audio_1, _ = librosa.load('data_music/Schubert_D911-03_HU33.wav', 22050)
        audio_2, _ = librosa.load('data_music/Schubert_D911-03_SC06.wav', 22050)
        f_pitch_1_gt = np.load('tests/data/f_pitch_1.npy')
        f_pitch_2_gt = np.load('tests/data/f_pitch_2.npy')

        f_pitch_1 = audio_to_pitch_features(f_audio=audio_1,
                                            Fs=22050,
                                            tuning_offset=1,
                                            feature_rate=50,
                                            verbose=False)

        f_pitch_2 = audio_to_pitch_features(f_audio=audio_2,
                                            Fs=22050,
                                            tuning_offset=7,
                                            feature_rate=50,
                                            verbose=False)

>       assert np.allclose(f_pitch_1, f_pitch_1_gt, atol=1e-5)
E       assert False
E        +  where False = <function allclose at 0x7fbc602755e0>(array([[0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       ...,\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.]]), array([[0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       ...,\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.],\n       [0., 0., 0., ..., 0., 0., 0.]]), atol=1e-05)
E        +    where <function allclose at 0x7fbc602755e0> = np.allclose

tests/test_features.py:96: AssertionError
------------------------------------------------------- Captured stdout call -------------------------------------------------------
................................................................................................................................................................................
____________________________________________________ test_pitch_onset_features _____________________________________________________

    def test_pitch_onset_features():
        audio_1, _ = librosa.load('data_music/Schubert_D911-03_HU33.wav', 22050)
        audio_2, _ = librosa.load('data_music/Schubert_D911-03_SC06.wav', 22050)
        f_pitch_onset_1_gt = load_dict('tests/data/f_pitch_onset_1.pickle')
        f_pitch_onset_2_gt = load_dict('tests/data/f_pitch_onset_2.pickle')

        f_pitch_onset_1 = audio_to_pitch_onset_features(f_audio=audio_1,
                                                        Fs=22050,
                                                        tuning_offset=1,
                                                        verbose=False)

        f_pitch_onset_2 = audio_to_pitch_onset_features(f_audio=audio_2,
                                                        Fs=22050,
                                                        tuning_offset=7,
                                                        verbose=False)

>       dict_allclose(f_pitch_onset_1, f_pitch_onset_1_gt, atol=1e-5)

tests/test_features.py:116: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

dict1 = {21: array([[[3.10917800e+04],
        [3.11484694e+04],
        [3.11824830e+04],
        [3.12164966e+04],
        [...0e-10],
        [4.25231229e-10],
        [2.19417328e-10],
        [4.31439692e-10],
        [1.28464351e-09]]]), ...}
dict2 = {21: array([[[3.10917800e+04],
        [3.11484694e+04],
        [3.11824830e+04],
        [3.12164966e+04],
        [...8e-10],
        [4.24328646e-10],
        [2.18858615e-10],
        [4.29832782e-10],
        [1.28280842e-09]]]), ...}
atol = 1e-05, rtol = 1e-05

    def dict_allclose(dict1, dict2, atol=1e-5, rtol=1e-5):
        """Checks whether the entries of two dictionaries are close to each other."""
        assert dict1.keys() == dict2.keys(), 'The midi indices of two feature arrays are not identical.'
        for k in dict1:
>           assert dict1[k].shape == dict2[k].shape, f'Features have different shapes for the midi index {k}.' \
                                                     f'The first feature array f_1[{k}] has {dict1[k].shape} elements;' \
                                                     f'The second feature array f_2[{k}] has {dict2[k].shape} elements:'
E           AssertionError: Features have different shapes for the midi index 21.The first feature array f_1[21] has (2, 138, 1) elements;The second feature array f_2[21] has (2, 136, 1) elements:

tests/utils.py:25: AssertionError

..warnings..

-- Docs: https://docs.pytest.org/en/stable/warnings.html
===================================================== short test summary info ======================================================
FAILED tests/test_features.py::test_filterbank - AssertionError
FAILED tests/test_features.py::test_pitch_features - assert False
FAILED tests/test_features.py::test_pitch_onset_features - AssertionError: Features have different shapes for the midi index 21.T...
=========================================== 3 failed, 12 passed, 1482 warnings in 40.02s ===========================================

So I'm getting one more error now. I'm quite confused since your tests are passing remotely. Do they run ok locally for you? Should we double check test config?

yiitozer commented 3 years ago

Hi @magdalenafuentes,

I'm also quite surprised. Yes, it works on my local machine.

Maybe that's a caching problem. (I don't really think though) Can you try it without caching:

pytest --cache-clear

If it doesn't work maybe installing the repository once more can help:

pip install -e .['tests']

As you mentioned, the remote tests have also been successful, after I fixed the numeric instabilities.

magdalenafuentes commented 3 years ago

Clearing the cache didn't work, and reinstalling the repository didn't work either but it reduced the errors to only one:

(joss_review_2) cusp-vmn244:synctoolbox mf3734$ pytest tests
================================================================================ test session starts ================================================================================
platform darwin -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/mf3734/Documents/repos/review/2021/JOSS/synctoolbox
collected 15 items                                                                                                                                                                  

tests/test_dtw.py ....                                                                                                                                                        [ 26%]
tests/test_features.py ....F......                                                                                                                                            [100%]

===================================================================================== FAILURES ======================================================================================
__________________________________________________________________________________ test_filterbank __________________________________________________________________________________

    def test_filterbank():
        fb_gt = load_dict('tests/data/fb.pickle')
        fb = generate_filterbank(semitone_offset_cents=7)
>       filterbank_equal(fb, fb_gt)

tests/test_features.py:67: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

fb = {21: array([[ 0.00315176, -0.00615725,  0.00315176,  1.        , -1.95801479,
         0.99720306],
       [ 1.       ...     0.99886093],
       [ 1.        , -1.94980891,  1.        ,  1.        , -1.94659207,
         0.99890433]]), ...}
fb_gt = {21: array([[ 0.00315176, -0.00615725,  0.00315176,  1.        , -1.95801479,
         0.99720306],
       [ 1.       ...     0.99886093],
       [ 1.        , -1.94980891,  1.        ,  1.        , -1.94659207,
         0.99890433]]), ...}

    def filterbank_equal(fb, fb_gt):
        """Checks whether the entries of two filterbanks are equal to each other."""
        assert fb.keys() == fb_gt.keys(), 'The MIDI indices of two filterbanks are not identical.'

        for k in fb:
>           assert np.array_equal(fb[k], fb_gt[k])
E           AssertionError

tests/utils.py:43: AssertionError
================================================================================= warnings summary ==================================================================================
../../../../../../miniconda3/envs/joss_review_2/lib/python3.8/site-packages/librosa/core/constantq.py:1059
  /Users/mf3734/miniconda3/envs/joss_review_2/lib/python3.8/site-packages/librosa/core/constantq.py:1059: DeprecationWarning: `np.complex` is a deprecated alias for the builtin `complex`. To silence this warning, use `complex` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.complex128` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    dtype=np.complex,

tests/test_features.py::test_tuning
tests/test_features.py::test_tuning
  /Users/mf3734/miniconda3/envs/joss_review_2/lib/python3.8/site-packages/librosa/util/utils.py:2099: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    np.dtype(np.float): np.complex,

tests/test_features.py::test_tuning
tests/test_features.py::test_tuning
  /Users/mf3734/miniconda3/envs/joss_review_2/lib/python3.8/site-packages/librosa/util/utils.py:2099: DeprecationWarning: `np.complex` is a deprecated alias for the builtin `complex`. To silence this warning, use `complex` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.complex128` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    np.dtype(np.float): np.complex,

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================================================================== short test summary info ==============================================================================
FAILED tests/test_features.py::test_filterbank - AssertionError
===================================================================== 1 failed, 14 passed, 5 warnings in 27.67s =====================================================================

If you change the line 43 in tests/utils.py to assert np.allclose(fb[k], fb_gt[k]) instead of using array_equal the tests pass for me. Would that make sense for you?

yiitozer commented 3 years ago

@magdalenafuentes , thank you for your suggestion.

Sure, I changed it to allclose, the condition doesn't have to be that strict.