musb.DB().test(myfunction) fails if i have the museval code in myfunction

[f90]

I wanted to generate source estimates and evaluate their quality using museval. In my main function, I do the following:

mus = musdb.DB(root_dir="/mnt/daten/Datasets/MUSDB18")
    if mus.test(predict):
         print "Function is valid"
    mus.run(predict, estimates_dir="/mnt/windaten/Source_Estimates")

In my function predict(track) I generate my estimates and then call the museval evaluation:

estimates = {
        'vocals' : voice_pred_audio,
        'accompaniment' : acc_pred_audio
    }

    # Evaluate using museval
    scores = museval.eval_mus_track(
        track, estimates, output_dir="/mnt/daten/Datasets/MUSDB18/eval"
    )

    ....
    return estimates

While this works without the museval call for just generating the source estimate files, it breaks with it in the mus.test function since the eval_mus_track call expects a track with some more source attributes or elements, which it doesn't have, so it gives me

AttributeError: 'NoneType' object has no attribute 'items'

It works if I remove the mus.test(predict) call, so it's not a major problem, but I feel like there should be some handling of the test track as a special case within eval_mus_track to prevent this from crashing?

[faroit]

I did notice this bug myself earlier and addressed it in #7. Did you make sure that you are running the newest version v0.2.2?

[f90]

After updating to musb 0.2.2 and museval 0.2.0, it still gives an error at the same position, but this time with a slightly different error:

Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/daniel/PycharmProjects/AdversarialSourceSeparation/Training.py", line 574, in dsd_100_experiment
    Evaluate.produce_source_estimates(model_config, sup_model_path)
  File "/home/daniel/PycharmProjects/AdversarialSourceSeparation/Evaluate.py", line 182, in produce_source_estimates
    if mus.test(predict):
  File "/home/daniel/tf-env/local/lib/python2.7/site-packages/musdb/__init__.py", line 333, in test
    user_results = user_function(test_track)
  File "/home/daniel/PycharmProjects/AdversarialSourceSeparation/Evaluate.py", line 108, in predict
    track, estimates, output_dir="/mnt/daten/Datasets/MUSDB18/eval"
  File "/home/daniel/tf-env/local/lib/python2.7/site-packages/museval/__init__.py", line 410, in eval_mus_track
    track.subset
  File "/home/daniel/tf-env/lib/python2.7/posixpath.py", line 68, in join
    if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'startswith'

[faroit]

okay, I think I've found the bug and fixed it in: https://github.com/sigsep/sigsep-mus-db/commit/f6843336e30e6ab01e695dbe3d2717bd950f03b0

Looking back, the test() function should have been done differently

Can you test it by manually installing the master branch?

[f90]

Updated to current master branch 0.2.3 but now I'm getting

Pre-trained model restored for song prediction
vocals              => SDR:-0.166dB, SIR:227.987dB, ISR:-0.160dB, SAR:7.732dB, 
accompaniment       => SDR:2.344dB, SIR:227.892dB, ISR:2.343dB, SAR:27.503dB, 

ERROR - Adversarial_Source_Separation - Failed after 0:00:36!
Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/daniel/PycharmProjects/AdversarialSourceSeparation/Training.py", line 574, in dsd_100_experiment
    Evaluate.produce_source_estimates(model_config, sup_model_path)
  File "/home/daniel/PycharmProjects/AdversarialSourceSeparation/Evaluate.py", line 182, in produce_source_estimates
    if mus.test(predict):
  File "/home/daniel/tf-env/local/lib/python2.7/site-packages/musdb/__init__.py", line 344, in test
    "Estimate is not of type numpy.float_"
ValueError: Estimate is not of type numpy.float_

My estimate is a np.float32 type, and it looks like float64 is expected, which doesn't make much sense for an audio signal to have that high bit depth

[faroit]

yes, sorry I was not aware of the changes in newer numpy versions regarding the use of float_ or np.floating which both refer to np.float64 now.

Should be addressed in: https://github.com/sigsep/sigsep-mus-db/commit/321a401384f0d9e16dbc7143234cc1854eb3bb7b

If you know of a more elegant solution, let me know.

[faroit]

My estimate is a np.float32 type, and it looks like float64 is expected, which doesn't make much sense for an audio signal to have that high bit depth

you are right in terms of audio quality. However, be aware of potential differences for some frames when it comes to the bsseval score compution.

[faroit]

@f90 does this work now?

[f90]

Yes it works with the current master branch now. Thanks!

[faroit]

ok. I've tagged a new version and uploaded to pypi... v0.2.3 thanks for the reporting