mir_eval integration

[bmcfee]

Quick thoughts about making integration with mir_eval simple:

• Provide functionality to generate and populate Annotation objects from .lab files
• Provide wrapper functions for all mir_eval submodules
  • This can be implemented via a handful of transparent decorators
  • The decorated evaluators accept two Annotation objects (ref_annotation, est_annotation), does the necessary unpacking into mir_eval array/list format, and returns the result
  • We may need multiple decorators: events, labeled events, intervals, and labeled intervals. Possibly more may be necessary depending on the task/metric.
  • We may also need to do docstring mangling to change the function signature while retaining the description.

[bmcfee]

Other functions we'll need

• lab_to_jam(namespace, filename, *parsing)
  • create a new jam object by importing a lab file

[bmcfee]

lab_to_jam(namespace, filename, *parsing)

first cut as of e5e13321577a33ac26adea1969d3a7d9cf5df7ee

[bmcfee]

776c0876b67821e613a216010e16bad95a95da69 implements a first cut at the decorator-wrapping for mir_eval functions.

The gist of this is that for each mir_eval submodule (beat, chord, onset, etc), we wrap the master "evaluate" function which calls all evaluation metrics and returns an ordered dict of scores. Since different tasks have different conventions for data (onsets just have event time, chords have interval labels, etc), we need a few different wrappers to make everything work.

TODO:

• Fix the call signatures and docstrings
• Add source separation. This still needs a #13 namespace schema before we can implement the evaluator.
• Add pattern discovery. Again, we need a namespace.

We can have the decorator verify that the input annotations have the right type for the evaluation in question.

• For something like segment, the estimation might disagree with the reference (segment_open vs segment_tut, for instance), but that's ok if we search on 'segment_*'.
• Will this pattern hold in general? Can we just make all evaluation wrappers validate on the pattern '^TASK(_.*)?$' ?

[bmcfee]

Finally started implementing tests for the eval module #26 ; this looks to be pretty badly broken.

The event-oriented evaluation wrappers (beat and onset) work, since their call signatures fit with mir_eval. The others (chord, segment, tempo) do not work, since the wrapped versions take two annotations as input, while mir_eval takes more than two input arguments (eg, ref_intervals, ref_labels, est_intervals, est_labels).

We'll probably have to rethink the structuring of this decorator/wrapper stuff.

[bmcfee]

Ok, all magic has been purged from the eval module. The docstrings are more static/redundant with mir_eval, but the code is much simpler. So that's probably a win.

Also, all currently implemented evaluators now have functioning tests #26.

All we're missing now is melody @justinsalamon and pattern discovery @urinieto linkage.

[bmcfee]

@justinsalamon I took a few minutes to hack in melody eval; please check it over to make sure I didn't mess something up. (See also the corresponding tests)

[justinsalamon]

@bmcfee just had a look at the melody eval code (and the tests) - looks kosher.

[bmcfee]

Thanks @justinsalamon

I've been thinking a bit about the pattern discovery integration, but am unsure about how exactly to make it work. @urinieto : when you get a minute, can you translate some pattern data so I have a concrete example to work with?