search

craffel / mir_eval

Evaluation functions for music/audio information retrieval/signal processing algorithms.
MIT License
599 stars 112 forks source link

Difference between transcription.precision_recall_f1_overlap and multipitch.metrics? #223

Open carlthome opened 7 years ago

carlthome commented 7 years ago

Considering that transcription.py's symbolic representation is piano rolls with non-quantized pitches, what's the difference between transcription.py and multipitch.py really? I'm extra confused by how transcription.py references the multi-f0 mirex estimation task. Could anyone help clarify why both modules exist? When should which be used?

craffel commented 7 years ago

cc @rabitt @justinsalamon

Well, the difference starts with the fact that their annotation format is quite different. multipitch annotations specify, on a fixed timing grid, "what fundamental frequencies are present at this point in time?" transcription is more like MIDI, specifying individual notes, with start times, end times, and pitches. This means that evaluating them ends up looking a good bit different. I think the confusion about transcription linking to MIREX multi-f0 is that MIREX lumps these two arguably different tasks into one "heading". (@justinsalamon please correct me if I'm wrong and there is in fact an error in the docs). See the annotation format section of the MIREX task description page - there are clearly two different formats/tasks here.

Maybe we can make a PR to clarify this in the documentation?

justinsalamon commented 7 years ago

(@justinsalamon please correct me if I'm wrong and there is in fact an error in the docs).

Nope, that's correct.

carlthome commented 7 years ago

Aha, so multipitch.py is for frame-based pitch estimation while transcription.py is for event-based pitch estimation? Thanks, @craffel!

It could be clarified in the docs, IMHO. Especially pros/cons of the respective formats, with some example use cases. Does it make sense to use multipitch.py for a solo piano? Does it make sense to use transcription.py for a string quartet? I'm confused.

craffel commented 7 years ago

It could be clarified in the docs, IMHO.

I agree. If you or @justinsalamon or @rabitt want to make a PR, it would be welcome.

Especially pros/cons of the respective formats, with some example use cases.

This is outside the scope of mir_eval.

rabitt commented 7 years ago

Another difference - multipitch is intended for time varying pitch curves/sequences whereas transcription is events with a single pitch throughout the note.

On Wednesday, September 21, 2016, Colin Raffel notifications@github.com wrote:

It could be clarified in the docs, IMHO.

I agree. If you or @justinsalamon https://github.com/justinsalamon or @rabitt https://github.com/rabitt want to make a PR, it would be welcome.

Especially pros/cons of the respective formats, with some example use cases.

This is outside the scope of mir_eval.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/craffel/mir_eval/issues/223#issuecomment-248763237, or mute the thread https://github.com/notifications/unsubscribe-auth/AHypc1Oe0QOIF_REPkV_LWU-IGuNvY2xks5qsbIAgaJpZM4KCzDL .