search

urinieto / msaf

Music Structure Analysis Framework
MIT License
501 stars 79 forks source link

Benchmark of algorithms #17

Open keunwoochoi opened 8 years ago

keunwoochoi commented 8 years ago

Hi, do you have any benchmark result of the algorithms? Accuracy(or any other measure), complexity, or whatever. If not, could you recommend me one? I'd like to get segmentations of 10k songs, and I can accept some degradation of performance for significantly low complexity.

urinieto commented 8 years ago

Hi, you can refer to my dissertation for some detailed scores (especially in the Appendices) for the existing algorithms in MSAF.

Generally, I would recommend Structural Features (sf) for boundaries, and 2D-Fourier Magnitude Coefficients (fmc2d) for labels.

Regardless, convolutional neural networks should outperform any of the existing algorithms in MSAF right now (this is the latest publication about it, I believe). I might include an implementation in MSAF sometime in the future (right, @f0k? :)).

f0k commented 8 years ago

I might include an implementation in MSAF sometime in the future

Yeah, thanks for the reminder! I'll still have to figure out what's possible. Maybe the most realistic setting would be us posting the implementation from MIREX 2014 under a license not for commercial use. That would mean it would live outside of MSAF, but it could produce outputs in a compatible format so you can still use MSAF for labelling -- I think I remember it was relatively straightforward to use MSAF to label existing segmentations? Alternatively, MSAF could probably import and use it.

I can accept some degradation of performance for significantly low complexity.

Our CNNs are not tuned towards low complexity -- if you need high throughput, a simple hand-crafted algorithm may be better. A complex hand-crafted algorithm will probably be slower than the CNNs. Would be interesting to do a direct comparison some time.

keunwoochoi commented 8 years ago

On MacPro 2014 (no multi processing) with 10 wave files. CNMF took 1555 seconds and Spectral clustering: 211 seconds. Posting for someone who might be also interested, performance was not evaluated though.

urinieto commented 8 years ago

Thanks! Yeah, C-NMF can be demanding. Probably the Checkerboard kernel (foote) is the fastest at finding boundaries. And 2D-FMC (fmc2d) the fastest at labeling the segments.