Shrink output file size by detecting audio repetition

[mk-pmb]

Let's discuss how we can get audio sample collections down to acceptable sizes. Compatibility with the original MIDI.js file format is a minor "nice to have" concern.

Method 1: Learn to actually parse sf2 and use existing loop info.

• :+1: The intuitively correct approach.
• :+1: Straight-forward on an archititectural level.
• :+1: Should provide (almost) loss-less conversion of the original .sf2 file.
• :+1: Almost no CPU cost when compared to actual audio processing.
• :hourglass: I still haven't found easy documentation and/or an easy to use parser module.

Method 2: Detect repetition in the actual audio output.

• :+1: Could achieve better results if the input wasn't optimized aggressively enough.
• :+1: Extends our reach to source material beyond .sf2, especially sources that don't provide repetition info at all.
• :+1: Might provide legal advantage in cases where the tones themselves aren't covered by license restrictions, just the meta data is.
• :book: A StackOverflow thread says we could try finding the peaks of an Autocorrelation function and might have better results if we FFT the signal to a spectrogram first.
  • :package: node-autocorrelation might be able to do both.
• :book: CCECE: Detection of Largest Possible Repeated Patterns in Indian Audio Songs using Spectral Features by Mathew Thomas, Y. V. Srinivasa Murthy and Shashidhar G. Koolagudi, 2016. PDF via ResearchGate via Wayback Machine

[hmoffatt]

I just came across your project after you mentioned it in the MIDI.js issues. I have been using the soundfont builder ruby tool included with MIDI.js.

I made a few modifications, partly to decrease the size of the output .js bundle but also to decrease the size of the resulting soundfont once loaded into RAM. MIDI.js decompresses the font when loading it so all the samples are in RAM, which is a killer on an old iPad when you have 3 seconds of audio for every note for every instrument, even if it's silence.

Trimming silence helped with the decompressed size, though it shouldn't affect the output file size much because the compressor should do well with silence. I notice you are doing this already.

Secondly I added support for using AAC audio, which gets you a better bang per bit than MP3, and is better supported than Ogg. But obviously that needs changes to MIDI.js to use it.

Proper looping in the player would be ideal, because you'd get the smallest sample size but also the ability to have infinite sustain, which the current approach can't do.

[mk-pmb]

Proper looping in the player would be ideal,

Yeah, for me that's an obvious must-have, I just didn't find enough time to build that myself yet. This thread shall help people who do have the time (future me maybe) have pointers to how to spend that time efficiently.

AAC audio […] gets you a better bang per bit

Thanks for that hint! Once we get repetition working, I'll consult some browser compat charts and consider adding easier support than via the custom codec. If someone wants it earlier, file a PR. :-)