Onsets and OnsetDetection

[gpeles]

The Onsets algorithm expects a 'matrix_real' as its first input. How can use the output of the OnsetDetection algorithm to create such a matrix in JavaScript? Also, the second input of OnsetDetection algorithm is 'phase'. Which algorithm should I use to compute that?

Many thanks in advance for your help!

[albincorreya]

Currently, our essentia.js bindings doesn't directly support algorithms with matrix_real I/O. Please see #27 for more details. However, you can still use these algorithms by writing a custom cpp wrapper and compile to JS if necessary. You can find a simple example here in that case.

Regarding OnsetDetection algorithm, you only need the phase component of the spectrum if you are using the parameter method='complex'. In all other cases, you can just pass an empty vector as phase input.

[gpeles]

Thanks! Will look into writing a custom cpp wrapper.

And regarding OnsetDetection, in case I do want to use the method 'complex', how can I compute the phase component?

[gpeles]

Trying OnsetDetection now, it doesn't seem to work for methods 'melflux' and 'rms'. The output is always 0. Methods 'hfc' and 'flux' seem to work fine. Also, when set to 'melflux', you get a warning if the size of the spectrum is not 1025 as this is the default of TriangularBands.

[jreus]

Does anyone perhaps have a working example of real-time onset detection running on microphone input in the browser? I'm able to run the essentiaExtractor.OnsetDetection algorithm, which returns a real value. The documentation states that this value then needs additional postprocessing to determine if an onset happened in the analyzed frame... however the postprocessing algorithm Onsets does not seem to exist in the javascript api.

[gpeles]

@jreus A very basic example would be something like

tmpOnset = onsetDetection > threshold
onset = !lastOnset && tmpOnset
lastOnset = tmpOnset

where onsetDetection is the real value you get from the OnsetDetection algorithm, threshold is the minimum value for an onset to be detected, tmpOnset and lastOnset are used to prevent consecutive onsets, and onset is true if an onset has just happened and false if not.

[jreus]

@gpeles

So if I am to understand correctly: the OnsetDetection algorithm returns a real value that could be understood as a measure of confidence that an onset occurred in that frame?

This also would mean that the temporal resolution of OnsetDetection is limited to the size of a frame?

thanks a bunch!

[gpeles]

@jreus

So if I am to understand correctly: the OnsetDetection algorithm returns a real value that could be understood as a measure of confidence that an onset occurred in that frame?

Yes i think so

This also would mean that the temporal resolution of OnsetDetection is limited to the size of a frame?

Yes but if you discard consecutive onsets it's actually the frame size * 2

[kmturley]

How was "onsets.module.js" generated?

When I run this code, using the same Matrix generated from the frontend:

import { Essentia, EssentiaWASM } from 'essentia.js';
import OnsetsWASM from './lib/onsets.module.js';
import matrixData from './matrix.json'
const essentia: Essentia = new Essentia(EssentiaWASM);
const alpha = 1 - 0.65;
const Onsets = new OnsetsWASM.Onsets(alpha, 5, file.buffer.sampleRate / 512, 0.02);
console.log(Onsets.compute(matrixData, [0.5, 0.5]));

It throws an error in NodeJS:

RuntimeError: abort(TypeError: Cannot convert "undefined" to int). Build with -s ASSERTIONS=1 for more info.

Wondering If I can generated a "onsets.module.js" file which is compatible

[kmturley]

I take that back, it is working in Node.JS. I had to generate the matrix using the Polar module for it to work:

import * as wav from 'node-wav';
import { readFileSync } from 'fs';
import { Essentia, EssentiaWASM } from 'essentia.js';
import PolarFFTWASM from './lib/polarFFT.module.js';
import OnsetsWASM from './lib/onsets.module.js';

function analyzeOnsets(buffer) {
  const params = {
    frameSize: 1024,
    hopSize: 512,
    odfs: ["hfc","complex"],
    odfsWeights: [0.5,0.5],
    sensitivity: 0.65
  };
  // Calculate polar frames.
  const polarFrames = [];
  let PolarFFT = new PolarFFTWASM.PolarFFT(params.frameSize);
  let frames = essentia.FrameGenerator(buffer.channelData[0], params.frameSize, params.hopSize);
  for (let i = 0; i < frames.size(); i++) {
      let currentFrame = frames.get(i);
      let windowed = essentia.Windowing(currentFrame).frame;
      const polar = PolarFFT.compute(essentia.vectorToArray(windowed));
      polarFrames.push(polar);
  }
  frames.delete();
  PolarFFT.shutdown();
  // Calculate onsets.
  const alpha = 1 - params.sensitivity; 
  const Onsets = new OnsetsWASM.Onsets(alpha, 5, buffer.sampleRate / params.hopSize, 0.02); 
  const odfMatrix = [];
  for (const func of params.odfs) {
      const odfArray = polarFrames.map( (frame) => {
          return essentia.OnsetDetection(
              essentia.arrayToVector(essentia.vectorToArray(frame.magnitude)), 
              essentia.arrayToVector(essentia.vectorToArray(frame.phase)), 
              func, buffer.sampleRate).onsetDetection;
      });
      odfMatrix.push(Float32Array.from(odfArray));
  }
  const onsetPositions = Onsets.compute(odfMatrix, params.odfsWeights).positions;
  Onsets.shutdown();
  if (onsetPositions.size() == 0) { return new Float32Array(0) }
  else { return essentia.vectorToArray(onsetPositions); }
}

const fileBuffer = readFileSync('./velocity-saw.wav');
const audioBuffer = wav.decode(fileBuffer);
analyzeOnsets(audioBuffer);

I also had to modify the modules last lines to be compatible with CommonJS modules:

./lib/onsets.module.js

// export { Module as OnsetsWASM };
exports.default = Module;

./lib/polarFFT.module.js

// export { Module as PolarFFTWASM };
exports.default = Module;

Would be good to have a non-compiled version of the *.modules.js which we could compile ourselves.