MTG / essentia.js

JavaScript library for music/audio analysis and processing powered by Essentia WebAssembly
https://essentia.upf.edu/essentiajs
GNU Affero General Public License v3.0
632 stars 41 forks source link

Emscripten bindings for the all set of available essentia algorithms #9

Closed albincorreya closed 4 years ago

albincorreya commented 4 years ago

Javascript bindings for the whole set of essentia C++ algorithms as similar to the python bindings.

albincorreya commented 4 years ago

Since writing it manually can be a tedious task, we decided to automate it by generating c++ source code using some python scripts which can parse essentia algorithm docstrings.

albincorreya commented 4 years ago

Currently, the python code_generator.py script creates cpp source code and bindings for all the algos listed in included_algos.md. This list of included algorithms can be customised using the configure_bindings.py script.

cd src/python
# configure default list of algorithms for creating the js bindings
python configure_bindings.py 
# OR
# specify a list of algorithms for which you need to create the js bindings
python configure_bindings.py -i "['Key', 'HPCP']"
# OR
# you can also specify the algorithm list by a txt file
python configure_bindings.py -i your_included_algos_list.txt
# for more cli options
python configure_bindings.py -h

On JavaScript end, the usage would be like,

let essentia = new Module.EssentiaJS(false);

// for algorithms with single output
let yourOutputVar = essentia.'<your-essentia-algo>'(<inputs> ..., <parameters> ...);

// for algorithms with multiple outputs
essentia.'<your-essentia-algo>'(<inputs> ..., <outputs> ..., <parameters> ...);

For example in real use-cases, it will look like below,


// Computing ReplayGain from an input audio signal
// The algorithm return float type
// check https://essentia.upf.edu/reference/std_ReplayGain.html
let replaygain = essentia.ReplayGain(inputSignalVector, // input
                                     44100); // sampleRate (parameter)

// Running PitchYinProbabilistic algorithm on an input audio signal
// create empty std::vector<float> vector for populating the output of 
// essentia.PitchYinProbabilistic algorithm
var pitches = new Module.VectorFloat();
var voicedProbabilities = new Module.VectorFloat();

// check https://essentia.upf.edu/reference/std_PitchYinProbabilistic.html
essentia.PitchYinProbabilistic(inputSignalVector, // input_1
                               pitches, // output_1
                               voicedProbabilities, // output_2
                               // parameters
                               4096, // frameSize
                               256, // hopSize
                               0.1, // lowRMSThreshold
                               'zero', // outputUnvoiced,
                               false, // preciseTime
                               44100); //sampleRate

As we can see in the above example, the user needs to specify all the required value for all the parameters. This can be irritating since a lot of essentia Algorithms has more than 10 parameters such as PredominantPitchMelodia, MultiPitchKlapurietc. The current way of binding the class EssentiaJS using embind is not binding the default parameter values specified in the class.

In order to add bindings for default parameter values in JS front of essentia, we need to do overload every method inside the class EssentiaJS as detailed in embind documentation.