JuliaDSP / MFCC.jl

Mel Frequency Cepstral Coefficients calculation for Julia
BSD 2-Clause "Simplified" License
33 stars 18 forks source link

MFCC

CI

A package to compute Mel Frequency Cepstral Coefficients.

The essential routine is re-coded from Dan Ellis's rastamat package, and parameters are named similarly.

Please note that the feature-vector array consists of a vertical stacking of row-vector features. This is consistent with the sense of direction of, e.g., Base.cov(), but inconsistent with, e.g., DSP.spectrogram() or Clustering.kmeans().

mfcc() has many parameters, but most of these are set to defaults that should mimick HTK default parameters (not thoroughly tested).

Feature extraction main routine

mfcc(x::Vector, sr=16000.0, defaults::Symbol; args...)

Extract MFCC features from the audio data in x, using parameter settings characterized by defaults

The actual routine for MFCC computation has many parameters, these are basically the same parameters as in Dan Ellis's rastamat package.

mfcc(x::AbstractVector{<:AbstractFloat}, sr::Real=16000.0; wintime=0.025, steptime=0.01, numcep=13, preemph=0.97, lifterexp=-22, nbands=20, minfreq=0.0, maxfreq=sr/2, fbtype=:htkmel, bwidth=1.0, modelorder=0, dcttype=3, dither::Real=false, sumpower::Bool=false, usecmp::Bool=false)

This is the main routine computing MFCCs. x should be a 1D vector of FloatingPoint samples of speech, sampled at a frequency of sr. Every steptime seconds, a frame of duration wintime is analysed. The log energy in a filterbank of nbands bins is computed, and a cepstral (discrete cosine transform) representation is made, keeping only the first numcep coefficients (including log energy). The result is a tuple of three values:

Pre-set feature extraction applications

We have defined a couple of standard sets of parameters that should function well for particular applications in speech technology. They are accessible through the higher level function feacalc(). The top-level interface for calculating features is

feacalc(wavfile::AbstractString, application::Symbol; kwargs...)

This will compute speech features suitable for a specific application, which currently can be one of:

The kwargs... parameters allow for various options in file format, feature augmentation, speech activity detection and MFCC parameter settings. They trickle down to versions of feacalc() and mfcc(), allowing for more detailed specification of these parameters.

feacalc() returns a tuple of three structures:

More fine-grained control of feacalc()

feacalc(wavfile::AbstractString; method=:wav, kwargs...)

This function reads an audio file from disk and represents the audio as a Matrix, and then runs the feature extraction.

The method parameter determines what method is used for reading in the audio file:

feacalc(x::Array; chan=:mono, augtype=:ddelta, normtype=:warp, sadtype=:energy, dynrange::Real=30., nwarp::Int=399, sr::AbstractFloat=8000.0, source=":array", defaults=:nbspeaker, mfccargs...)

The chan parameter specifies for which channel in the audio file you want features. Possible values are:

The augtype parameter specifies how the speech features are augmented. This can be:

The normtype parameter specifies how the features are normalized after extraction

The sad parameter controls if Speech Activity Detection is carried out on the features, filtering out frames that do not contain speech

The various applications actually have somewhat different parameter settings for the basic MFCC feature extraction, see the defaults parameter of mfcc() above.

Feature warping, or short-time Gaussianization (Jason Pelecanos)

warp(x::Matrix, w=399)

This transforms columns of x by short-time Gaussianization. Each value in the middle of w rows is replaced with its normal deviate (the quantile function of the normal distribution) based on its rank within the w values. The result has the same dimensions as x, but the values are chosen from a discrete set of w normal deviates.

znorm(x::Matrix)
znorm!(x::Matrix)

This normalizes the data x on a column-by-column basis by an affine transformation, making the per-column mean 0 and variance 1.

Short-term mean and variance normalization

As an alternative to short time Gaussianization, and similar to znorm(), you can compute the znorm for a sample in the centre of a sliding window of width w, where mean and variance are computed just over that window using

stmvn(x::Matrix, w=399)

Derivatives

Derivative of features, fitted over width consecutive frames:

deltas(x::Matrix, width::Int)

The derivatives are computed over columns individually, and before the derivatives are computed the data is padded with repeats of the first and last rows. The resulting matrix has the same size as x. deltas() can be applied multiple times in order to get higher order derivatives.

Shifted-Delta-Cepstra

SDCs are features used for spoken language recognition, typically derived from MFCCs

sdc(x::Matrix, n::Int=7, d::Int=1, p::Int=3, k::Int=7)

This function expands (MFCC) features in x by computing derivatives over 2d+1 consecutive frames for the first n columns of x, stacking derivatives shifted over p frames k times. Before the calculation, the deltas are zero padded so that the results have the same number of rows as x.