Whisper fbank features vs Kaldi fbank features – Clarification Needed

[sangeet2020]

Hi, I would like to understand the differences between Whisper's fbank features and Kaldi's fbank features? I get that conceptually both features are derived from Mel-filterbank energies, but then what makes them different?

thank you.

[csukuangfj]

There are many many differences.

For instance, the function for converting hz to mel is different.

Please have a look at the code to find them out by yourself.

[sangeet2020]

I went through it, and indeed there were several minute differences:

• filterbank size (80 in kaldi vs 123 in whisper v3)
• fft size (dynamically computed in kaldi, while whisper uses 400)
• window function (povey in kaldi vs hann in whisper)
• normalization (in whisper log-mel values are normalized to [-4, 0])
• difference in the data padding step.

would there be more?

[csukuangfj]

Is there pre-emphasis?

How is the filter bank matrix computed?

[sangeet2020]

oh yeah, pre-emphasis, missed writing that.

How is the filter bank matrix computed?

kaldi uses dot product between spectrum magnitude and mel_banks, while whisper uses matrix multiplication between stft magnitudes and mel_filters.

[csukuangfj]

I am afraid you don't know the details of mel.filter bank matrix.

[csukuangfj]

or you don't know how the matrix is computed.

[sangeet2020]

Whisper uses librosa.filters.mel() to generate mel filter bank matrix, while kaldifeat fbank computer mel filter banks by creating triangular filters and applying them to the FFT bins. I am not sure if there is any more to it...perhabs, I need to closely study the code