search

mravanelli / pytorch-kaldi

pytorch-kaldi is a project for developing state-of-the-art DNN/RNN hybrid speech recognition systems. The DNN part is managed by pytorch, while feature extraction, label computation, and decoding are performed with the kaldi toolkit.
2.36k stars 446 forks source link

Chunk Mean and Variance Normalization #233

Closed timolohrenz closed 4 years ago

timolohrenz commented 4 years ago

Dear PyTorch-Kaldi Team,

in the recent week, I tried to deploy one of your trained TIMIT models in our own framework and was wondering why I won't be able to achieve the same PER as when I evaluate the model with the pytorch-kaldi toolkit as I always was about 5% absolute below the PER of PyTorch-Kaldi. Instead of using chunks or batches we call the model "one-sequence-at-a-time".

I double-checked if the module call gets the same arguments as in PyTorch-Kaldi and if the net inputs are the same features, use the same precision, got the right dimensions, and if the outputs of the subnets are correctly passed on etc. etc., but the final net outputs always looked slightly different than in PyTorch-Kaldi

In the end the difference was due to the function load_chunk which always normalizes means and variances of the input features over the whole chunk. Before I found this line (should've looked there earlier -.-) I thought all normalization were done by the Kaldi functions (fea_opts=)

Even though this isn't an issue by itself, as this processing is also mentioned in the PyTorch-Kaldi paper, it might help others to raise more awareness of this function and maybe make it an configurable option, as normalizing over the whole test-set is not always possible when deploying a model.

Best regards, Timo

TParcollet commented 4 years ago

This is a good point. In SpeechBrain normalisation is a specific and well-detailed step. We won't add such an option in PyTorch-Kaldi at this stage of the process. But thank you for your experience that could help others !

timolohrenz commented 4 years ago

Thanks for your quick response, I am excited to get my hands on Speechbrain as soon as its public.

Keep on the good work!

TParcollet commented 4 years ago

Should be around this year Fall :D