search

mlcommons / training

Reference implementations of MLPerf™ training benchmarks
https://mlcommons.org/en/groups/training
Apache License 2.0
1.58k stars 549 forks source link

[RNN-T] Reference has dithering included with validation pipeline #495

Closed qpjaada closed 1 year ago

qpjaada commented 3 years ago

A non-zero value of dither_coeff is specified for feature extraction during validation here: https://github.com/mlcommons/training/blob/master/rnn_speech_recognition/pytorch/configs/baseline_v3-1023sp.yaml#L38

This leads to non-deterministic results during validation because of randomness here: https://github.com/mlcommons/training/blob/master/rnn_speech_recognition/pytorch/common/data/dali/pipeline.py#L197

Seems like for validation-pipeline, dithering should be disabled.

mwawrzos commented 3 years ago

We had an internal discussion about dither in the validation pipeline, and the conclusion was, that

dither is just small noise added to even out quantization artifacts. And it should not change WER. https://kaldi-asr.org/doc/faq.html#indeterminacy_in_feature_extraction

so this is not considered as an augmentation and represents industry use.