Number of Words in Test Set

[xiaoxiao26-zz]

Hi, is there an official number for the final number of words in the test set for scoring?

WeNet results say there are 19928 sentences and 390656 words: https://github.com/wenet-e2e/wenet/tree/main/examples/gigaspeech/s0

Kaldi results say there are 390721 words: https://github.com/kaldi-asr/kaldi/blob/master/egs/gigaspeech/s5/RESULTS

I am using my own toolkit, and after removing the noise/punc tags and running the provided scoring script, I get 19930 sentences and 390744 words. Note I do match WeNet's number of words/sentences on the dev set.

Although these are small differences, I guess this still matters for final reporting. It would be super helpful if:

1. There was an official number of sentences/words, Kaldi vs WeNet seems to suggest there are inconsistencies.
2. To aid with debugging, a file listing the final references post filtering + normalization for each segment would be helpful. This way people can easily check if they are evaluating on the exact same reference, and if not where the filtering/normalization went wrong.

Thank you!

[chenguoguo]

Thanks, we will get back to this.

[dophist]

Another reference of GigaSpeech EvaluationSets numbers can also be found here in Espnet recipe, it is 19930 utts & 390744 words.

Guoguo has assigned @chaisz19 to confirm these numbers, something that I can think of:

1. some downstream toolkits might filter long utterances(say longer than 20 sec). GigaSpeech utterances are between [0.5, 20] seconds, but float-point rounding error(around 20) may be a source of inconsistency during utterance filtering.
2. After text post-processing, some utterances might have empty reference(say, an utterance with raw reference 'ah um'), I'm not sure Kaldi's compute-wer and sclite handle denominator numbers exactly the same way in this situation.

Given the large scale of GigaSpeech evaluation sets, this shouldn't have any significant impact on final WER, but it's good though that we can confirm on these numbers. So let's wait for Shuzhou's confirmation.

By the way, @xiaoxiao26 it would be great if you can provide the md5 of your local GigaSpeech.json, so we can make sure the investigation is on the right basis.

[xiaoxiao26-zz]

Thanks @dophist, good to know that the Espnet recipe exactly matches my numbers. md5sum returns the following:

19c777dc296ff3eb714bc677a80620a3 data/GigaSpeech.json

[dophist]

Leave this thread open for documentation purpose. As long as the MD5 is consistent with above, everything should be fine.