Open MetroCat69 opened 5 months ago
in order to use the old text cleaning function just switch normalizer = clean_text to normalizer = whisper.normalizers.BasicTextNormalizer() in the func evaluate_model(
We chose Whisper's BasicTextNormalizer to have an apples-to-apples comparison when analyzing Whisper's results on our datasets. At a minimum, it makes sense to keep both evaluation methods, having BasicTextNormalizer as the default but allowing a switch to the second method.
What is the actual difference between both methods? Which cases did you find where BasicTextNormalizer was subpar? It would be good to know what type of improvements we're expecting.
for example when the difference is different punctuation. for example using the old algorithm I love trains! is different that I love trains. and you will get a wer of .33 Using the improved text cleaner the wer is be 0
Maybe we should do one text with the original text cleaner and one test with the improved text cleaner
in order to use the old text cleaning function just switch normalizer = clean_text to normalizer = whisper.normalizers.BasicTextNormalizer() in the func evaluate_model(