how to train for mandarin asr?

[MMMMichaelzhang]

if I want to train mandarin asr,dict is like thisdict.txt,and I use g2pM as phonemizer,and train.txt is like this: SSB06930002.wav | 武 wu3 术 shu4 始 shi3 终 zhong1 被 bei4 看 kan4 作 zuo4 我 wo3 国 guo2 的 de5 国 guo2 粹 cui4 | 0 I don't know how to change my format（dict.txt \train.txt） to suit this project, and how to change it in meldataset. can you help me? Thank you very much

[Charlottecuc]

You shouldn't put hanzi characters in train.txt, and it will be better if you transform the pinyin sequences into phoneme sequences as in Mandarin TTS projects (e.g. kan4-> k an4) because there are too many unique pinyins in Mandarin.

[yl4579]

I believe it should be like SSB06930002.wav | wu shu shi zhong bei kan zuo wo guo de guo cui | 0. You don't want the tones (that should be handled by the F0 model) nor the Chinese characters (they aren't phonemes) here. The ASR model should only be used for phonemes.

[MMMMichaelzhang]

Thanks ,it helps a lot!!! I am still confused about dict.txt for mandarin.my fomart is like this: "a",1 "ao",2 "an",3 "w",4 "t",5 "d",6 "n",7 or： “wa”,1 "ta",2 "na",3 "tao",4 "dao",5 "wan",6 "tan",7 which one should i use? @yl4579 @Charlottecuc

[MMMMichaelzhang]

I use g2pM as phonemizer,

sentence = "然而，他红了20年以后，他竟退出了大家的视线。" model(sentence, tone=False, char_split=False) ['ran', 'er', '，', 'ta', 'hong', 'le', '2', '0', 'nian', 'yi', 'hou', '，', 'ta', 'jing', 'tui', 'chu', 'le', 'da', 'jia', 'de', 'shi', 'xian', '。'] so the dict.txt 's format should be like this? “wa”,1 "ta",2 "na",3 "tao",4 "dao",5 "wan",6 "tan",7 @yl4579 @Charlottecuc

[Charlottecuc]

I use g2pM as phonemizer,

sentence = "然而，他红了20年以后，他竟退出了大家的视线。" model(sentence, tone=False, char_split=False) ['ran', 'er', '，', 'ta', 'hong', 'le', '2', '0', 'nian', 'yi', 'hou', '，', 'ta', 'jing', 'tui', 'chu', 'le', 'da', 'jia', 'de', 'shi', 'xian', '。'] so the dict.txt 's format should be like this? “wa”,1 "ta",2 "na",3 "tao",4 "dao",5 "wan",6 "tan",7 @yl4579 @Charlottecuc

Thanks ,it helps a lot!!! I am still confused about dict.txt for mandarin.my fomart is like this: "a",1 "ao",2 "an",3 "w",4 "t",5 "d",6 "n",7 or： “wa”,1 "ta",2 "na",3 "tao",4 "dao",5 "wan",6 "tan",7 which one should i use? @yl4579 @Charlottecuc

I use the first type of format (which is quite common when training Mandarin TTS models) because I don't want too many tokens in my dictionary . Besides, I suggest you transform "20" to "er sh i" in this context.

[Moonmore]

I believe it should be like SSB06930002.wav | wu shu shi zhong bei kan zuo wo guo de guo cui | 0. You don't want the tones (that should be handled by the F0 model) nor the Chinese characters (they aren't phonemes) here. The ASR model should only be used for phonemes.

@yl4579 I would like to ask if tones information is not added to the asr model training, then how to align the text encoder and mel. For example: (你好 ->ni3 hao3 -> n i3 h ao3), (n i h ao).

[yl4579]

@Moonmore You do not need tones for this. The pre-trained aligner in StyleTTS and StyleTTS 2 repo was already trained on Mandarin (AiShell) without tones. The conversion table between IPA and pinyin is here: https://github.com/yl4579/StyleTTS/issues/10#issuecomment-1407788234 Once you use this table to convert pinyin to IPA, you can directly use the pre-trained aligner and you don't need to train it yourself. However, if you want to train it yourself on pinyin, you do not need to include tones.

[Moonmore]

@Moonmore You do not need tones for this. The pre-trained aligner in StyleTTS and StyleTTS 2 repo was already trained on Mandarin (AiShell) without tones. The conversion table between IPA and pinyin is here: yl4579/StyleTTS#10 (comment) Once you use this table to convert pinyin to IPA, you can directly use the pre-trained aligner and you don't need to train it yourself. However, if you want to train it yourself on pinyin, you do not need to include tones.

@yl4579 thanks for your reply. I successfully used pinyin to train the model.