aishell3处理：使用mfa官方dict和声学模型处理aishell3

tuntun990606 commented 1 year ago

原作给的代码，生成的lab文件是拼音的，但是如果使用mfa官方方法直接对拼音对齐的话，由于mfa官方提供的拼音词典不正确，会导致textgrid里的phone全是spn。将生成lab文件的代码/preprocessor/preprocessor.py中的：text = text.split(" ")[1::2] 改成text = text.split(" ")[0::2] ，这样获得的lab是中文数据，再用mfa给的词典和声学模型对齐就可以了

tuntun990606 commented 1 year ago

原作给的应该是拼音作为原句元辅音拆开作为音素经过处理后的train.txt内容像这样：SSB05440297|SSB0544|{w uei4 j ia1 y i5}|wei4 jia1 yi5 但是如果使用mfa官方方法直接对拼音对其的话，由于mfa官方提供的拼音词典不正确，会导致textgrid里的phone全是spn。所以改用生成文字到音素，会避免这个错误。（不使用mfa提供的pinyin.dict 使用mfa提供的其他字典） train.txt文件的内容就会像这样：SSB01120099|SSB0112|{m ej˧˥ tʰ i˨˩˦ ʂ u˥˥ tɕ i˧˥ j ow˨˩˦ ʂ ə˧˥ n m ə˩}|媒体书籍有什么| （之后就只需要把新的这些音素替换到代码text/pinyin里就可以了

tuntun990606 commented 1 year ago

"ai˥˥","ai˥˩","ai˦","ai˧˥","ai˨","ai˨˩˦","ai˩","au˥˥","au˥˩","au˦","au˧˥","au˨","au˨˩˦","au˩","a˥˥","a˥˩","a˦","a˧˥","a˨","a˨˩˦","a˩","ei˥˥","ei˥˩","ei˦","ei˧˥","ei˨","ei˨˩˦","ei˩","e˥˥","e˥˩","e˦","e˧˥","e˨","e˨˩˦","e˩","f","i˥˥","i˥˩","i˦","i˧˥","i˨","i˨˩˦","i˩","j","k","kʰ","l","m","n","ou˥˥","ou˥˩","ou˦","ou˧˥","ou˨","ou˨˩˦","ou˩","o˥˥","o˥˩","o˦","o˧˥","o˨","o˨˩˦","o˩","p","pʰ","s","t","ts","tsʰ","tɕ","tɕʰ","tʰ","u˥˥","u˥˩","u˦","u˧˥","u˨","u˨˩˦","u˩","w","x","y˥˥","y˥˩","y˦","y˧˥","y˨","y˨˩˦","y˩","z̩˥˥","z̩˥˩","z̩˦","z̩˧˥","z̩˨","z̩˨˩˦","z̩˩","ŋ","ŋ̍˧˥","ɕ","ə˥˥","ə˥˩","ə˦","ə˧˥","ə˨","ə˨˩˦","ə˩","ɥ","ɻ","ʂ","ʈʂ","ʈʂʰ","ʐ","ʐ̩˥˥","ʐ̩˥˩","ʐ̩˦","ʐ̩˧˥","ʐ̩˨","ʐ̩˨˩˦","ʐ̩˩","ʔ","ow˥˩", "aj˨˩˦", "aw˥˥", "ej˥˩", "aw˧˥","aj˥˩", "ej˧˥", "ow˨˩˦","aw˨˩˦","ow˧˥", "ej˨˩˦", "aw˥˩", "aj˧˥", "aj˥˥","ow˥˥","ej˥˥","ow˥˥","ow˨" 这是所有音素

xuriliuhen commented 1 year ago

我是按照 https://montreal-forced-aligner.readthedocs.io/en/latest/first_steps/index.html#first-steps-align-pretrained 里面case3的流程跑的。最开始是拿AISHELL-3的数据集重新训练了pinyin方式的音素词典和声学模型，但是同样的对齐模型，用自己采集的中文数据集训练后的效果不清楚，没AISHELL-3数据集训练的效果好，是不是因为pinyin的对齐模型没官方提供的其他模型对齐效果好呀？

xuriliuhen commented 1 year ago

我安装你的方法跑了AISHELL3的数据集和自己采集的数据集，AISHELL3的语音合成效果要好，自己采集的数据集合成后听着很多杂音，请问这个是什么问题呢，自己的数据集本身也是在安静环境下采集并没有很多杂音的。

tuntun990606 commented 1 year ago

我没用过自己跑的mfa，但听别人说自己训得速度会慢一点？

tuntun990606 commented 1 year ago

不好意思，我没尝试过使用别的数据集，是不是采集率的问题？（如果您之后解决了的话麻烦也回复我一下，我很想知道）

hhm853610070 commented 1 year ago

请问你有没有尝试使用LJSpeech_paper那种配置？就是pitch的量化方式为log，pitch和energy都是帧级别特征，然后都不normalization，就是原文那种配置，我试了这种发现损失函数震荡，特别是duration的loss一直震荡，合成的音频质量非常差，有奇怪的杂音和错别读音。

1417889711 commented 1 year ago

请问这个问题解决了吗我也是用自己的数据集跑的效果不好

xuriliuhen commented 1 year ago

不确定这个具体是什么问题了，可能是没有预训练模型吧。我现在改用PaddleSpeech了，里面也包含FastSpeech2的具体实现，还提供了AISHELL3的预训练模型，可以把自己数据在提供的预训练模型上训练，效果比这个项目的效果要好很多。

sunnnnnnnny commented 11 months ago

你好，怎么把mfa得到的这样的音素映射到text/pinyin里呢，有参考资料吗

tuntun990606 commented 11 months ago

mfa官网的词典网页里就有 https://mfa-models.readthedocs.io/en/latest/dictionary/index.html 可能会缺几个跑的过程中遍历一下加进去就行了

tuntun990606 commented 11 months ago

sunnnnnnnny commented 11 months ago

这个字典我知道，这个字典是char/word和mfa_phone的对应关系，我好奇怎么对齐拼音和mfa_phone的对应关系，能加个qq交流下吗？863406650

tuntun990606 commented 11 months ago

我就是直接用char/phone训练的，我前面有写https://github.com/ming024/FastSpeech2/issues/188#issue-1579260899 没有使用拼音训练过不好意思哈～

sunnnnnnnny commented 11 months ago

但是这样没法解决多音字的问题，输入句子根据字典转换成mfa_phone的时候，一个字的读音是固定的，你有注意到这个问题吗

tuntun990606 commented 11 months ago

我有注意到，但我的方向是其他小语种，使用aishell3是为了听一下合成效果，中文文本正则化这方面没深入了解

ming024 / FastSpeech2

aishell3处理：使用mfa官方dict和声学模型处理aishell3 #188