yl4579
commented
1 year ago I did try training for other languages including Mandarin, Japanese, Hindi etc., though it requires a few changes:
- You need to phonemize Chinese into IPAs. You can use either https://github.com/bootphon/phonemizer or a look-up table to replace Chinese characters into IPAs. The pre-trained text aligner already includes AiShell (Mandarin dataset), with the following IPA conversion table from Pinyin. It may be slightly different from phonemizer, as it didn't work for me for Chinese.
ba pˈa bo pˈwɔ bai pˈaɪ bei pˈeɪ bao pˈaʊ ban pˈan ben pˈən bang pˈɑŋ beng pˈəŋ bi pˈi biao pˈjaʊ bie pˈjɛ bian pˈjɛn bin pˈin bing pˈiŋ bu pˈu pa pʰˈa po pʰˈwɔ pai pʰˈaɪ pei pʰˈeɪ pao pʰˈaʊ pou pʰˈoʊ pan pʰˈan pen pʰˈən pang pʰˈɑŋ peng pʰˈəŋ pi pʰˈi piao pʰˈjaʊ pie pʰˈjɛ pian pʰˈjɛn pin pʰˈin ping pʰˈiŋ pu pʰˈu ma mˈa me mˈɤ mo mˈwɔ mai mˈaɪ mei mˈeɪ mao mˈaʊ mou mˈoʊ man mˈan men mˈən mang mˈɑŋ meng mˈəŋ mi mˈi miao mˈjaʊ mie mˈjɛ miu mˈju mian mˈjɛn min mˈin ming mˈiŋ mu mˈu fa fˈa fo fˈwɔ fei fˈeɪ fou fˈoʊ fan fˈan fen fˈən fang fˈɑŋ feng fˈəŋ fu fˈu da tˈa de tˈɤ dai tˈaɪ dei tˈeɪ dao tˈaʊ dou tˈoʊ dan tˈan dang tˈɑŋ deng tˈəŋ dong tˈʊŋ di tˈi diao tˈjaʊ die tˈjɛ diu tˈjoʊ dian tˈjɛn ding tˈiŋ du tˈu duo tˈwɔ dui tˈweɪ duan tˈwan dun tˈwən ta tʰˈa te tʰˈɤ tai tʰˈaɪ tao tʰˈaʊ tou tʰˈoʊ tan tʰˈan tang tʰˈɑŋ teng tʰˈəŋ tong tʰˈʊŋ ti tʰˈi tiao tʰˈjaʊ tie tʰˈjɛ tian tʰˈjɛn ting tʰˈiŋ tu tʰˈu tuo tʰˈwɔ tui tʰˈweɪ tuan tʰˈwan tun tʰˈwən na nˈa ne nˈɤ nai nˈaɪ nei nˈeɪ nao nˈaʊ nou nˈoʊ nan nˈan nen nˈən nang nˈɑŋ neng nˈəŋ nong nˈʊŋ ni nˈi niao nˈjaʊ nie nˈjɛ niu nˈjoʊ nian nˈjɛn nin nˈin niang nˈiɑŋ ning nˈiŋ nu nˈu nuo nˈwɔ nuan nˈwan nü nˈy nüe nˈyɛ la lˈa le lˈɤ lai lˈaɪ lei lˈeɪ lao lˈaʊ lou lˈoʊ lan lˈan lang lˈɑŋ leng lˈəŋ long lˈʊŋ li lˈi lia lˈja liao lˈjaʊ lie lˈjɛ liu lˈjoʊ lian lˈjɛn lin lˈin liang lˈiɑŋ ling lˈiŋ lu lˈu luo lˈwɔ luan lˈwan lun lˈwən lü lˈy lüe lˈyɛ za tsˈa ze tsˈɤ zi tsˈɹ zai tsˈaɪ zei tsˈeɪ zao tsˈaʊ zou tsˈoʊ zan tsˈan zen tsˈən zang tsˈɑŋ zeng tsˈəŋ zong tsˈʊŋ zu tsˈu zuo tsˈwɔ zui tsˈweɪ zuan tsˈwan zun tsˈwən ca tsʰˈa ce tsʰˈɤ ci tsʰˈɹ cai tsʰˈaɪ cao tsʰˈaʊ cou tsʰˈoʊ can tsʰˈan cen tsʰˈən cang tsʰˈɑŋ ceng tsʰˈəŋ cong tsʰˈʊŋ cu tsʰˈu cuo tsʰˈwɔ cui tsʰˈweɪ cuan tsʰˈwan cun tsʰˈwən sa sˈa se sˈɤ si sˈɹ sai sˈaɪ sao sˈaʊ sou sˈoʊ san sˈan sen sˈən sang sˈɑŋ seng sˈeŋ song sˈʊŋ su sˈu suo sˈwɔ sui sˈweɪ suan sˈwan sun sˈwən zha ʈʂˈa zhe ʈʂˈɤ zhi ʈʂˈʐ zhai ʈʂˈaɪ zhei ʈʂˈeɪ zhao ʈʂˈaʊ zhou ʈʂˈoʊ zhan ʈʂˈan zhen ʈʂˈən zhang ʈʂˈɑŋ zheng ʈʂˈəŋ zhong ʈʂˈʊŋ zhu ʈʂˈu zhua ʈʂˈwa zhuo ʈʂˈwɔ zhuai ʈʂˈwaɪ zhui ʈʂˈweɪ zhuan ʈʂˈwan zhun ʈʂˈwən zhuang ʈʂˈwɑŋ cha ʈʂʰˈa che ʈʂʰˈɤ chi ʈʂʰˈʐ chai ʈʂʰˈaɪ chao ʈʂʰˈaʊ chou ʈʂʰˈoʊ chan ʈʂʰˈan chen ʈʂʰˈən chang ʈʂʰˈɑŋ cheng ʈʂʰˈəŋ chong ʈʂʰˈʊŋ chu ʈʂʰˈu chua ʈʂʰˈwa chuo ʈʂʰˈwɔ chuai ʈʂʰˈwaɪ chui ʈʂʰˈweɪ chuan ʈʂʰˈwan chun ʈʂʰˈwən chuang ʈʂʰˈwɑŋ sha ʂˈa she ʂˈɤ shi ʂˈʐ shai ʂˈaɪ shei ʂˈeɪ shao ʂˈaʊ shou ʂˈoʊ shan ʂˈan shen ʂˈən shang ʂˈɑŋ sheng ʂˈəŋ shu ʂˈu shua ʂˈwa shuo ʂˈwɔ shuai ʂˈwaɪ shui ʂˈweɪ shuan ʂˈwan shun ʂˈwən shuang ʂˈwɑŋ re ɹˈɤ ri ɹˈʐ rao ɹˈaʊ rou ɹˈoʊ ran ɹˈan ren ɹˈən rang ɹˈɑŋ reng ɹˈəŋ rong ɹˈʊŋ ru ɹˈu ruo ɹˈwɔ rui ɹˈweɪ ruan ɹˈwan run ɹˈwən ji tɕˈi jia tɕˈja jiao tɕˈjaʊ jie tɕˈjɛ jiu tɕˈjoʊ jian tɕˈjɛn jin tɕˈin jiang tɕˈiɑŋ jing tɕˈiŋ jiong tɕˈjʊŋ ju tɕˈy jue tɕˈyɛ juan tɕˈyɛn jun tɕˈyn qi tɕʰˈi qia tɕʰˈja qiao tɕʰˈjaʊ qie tɕʰˈjɛ qiu tɕʰˈjoʊ qian tɕʰˈjɛn qin tɕʰˈin qiang tɕʰˈjɑŋ qing tɕʰˈiŋ qiong tɕʰˈjʊŋ qu tɕʰˈy que tɕʰˈyɛ quan tɕʰˈyɛn qun tɕʰˈyn xi ɕˈi xia ɕˈja xiao ɕˈjaʊ xie ɕˈjɛ xiu ɕˈjoʊ xian ɕˈjɛn xin ɕˈin xiang ɕˈiɑŋ xing ɕˈiŋ xiong ɕˈjʊŋ xu ɕˈy xue ɕˈyɛ xuan ɕˈyɛn xun ɕˈyn ga kˈa ge kˈɤ gai kˈaɪ gei kˈeɪ gao kˈaʊ gou kˈoʊ gan kˈan gen kˈən gang kˈɑŋ geng kˈəŋ gong kˈʊŋ gu kˈu gua kˈwa guo kˈwɔ guai kˈwaɪ gui kˈweɪ guan kˈwan gun kˈwən guang kˈwɑŋ ka kʰˈa ke kʰˈɤ kai kʰˈaɪ kei kʰˈeɪ kao kʰˈaʊ kou kʰˈoʊ kan kʰˈan ken kʰˈən kang kʰˈɑŋ keng kʰˈəŋ kong kʰˈʊŋ ku kʰˈu kua kʰˈwa kuo kʰˈwɔ kuai kʰˈwaɪ kui kʰˈweɪ kuan kʰˈwan kun kʰˈwən kuang kʰˈwɑŋ ha xˈa he xˈɤ hai xˈaɪ hei xˈeɪ hao xˈaʊ hou xˈoʊ han xˈan hen xˈən hang xˈɑŋ heng xˈəŋ hong xˈʊŋ hu xˈu hua xˈwa huo xˈwɔ huai xˈwaɪ hui xˈweɪ huan xˈwan hun xˈwən huang xˈwɑŋ a ˈa o ˈo e ˈɤ er ˈɚ ai ˈaɪ ei ˈeɪ ao ˈaʊ ou ˈoʊ an ˈan en ˈən ang ˈɑŋ eng ˈəŋ yi ˈi ya jˈa yao jˈaʊ ye jˈɛ you jˈoʊ yan jˈɛn yin ˈin yang jˈɑŋ ying ˈiŋ yong ˈjʊŋ wu ˈu wa wˈa wo wˈɔ wai wˈaɪ wei wˈeɪ wan wˈan wen wˈən wang wˈɑŋ weng wˈəŋ yu ˈy yue ɥˈɛ yuan ɥˈɛn yun ɥˈn hair xˈaɹ dianr tˈjaɹ wanr wˈaɹ nar nˈaɹ yanr jˈaɹ huor xˈwɔɹ duanr tˈwaɹ lir lˈjɚ huir xˈwjɚ zher ʈʂˈɚ dour xˈɔɹ weir wˈɚ kuair kʰˈwaɹ guanr gˈwɐʴ shir ʂˈɚ yuanr ɥˈɚ jianr tɕˈjɚ her xˈɚ jiar tɕˈjaɹ
bor pˈwɔɹ xir ɕˈɚ bianr pˈjɚ fenr fˈɚ wenr wˈɚ der tˈɚ por pʰˈwɔɹ yuer ɥˈɚ mingr mˈjɚ char ʈʂʰˈaɹ xingr ɕˈjɚ zhour ʈʂˈoʊɹ shour ʂˈoʊɹ ter tʰˈɚ yingr ˈjɚ paor pʰˈaɹ fangr fˈɑɹ jingr tɕˈjɚ shur ʂˈuɹ qunr tɕʰˈyɹ hur xˈuɹ miaor mˈjaʊɹ biaor pˈjaʊɹ zhengr ʈʂˈɚ gour kˈoʊɹ pair pʰˈaɹ renr ɹˈɚ gaor kˈaʊɹ lo lˈoʊ tuir tʰˈwɚ huanr xˈwaɹ genr kˈɚ nvr nˈyɹ qianr tɕʰˈjɚ hangr xˈɑɹ chenr ʈʂʰˈɚ den tˈɚ lar lˈaɹ niur nˈjoʊɹ liur lˈjoʊɹ tunr tʰˈwɚ lunr lˈwɚ tour tʰˈoʊɹ hour xˈoʊɹ tianr tʰˈjɚ mianr mˈjɚ mar mˈaɹ pianr pʰˈjɚ maor mˈaʊɹ cair tsʰˈɚ far fˈaɹ shuor ʂˈwɔɹ kanr kʰˈaɹ banr pˈaɹ ger kˈɚ sher ʂˈɚ gunr kˈwɚ beir pˈɚ chuanr ʈʂʰˈwɚ bar pˈaɹ cunr tsʰˈwɚ tiaor tʰˈjaʊɹ shuar ʂˈwaɹ tur tʰˈuɹ zhaor ʈʂˈaʊɹ cher ʈʂʰˈɚ menr mˈɚ qingr tɕʰˈjɚ shanr ʂˈaɹ mor mˈwɔɹ zhur ʈʂˈuɹ wangr wˈɑɹ zhunr ʈʂˈwɚ zhir ʈʂˈɚ haor xˈaʊɹ shuir ʂˈwɚ guor kˈwɔɹ zaor tsˈaʊɹ juanr tɕˈyɚ jiar tɕˈjaɹ xiaor ɕˈjaʊɹ suor sˈwɔɹ shaor ʂˈaʊɹ yir ˈɚ dir tˈɚ ganr kˈaɹ duir tˈwɚ taor tʰˈaʊɹ lianr lˈjɚ benr pˈɚ fanr fˈaɹ xuer ɕˈyɚ pur pʰˈuɹ jinr tɕˈɚ kour kʰˈoʊɹ ker kʰˈɚ mur mˈuɹ liaor lˈjaʊɹ juer tɕˈyɚ your jˈoʊɹ xianr ɕˈjɚ quanr tɕʰˈyɚ yo jˈoʊ sanr sˈaɹ zhuor ʈʂˈwɔɹ tuor tʰˈwɔɹ naor nˈaʊɹ dar tˈaɹ fur fˈuɹ dunr tˈwɚ langr lˈɑɹ dair tˈaɹ huar xˈwaɹ yangr jˈɑɹ
2. You need to add a tone embedding for languages like Chinese and Japanese. For example, replacing the [ProsodyPredictor ](https://github.com/yl4579/StyleTTS/blob/main/models.py#L503) with the following code (i.e. concatenating the prosody embedding with the text embedding):
```python
class ProsodyPredictor(nn.Module):
def __init__(self, n_prods, prod_embd, style_dim, d_hid, nlayers, dropout=0.1):
super().__init__()
self.embedding = nn.Embedding(n_prods, prod_embd * 2)
self.text_encoder = DurationEncoder(sty_dim=style_dim,
d_model=d_hid,
nlayers=nlayers,
dropout=dropout)
self.lstm = nn.LSTM(d_hid + style_dim, d_hid // 2, 1, batch_first=True, bidirectional=True)
self.duration_proj = LinearNorm(d_hid, 1)
self.lstm = nn.LSTM(d_hid + prod_embd * 2 + style_dim, d_hid // 2, 1, batch_first=True, bidirectional=True)
self.duration_proj = LinearNorm(d_hid, 1)
self.shared = nn.LSTM(d_hid + prod_embd * 2 + style_dim, d_hid // 2, 1, batch_first=True, bidirectional=True)
self.F0 = nn.ModuleList()
self.F0.append(AdainResBlk1d(d_hid, d_hid, style_dim, dropout_p=dropout))
self.F0.append(AdainResBlk1d(d_hid, d_hid // 2, style_dim, upsample=True, dropout_p=dropout))
self.F0.append(AdainResBlk1d(d_hid // 2, d_hid // 2, style_dim, dropout_p=dropout))
self.N = nn.ModuleList()
self.N.append(AdainResBlk1d(d_hid, d_hid, style_dim, dropout_p=dropout))
self.N.append(AdainResBlk1d(d_hid, d_hid // 2, style_dim, upsample=True, dropout_p=dropout))
self.N.append(AdainResBlk1d(d_hid // 2, d_hid // 2, style_dim, dropout_p=dropout))
self.F0_proj = nn.Conv1d(d_hid // 2, 1, 1, 1, 0)
self.N_proj = nn.Conv1d(d_hid // 2, 1, 1, 1, 0)
def forward(self, texts, prosody, style, text_lengths, alignment, m):
prosody = self.embedding(prosody)
texts = torch.cat([texts, prosody], axis=1)
d = self.text_encoder(texts, style, text_lengths, m)
batch_size = d.shape[0]
text_size = d.shape[1]
# predict duration
input_lengths = text_lengths.cpu().numpy()
x = nn.utils.rnn.pack_padded_sequence(
d, input_lengths, batch_first=True, enforce_sorted=False)
m = m.to(text_lengths.device).unsqueeze(1)
self.lstm.flatten_parameters()
x, _ = self.lstm(x)
x, _ = nn.utils.rnn.pad_packed_sequence(
x, batch_first=True)
x_pad = torch.zeros([x.shape[0], m.shape[-1], x.shape[-1]])
x_pad[:, :x.shape[1], :] = x
x = x_pad.to(x.device)
duration = self.duration_proj(nn.functional.dropout(x, 0.5, training=self.training))
en = (d.transpose(-1, -2) @ alignment)
return duration.squeeze(-1), en
def F0Ntrain(self, x, s):
x, _ = self.shared(x.transpose(-1, -2))
F0 = x.transpose(-1, -2)
for block in self.F0:
F0 = block(F0, s)
F0 = self.F0_proj(F0)
N = x.transpose(-1, -2)
for block in self.N:
N = block(N, s)
N = self.N_proj(N)
return F0.squeeze(1), N.squeeze(1)
def length_to_mask(self, lengths):
mask = torch.arange(lengths.max()).unsqueeze(0).expand(lengths.shape[0], -1).type_as(lengths)
mask = torch.gt(mask+1, lengths.unsqueeze(1))
return mask- Modify meldataset.py to return the tones for each IPA and change your train_list.txt in the following format:
data/aishell/train/wav/SSB1100/SSB11000297.wav|$ʈʂˈɑŋxˈweɪˈi ʈʂʰˈujˈɛntˈɤ tˈjɛnˈin jˈoʊʂˈənmˈɤ$|X111114444422 111113333555 44444333 33332222555X|382 data/aishell/train/wav/SSB1567/SSB15670392.wav|$ʂˈʐ fˈuʂˈʐ ʂˈʐlˈɤ ʈʂˈəŋtʰˈi jˈɛˈu$fˈaʈʂˈantˈɤ xˈɤɕˈin tɕʰˈytˈʊŋlˈi$|X444 444444 111444 222223333 44444 11133333555 2221111 111114444444X|274 data/aishell/train/wav/SSB0603/SSB06030228.wav|$xˈwɔtˈjɛn tˈəŋ tˈwɔxˈɑŋjˈɛ tɕˈiɑŋʂˈoʊ pˈwɔtɕˈi$|X333344444 3333 11112222444 1111114444 11112222X|223 data/aishell/train/wav/SSB0588/SSB05880296.wav|$ˈinɥˈɛ ˈiʂˈəŋ sˈwɔˈaɪ$|X111444 441111 3333444X|378 data/aishell/train/wav/SSB0315/SSB03150316.wav|$ʈʂʰˈuɕˈyɛʈʂˈɤ kʰˈɤ ʂˈʐˈjʊŋ tɕˈjaʊʈʂʰˈɑŋtˈɤ ˈiɕˈjɛ tʰˈjaʊʂˈəŋ$|X1111122223333 2222 3334444 444444222222555 441111 4444442222X|241 data/aishell/train/wav/SSB0631/SSB06310452.wav|$ɕˈjɛntsˈaɪ tɕˈitɕʰˈi ɕˈyɛxˈweɪ kˈənɹˈən kˈoʊtʰˈʊŋ$|X4444444444 111144444 222244444 11112222 111111111X|229 data/aishell/train/wav/SSB1935/SSB19350402.wav|$xˈwansˈwɔtˈɤ ʂˈʐ tɕʰˈyʂˈʐ lˈaʊpˈaɹtˈɤsˈwən tsˈɹ$|X444443333555 444 44444444 3333444455511111 5555X|345 data/aishell/train/wav/SSB1203/SSB12030292.wav|$pˈiɹˈu tsˈweɪtɕˈin sˈannˈjɛn tɕˈiŋˈiŋ ʈʂˈwɑŋkʰˈwɑŋ lˈiɑŋxˈaʊtˈəŋ$|X333222 44444444444 111122222 11111222 444444444444 2222222223333X|377 data/aishell/train/wav/SSB1024/SSB10240312.wav|$xˈaˈɚ pˈinʂˈʐ tˈiˈu sˈɹʈʂˈʊŋɕˈyɛtˈɤ ʈʂˈaʊpʰˈaɪ ˈy pʰˈɑŋpˈjɛn ʂˈɑŋxˈu ɕˈiɑŋpˈi$|X11133 1111444 44433 444111112222555 1111155555 33 2222211111 1111444 11111333X|231 data/jvs_ver1/jvs088/parallel100/wav24kHz16bit/VOICEACTRESS100_037.wav|$kˈomˈʲɯːɴ ɯˈa $ sˈeːnˈɯ gˈaɯˈa tˈo $ esˈo ɴ nˈɯ kˈaɯˈa nˈo $ gˈoːɽˈʲɯː tɕˈitˈeɴ tˈo nˈaʔ tˈe iɽˈɯ$|XLLLHHHHLL LLL X LLLHHHH LLLLLL LLL X LHHH H HHH LLLLLL LLL X LLLHHHHHH HHHHLLLL LLL HHHL LLL LHHHX|88 data/aishell/train/wav/SSB0671/SSB06710188.wav|$tɕˈiɑŋɕˈjɛn nˈanfˈan ˈjʊŋxˈʊŋ fˈɑŋɕˈin lˈiɑŋjˈoʊ ʈʂˈʊŋɕˈintˈjɛn$|X44444444444 22222222 33332222 44441111 222222222 11111111144444X|363 data/aishell/train/wav/SSB0380/SSB03800184.wav|$kʰˈɤ ɥˈɛxˈan tɕˈjoʊʂˈʐ tʰˈiŋpˈu tɕˈintɕʰˈy$|X3333 1114444 444444444 11111222 4444444444X|323 data/aishell/train/wav/SSB0760/SSB07600247.wav|$tˈɑŋɹˈan wˈɔ ɕˈjɛntsˈaɪ ˈitɕˈiŋ mˈeɪjˈoʊ ʈʂˈɤkˈɤ tsˈɹkˈɤ tsˈaɪkˈən nˈiʂˈwɔ ʈʂˈɤkˈɤ xˈwaɹ$|X11112222 333 4444444444 3311111 22223333 4444444 1111222 444441111 3331111 4444444 44444X|237 data/aishell/train/wav/SSB0016/SSB00160083.wav|$pˈaʂˈʐˈutˈjɛn lˈjoʊlˈiŋtɕʰˈi$|X1112222233333 44444222211111X|245where X and $ represent the SOS and EOS.
I'll leave this issue open for someone to fork the repo and modify it for Mandarin and Japanese support. I'm unfortunately too busy to work on it now.
c9412600
liuhuang31
CONGLUONG12
MMMMichaelzhang
JohnHerry
zdj97
Zhongxu-Wang
yihuitang
leminhnguyen
GuangChen2016
hdmjdp
sunnnnnnnny
SaltedSlark
mandrain support?