dunky11
commented
2 years ago Thanks for opening the issue! Pre-training the model will take me roughly one more week. Afterward, I will refactor the code, and get the project into a usable state, then I will implement this into coqui, so it will probably take me 6+ weeks.
Some info about the model: It's based upon DelightfulTTS with some modifications. Many components of the model weren't fully explained in the paper. They especially didn't get into details about both the phoneme and utterance level prosody encoders as well as hyperparameters used, so their implementations were heavily influenced by Comprehensive-Transformer-TTS. The model also uses a different scheme to provide both language and speaker embeddings. The scheme of DelightfulTTS may have worked for the blizzard challenge but didn't work when using more speakers/languages.
For the G2P model, I used DeepPhonemizer, which implements Transformer-based Grapheme-to-Phoneme Conversion, and increased the parameter count to ~23M. A single G2P model is trained on the global phone set of Montreal Forced Aligner in the following languages:
- English
- German
- French
- Castilian Spanish
- Russian
- Polish
- Bulgarian
- Czech
- Croatian
- European Portuguese
- Swedish
- Thai
- Turkish
- Ukrainian
Also, I increased the parameter count of DelightfulTTS to ~120M, otherwise, it would underfit the dataset. The dataset is ~20% stuff from public datasets like LibriTTS (100h and 360h split) and VCTK and ~80% stuff crawled by me. If you want to see some statistics about the dataset, you can click here.
The purpose of the model is to be fine-tuned on smaller datasets. It should provide a way to create TTS models in languages with limited data. It can also be used to code-switch: Since the model was pre-trained in English, German, French, Spanish, Russian and Polish voices it can be used to fine-tune the model on an English voice and then make it speak the other languages.
The parameter count may seem intimidating for a TTS model, but it can be fine-tuned without a problem on 6GB of VRAM using gradient accumulation. Also since the architecture is FastSpeech-based, and avoids autoregression, both training and inference are relatively quick and stable.
UnivNet is used as vocoder, but any vocoder that shares its STFT configuration should do its job.
In the future, I will further increase the size of the dataset, especially for the languages which contain no data yet. I also plan on further increasing the size of the model, since the current model still underfits the dataset. Then I will try to create a smaller model using knowledge distillation.
I really hope the model turns out fine, I will probably fine-tune it tomorrow on an English single-speaker dataset to check how well it speaks in the other languages, even though it hasn't fully pre-trained, since I'm always impatient :)
If you are interested in the progress, check out VoiceSmith (still a WIP) which provides a GUI to fine-tune the multilingual model and preprocess multilingual TTS datasets.
erogol
lucasjinreal
neurlang
stale[bot]
loganhart02
agilebean
ludwikbukowski
raul-parada
iamkhalidbashir
catselectro
Paper: https://arxiv.org/abs/2110.12612
👑 @loganhart420 is going to do the heavy lifting !!!
We can discuss here how we want to go about it.