begeekmyfriend
commented
6 years ago Hi all, I have been conducting series of ablation studies to understand the key components affecting the quantity of training dataset needed these days. Now I have trained on small dataset (~10h) and get to convergence in 10K steps based on the Keith Ito's tacotron2-work-in-progress branch. And I have released my minimal modification on my repo. Most of the code is derived from Rayhane Mamah's version.
We can see that the alignment is not very clear and consecutive because this is only the ablation study on small dataset convergence and the modification is the minimum. Later I will give some advice to improve alignments such as trimming leading and trailing silence of the audio.
In my opinion, there are two dispensable factors: The decoder RNN architecture and the normalization of mel-spectrograms. I say they are dispensable because it would fail to convergence on small dataset without either of two modifications.
First let us talk about the decoder RNN architecture. The designs between Keith and Rayhane are different. Keith used AttentionWrapper with an extra attention RNN wrapped in it and the query of the attention mechanism is the hidden states of the attention RNN. In Rayhane's design there is no attention RNN and the query is the hidden states of decoder LSTM. The modification is in rnn_wrapper.py. And this post illustrates the structure of both two decoder RNN architecture. The conclusion we can learn from it is that AttentionWrapper might be inappropriate for Tacotron model because attention RNN is redundant in decoder structure. 2-layer LSTM does the same thing and it is closer to the output frames.
Second we should focus on the normalization of mel-spectrograms. As Rayhane has illustrated, the distances of spectrograms data points in his model is bigger than Keith's, and the mean value of the distribution is zero which is known as symmetry. I find that crucial for convergence on small dataset because when I scale the range [-4, 4] to [0, 1], [0, 8] or [-1, 1] it would fail to convergence. We can see the modification in audio.py. And here comes the conclusion that [0, 1] normalization might not be the best solution for small dataset convergence.
These two components above are the key for whether we can use less dataset to get to convergence or not. However, there are still some optional methods to improve or speed up the convergence. For instance, we can conduct stricter silence trim for audio to get better alignments. Currently I am using librosa.effects.trim but the alignment seems not good enough. I am going to improve and show the results later.
Last but not least, my repo currently is only for ablation study. Please do NOT treat it as an improved Tacotron model and expect better results. You can try to use Rayhane's version for validation.
ryancwalsh
I have tried 10h dataset on Rayhane Mamah's version which is derived from this project https://github.com/Rayhane-mamah/Tacotron-2/issues/35. I wonder what improvement can result in less dataset needed and convergence at less than 5K steps.