I train tortoise with 10w hours of chinese audio, and then finetune it with 20 hours dataset of a female voice.
I tried:
1.lr = 0.00001, finetune all the parameters
2.lr = 0.00001, finetune using lora
3.lr = 0.00001, freeze part of the model and only finetune few layers
Thought the synthesized waves sounds good, but the robustness declained.
Repeat sometimes happens especially for long sentences.
For example, the input text is "I think .. blabla ......, I am happy",
the syntheszied wave is "I think .. blabla ......, I am happy, I am happy"
I train tortoise with 10w hours of chinese audio, and then finetune it with 20 hours dataset of a female voice. I tried:
1.lr = 0.00001, finetune all the parameters 2.lr = 0.00001, finetune using lora 3.lr = 0.00001, freeze part of the model and only finetune few layers
Thought the synthesized waves sounds good, but the robustness declained. Repeat sometimes happens especially for long sentences. For example, the input text is "I think .. blabla ......, I am happy", the syntheszied wave is "I think .. blabla ......, I am happy, I am happy"
How to finetune to keep robustness?