search

erew123 / alltalk_tts

AllTalk is based on the Coqui TTS engine, similar to the Coqui_tts extension for Text generation webUI, however supports a variety of advanced features, such as a settings page, low VRAM support, DeepSpeed, narrator, model finetuning, custom models, wav file maintenance. It can also be used with 3rd Party software via JSON calls.
GNU Affero General Public License v3.0
1.06k stars 113 forks source link

finetuning: although EVAL metrics are better, the script does not save the model checkpoint as “best”. #359

Closed Mixomo closed 3 weeks ago

Mixomo commented 3 weeks ago

Since I started training XTTS, it always happened to me that even if the evaluation metrics in certain epochs are better, the script does not save the model as “best”. It seems that it only saves the “best” in the first 10 epochs (the default value of epochs), and if I want to increase the number of training epochs, it happens what I commented above. (Or otherwise, the model never seems to improve beyond the first 10 epochs, no matter how good the dataset is).

image

I also don't understand why in the LOSS graph even though it shows a decay, the AVG LOSS goes up....

image

erew123 commented 3 weeks ago

Hi @Mixomo

There are 2x updates coming out to do with re-work on parts of the training code. However, I am away from my home dealing with an urgent family matter and unable to test the updates atm.

One update dropped 8 hours ago, which is updates to the Coqui TTS engine and include code on the training scripts https://github.com/idiap/coqui-ai-TTS/releases/tag/v0.24.2

The other update is in a PR on here https://github.com/erew123/alltalk_tts/pull/288 and directly relates to the fintuning scripts.

Between the two of them, they should improve matters all around and Im just waiting to get time to test them. The Coqui one has been out of my hands, so glad to see its finally dropped and should improve the XTTS trainining.

Youre welcome to try the 24.2 update, but as I say, I have no reference on it at the moment and no way to personally test it, so cannot provide any support on it currently. You would pip install --upgrade coqui-tts in the AllTalk Python environment.

Thanks

Mixomo commented 3 weeks ago

@erew123 Thank you for your quick response! No rush! :)