search

erew123 / alltalk_tts

AllTalk is based on the Coqui TTS engine, similar to the Coqui_tts extension for Text generation webUI, however supports a variety of advanced features, such as a settings page, low VRAM support, DeepSpeed, narrator, model finetuning, custom models, wav file maintenance. It can also be used with 3rd Party software via JSON calls.
GNU Affero General Public License v3.0
1.17k stars 123 forks source link

Continuing where you left off. No Training effect. #280

Closed RenNagasaki closed 4 months ago

RenNagasaki commented 4 months ago

Hey. If you continue a training from before the training doesnt work. grafik

He doesnt seem to restore the old averages so he can't discern if the next epoch is better than the old. He also doesnt restore the amounts of epochs that already ran.

erew123 commented 4 months ago

Hi @RenNagasaki You can look at the features that were added here https://github.com/erew123/alltalk_tts/pull/255

The continuing where you left off is more of a crash recovery than anything else. The actual values used in the previous training session arent stored anywhere by Coqui's script and there is no way to fully compare the two model files in the training session the best_modelXXX.pth and the best_model.pth.

Really all this does it pick up from the last best_modelXXX.pth model number and allow you to re-pick your training values/configureation and continue training at the last epoch file. The alternative is to "compact and move model" at the end of the training, which gives you the best_model.pth (second to last epoch) and then just train that model from there onwards.

In effect, using that setting is kicking off a new training session, with new setup/config values, its just allowing you to use the uncompressed/finalised model for the training session.

Hope that explains it.