search

RVC-Boss / GPT-SoVITS

1 min voice data can also be used to train a good TTS model! (few shot voice cloning)
MIT License
34.4k stars 3.95k forks source link

loss suddenly increasing after some training steps #1264

Closed pashanitw closed 2 months ago

pashanitw commented 3 months ago

Hello, Thanks for the great codebase. i am working on training the model for hindi lang. i have 4k hours of training data. so i am training it from scratch. i changed the hubert model and using hindi bert models instead. then i am training the model from scratch s2_train. but loss suddenly increasing after some steps. i attached the screenshot of training losses below image

RVC-Boss commented 3 months ago

when meeting loss suddenly increasing issue, maybe fp32 training can fix it

pashanitw commented 2 months ago

yes fp32 worked thanks !