search

coqui-ai / TTS

🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production
http://coqui.ai
Mozilla Public License 2.0
31.75k stars 3.8k forks source link

Distributed training #3802

Open thanhlong1997 opened 5 days ago

thanhlong1997 commented 5 days ago

Hi, I want to train your XTTS-v2 in distributed mode since my data is too large. How can I do that ? thank you!!!

NikitaKononov commented 1 day ago

Hi, I want to train your XTTS-v2 in distributed mode since my data is too large. How can I do that ? thank you!!!

python -m trainer.distribute --script recipes/ljspeech/xtts_v2/train_gpt_xtts.py --gpus 0,1,2,3,4,5

But xtts scripts or Trainer framework is laggy and it doesn't work properly< you can check my issue

NikitaKononov commented 4 hours ago

tried num_workers=0, >0, MP_THREADS_NUM and so on, nothing helps lots of ram and shared memory