Hello, training XTTSv2 leads to weird training lags - training gets stuck with no errors
with using DDP
x6 RTX a6000 and 512GB RAM
Here is monitoring GPU load graph. Purple - gpu0, green - gpu1 (all the rest GPUs behave like gpu1)
Without DDP
Tried different dataset sizes - 2500hrs, 250hrs - result remains the same
I think there's some kind of error in Trainer or in xtts scripts maybe, don't know where to dig, thank you
no swap memory usage, no cpu overloading, no RAM overloading (by clearml, htop and top at least)
disk is fast NVME
Describe the bug
Hello, training XTTSv2 leads to weird training lags - training gets stuck with no errors
with using DDP x6 RTX a6000 and 512GB RAM Here is monitoring GPU load graph. Purple - gpu0, green - gpu1 (all the rest GPUs behave like gpu1)![image](https://github.com/idiap/coqui-ai-TTS/assets/69078939/c0f3405b-a1d8-496c-b49c-6e8d2e6d1142)
Without DDP![image](https://github.com/idiap/coqui-ai-TTS/assets/69078939/c9b25da1-3086-4acb-800b-e5cac9de4197)
Tried different dataset sizes - 2500hrs, 250hrs - result remains the same
I think there's some kind of error in Trainer or in xtts scripts maybe, don't know where to dig, thank you no swap memory usage, no cpu overloading, no RAM overloading (by clearml, htop and top at least) disk is fast NVME
To Reproduce
python -m trainer.distribute --script recipes/ljspeech/xtts_v2/train_gpt_xtts.py --gpus 0,1,2,3,4,5 python -m trainer.distribute --script recipes/ljspeech/xtts_v2/train_gpt_xtts.py --gpus 0,1 python3 recipes/ljspeech/xtts_v2/train_gpt_xtts.py
Expected behavior
No response
Logs
No response
Environment
Additional context
No response