search

HirCoir / Piper-tts

https://tts.hircoir.eu.org/
1 stars 2 forks source link

Multi-GPU training possible? #2

Closed KiON-GiON closed 5 months ago

KiON-GiON commented 5 months ago

Since this script is made for Kaggle, they have a T4 x2 GPU. This could speed the training and finetuning process of a voice. I tried to do it, modifying the option --devices 1 to --devices 2, but I got no success. Have you tried it?

Thanks!

HirCoir commented 5 months ago

Since this script is made for Kaggle, they have a T4 x2 GPU. This could speed the training and finetuning process of a voice. I tried to do it, modifying the option --devices 1 to --devices 2, but I got no success. Have you tried it?

Thanks!

I have tried to do the same but in Kaggle it is not possible to use both GPUs, I have consulted with other developers and they have the same problem

KiON-GiON commented 5 months ago

Oh I see. It is really a shame.

KiON-GiON commented 4 weeks ago

Hi there @HirCoir. I made a personal branch, modified a bit the Piper trainer to allow Distributed Data Parallel, so the two GPUs work now. In the Kaggle notebook, I did this too:

!pip install -q cython>=0.29.0 piper-phonemize==1.1.0 librosa>=0.9.2 numpy==1.24 onnxruntime>=1.11.0 pytorch-lightning~=1.9.0 torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118
!pip install --force-reinstall numpy==1.26.4
# fixing recent compativility isswes:+
!pip install -q torchmetrics==0.11.4

However, keep in mind you won't suddenly get 32 GB VRAM, it just means it will speed up the training batches. You're still limited to the memory each GPU has. Also I modified a bit the logger because for some reason Kaggle didn't print information like Colab. Please test if this works for you.