search

imdanboy / jets

JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech
Apache License 2.0
104 stars 12 forks source link

GPU memory requirements #5

Open iamanigeeit opened 1 year ago

iamanigeeit commented 1 year ago

Hi @imdanboy thanks for making this (it's better than plain FS2 + Hifigan). I'm trying to train on LibriTTS-100 dataset.

I see you used 4 V100 GPUs to train. I only have a single RTX 3090 24GB and i still get CUDA out of memory with batch_bins: 1000000. Do you have the GPU usage stats?

imdanboy commented 1 year ago

Hi, batch_bins: 3000000 with 4 V100 32GB seems to use almost all memory. Thus how about trying 3000000/4 * (24/32) = 562500 Or reducing the value of segment_size which also affect memory usage.