search

NVIDIA / mellotron

Mellotron: a multispeaker voice synthesis model based on Tacotron 2 GST that can make a voice emote and sing without emotive or singing training data
BSD 3-Clause "New" or "Revised" License
854 stars 187 forks source link

require large memory #69

Open aijianiula0601 opened 4 years ago

aijianiula0601 commented 4 years ago

Thanks for your jobs!

my environment: V100，8gpu，per gpu 32g memoy.

I train with 4 gpu use multiproc and it take away almost 32g each gpu.The batch_szie is set to 32 instead of 4*32=128. Does it need that much memory? Why not switch to dataparallel.Thank you!

CookiePPP commented 4 years ago

@aijianiula0601 VRAM (GPGPUs) or System Memory (RAM)?

https://github.com/NVIDIA/mellotron/blob/master/distributed.py#L51

The multiproc is just a slightly modified version of Dataparallel, memory usage should be the same (or very close) to pytorch's built-in copy.

aijianiula0601 commented 4 years ago

@aijianiula0601 VRAM (GPGPUs) or System Memory (RAM)?

https://github.com/NVIDIA/mellotron/blob/master/distributed.py#L51

The multiproc is just a slightly modified version of Dataparallel, memory usage should be the same (or very close) to pytorch's built-in copy.

I change it to Dataparallel.The batch size can set to 128, but train slow.

rafaelvalle commented 4 years ago

With our implementation, try decreasing the batch size.