CUDA Memory Error During Training on k80

lmnt-com / diffwave

DiffWave is a fast, high-quality neural vocoder and waveform synthesizer.

Apache License 2.0

754 stars 111 forks source link

CUDA Memory Error During Training on k80 #18

Closed Malik7115 closed 2 years ago

Malik7115 commented 2 years ago

Hello,

I have been trying to train your awesome work on my custom dataset, however, I get the following error on a k80 irrespective of Batch Size (even tried with BS = 1).

RuntimeError: CUDA out of memory. Tried to allocate 124.00 MiB (GPU 0; 11.17 GiB total capacity; 10.65 GiB already allocated; 94.31 MiB free; 10.68 GiB reserved in total by PyTorch)

Would appreciate any help in this regard so that I might be able to put the model on training as soon as possible.

Thanks.

sharvil commented 2 years ago

I've just submitted a change that should substantially reduce memory use during training and inference. Let me know if that works out for you -- if not, there may be some other factor at play.