What kind of HW is needed to run the "best performing model"?

drethage / speech-denoising-wavenet

A neural network for end-to-end speech denoising

MIT License

673 stars 165 forks source link

What kind of HW is needed to run the "best performing model"? #27

Open KDuzinkiewicz opened 5 years ago

KDuzinkiewicz commented 5 years ago

Hi, I have a setup with NVIDIA GTX 1080 card with 8GB RAM but I'm still unable to perform a successfull denoising run on the default dataset without getting an out-of-memory error from TensorFlow :( I'm currently retraining the model with a smaller "dilations" parameter value, but just out of curiosity - what kind of HW are you using to run denoising without OOM error on your side?

jordipons commented 5 years ago

We used a Titan X Pascal (12GB-VRAM).

Actually, having enough VRAM is important for these wavenet-like models.

Maybe you can check our recent paper where we explore several hyper-parameters choices with the idea to investigate that memory issue. See in Section 3 the paragraph "Wavenet: wider or deeper?": https://arxiv.org/abs/1810.12187

KDuzinkiewicz commented 5 years ago

Thanks for your answer! BTW, have you used the same card for training & denoising? My problem is that even on a setup with 2 x 10GB cards I'm unable to go higher than dilations=6 during training.