search

bshall / UniversalVocoding

A PyTorch implementation of "Robust Universal Neural Vocoding"
https://bshall.github.io/UniversalVocoding/
MIT License
238 stars 41 forks source link

How long does it takes to train from the scratch? #16

Closed sbkim052 closed 4 years ago

sbkim052 commented 4 years ago

Thank you for sharing your great work. As i have changed many parameters(n_mel, fft, hop, window etc), I am training this model from scratch with VCTK dataset. Could you tell me the environment you had and how long it took? I have geforce rtx 2080 ti, and it seems to take whole month :(

bshall commented 4 years ago

Hi @sbkim052,

It'll depend on what the settings you use but it definitely shouldn't take a whole month. First, I'd advise not to use the full sample rate for VCTK. It'll make training a lot faster if you resample to 22050Hz or something lower. I have an rtx 2070 super and can train the model in 6ish hours. Maybe you could post your parameters just to see if things are reasonable?

Also, issue #5 has some discussion around training times so it might be worth checking out.

sbkim052 commented 4 years ago

Thank you @bshall . I have an additional question. Why does it take so long to load the checkpoint when executing generate.py?

bshall commented 4 years ago

Sorry about the delay @sbkim052,

How long is does it take for you? It will take a bit of time to move the model to the GPU but it shouldn't be very long.

sbkim052 commented 4 years ago

Hi @bshall It took about 2minutes or so to load the model. First I thought it took 2minutes for inferencing, but as you said, the time for moving the model to GPU takes long and the inferencing time is not that long.