Closed Seventhen closed 7 years ago
Training happens on multiple utterances in parallel, on a GPU. If those utterances have he same length (that's what sort_by_len does), training is very efficient. If that is not the case, training is slow.
If I recall correctly, when sorting, the shortest utterances train first (and maybe shorter = faster?) If you turn off sorting, then you are training on possibly long utterances, which may proceed more slowly at first.
On 11/25/2016 03:33 AM, Seventhen wrote:
I use 300 hour corpus to train eesen with train-ctc-parallel.sh script. Everything goes ok.
But when I chaned the parameter sort_by_len in train-ctc-parallel.sh from defaut true to false. The training speed suddenly drop.I can't fingure it out.
I have checked running process, both use GeForce GTX 1070 GPU and memory enouch. Can anyone help me to explain it? I will be very grateful.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/srvk/eesen/issues/109, or mute the thread https://github.com/notifications/unsubscribe-auth/ACX11tiLdN6tgJ3lNe_MXyMsz7nQ3UTUks5rBp1tgaJpZM4K8NdI.
--
Eric Riebling Interactive Systems Lab er1k@cs.cmu.edu 407 South Craig St.
I use 300 hour corpus to train eesen with train-ctc-parallel.sh script. Everything goes ok.
But when I chaned the parameter sort_by_len in train-ctc-parallel.sh from defaut true to false. The training speed suddenly drop.I can't fingure it out.
I have checked running process, both use GeForce GTX 1070 GPU and memory enouch. Can anyone help me to explain it? I will be very grateful.