search

srvk / eesen

The official repository of the Eesen project
http://arxiv.org/abs/1507.08240
Apache License 2.0
822 stars 342 forks source link

training use sort_by_len false cause training speed suddenly drop #109

Closed Seventhen closed 7 years ago

Seventhen commented 7 years ago

I use 300 hour corpus to train eesen with train-ctc-parallel.sh script. Everything goes ok.

But when I chaned the parameter sort_by_len in train-ctc-parallel.sh from defaut true to false. The training speed suddenly drop.I can't fingure it out.

I have checked running process, both use GeForce GTX 1070 GPU and memory enouch. Can anyone help me to explain it? I will be very grateful.

fmetze commented 7 years ago

Training happens on multiple utterances in parallel, on a GPU. If those utterances have he same length (that's what sort_by_len does), training is very efficient. If that is not the case, training is slow.

riebling commented 7 years ago

If I recall correctly, when sorting, the shortest utterances train first (and maybe shorter = faster?) If you turn off sorting, then you are training on possibly long utterances, which may proceed more slowly at first.

On 11/25/2016 03:33 AM, Seventhen wrote:

I use 300 hour corpus to train eesen with train-ctc-parallel.sh script. Everything goes ok.

But when I chaned the parameter sort_by_len in train-ctc-parallel.sh from defaut true to false. The training speed suddenly drop.I can't fingure it out.

I have checked running process, both use GeForce GTX 1070 GPU and memory enouch. Can anyone help me to explain it? I will be very grateful.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/srvk/eesen/issues/109, or mute the thread https://github.com/notifications/unsubscribe-auth/ACX11tiLdN6tgJ3lNe_MXyMsz7nQ3UTUks5rBp1tgaJpZM4K8NdI.

--

Eric Riebling Interactive Systems Lab er1k@cs.cmu.edu 407 South Craig St.