search

deepgram / kur

Descriptive Deep Learning
Apache License 2.0
814 stars 107 forks source link

The speed of training gradually decreases when using gpu #97

Closed zhaoqxu-eth closed 5 years ago

zhaoqxu-eth commented 6 years ago

Hi, I'd like to know what I could do to prevent the decreasing of speed (samples/s). When I start training it, the first second it's over 6 samples/s, and it shows (00:00<7:57), but after a few minutes, it gradually decreases to under 2 samples/s, which made the time longer(09:27<10:19). What made this problem and how could I fix it? Thanks!

scottstephenson commented 6 years ago

That’s normal with sortagrad turned on. Is sortagrad turned on?

Sortagrad is a curriculum learning method that feeds the model the sorted dataset (sorted from shortest to longest, shortest first) during training.

zhaoqxu-eth commented 6 years ago

hi , I'm just running the example speech.yml with BaiduSpeech model. I find sortagrad is duration, what should I do to turn it off? Is that important? thanks!

zhaoqxu-eth commented 6 years ago

I tried deleting it directly, which makes training speed too slow

scottstephenson commented 5 years ago

That's normal because now you are training with the average utterance length all the time versus the shortest of the files with sortagrad.