search

deepgram / kur

Descriptive Deep Learning
Apache License 2.0
814 stars 107 forks source link

Validation loss diverging in speech example #101

Closed sgangs closed 5 years ago

sgangs commented 5 years ago

I've been training the speech example for around 25 hrs now. And it seems from epoch no 28 validation loss is diverging and I'm now in epoch 105. At the same time test error is converging to nearly zero. And the latest prediction vs truthis as follows:

Prediction: he i gaeil pefit rterito a oul tod Truth: none of you will be fit for duty tomorrow

The prediction vs truth on epoch 26 was: Prediction: e xfete coppopo chiosflthearo ithur on mi vitto eo Truth: looks like a clump of bushes down there but i ain't sure can you make it out

It seems the language model isnt ready yet. The system is not spitting out english. Yet the test error is nearly zero and validation has diverged from 220 (around) to 602

Any suggestion on what could/should be done?

scottstephenson commented 5 years ago

Use a bigger dataset. Definitely don't let the train error get to zero, you are overfitting that way.

See #6 for a bigger dataset.