search

deepgram / kur

Descriptive Deep Learning
Apache License 2.0
814 stars 107 forks source link

Validation loss become higher after 20 hours training time #88

Open misbullah opened 6 years ago

misbullah commented 6 years ago

Hi, Yesterday, I tried to run speech.yml from examples. In the beginning, the validation loss is about 200.xxx, but after training for 20 hours, the validation become higher. Now, it is about 630.xxx.

Is the any problem with this training process?

Because, I checked the graph on deepgram blog: http://blog.deepgram.com/how-to-train-baidus-deepspeech-model-with-kur/, the validation loss become smaller for more iteration.

Thanks.

scottstephenson commented 6 years ago

Does looking at #6 help? Seeing your loss plot would help too.

misbullah commented 6 years ago

Hi @scottstephenson,

Yes, I checked it already. Simple question, how to create loss plot from deepgram training process? Is there any documentation for it?

I checked that the kur use tensorflow for backend. Does it also support wrap-ctc like implemented in the following git: https://github.com/mozilla/DeepSpeech

Thanks.

scottstephenson commented 6 years ago

Have a look at the tutorial: https://kur.deepgram.com/tutorial.html

The tensorflow backend uses the Tensorflow CTC implementation, PyTorch and Theano use warp CTC.