search

yxtay / char-rnn-text-generation

Character Embeddings Recurrent Neural Network Text Generation Models
MIT License
61 stars 23 forks source link

Question on how model selects the best number of epochs #2

Closed surangak closed 6 years ago

surangak commented 6 years ago

Hi there!

Im a noob exploring your text generation project. Its been excellent so far, but I was hoping that you could help me with one question: what is the approach used to pick the best number of epochs? for example, if I were to assign a random number of epochs when I run the script, does the model intelligently stop at a point where it can no longer learn? Is there a way for me to find out how many epochs are best, even though I may have assigned and run a large number of epochs at start?

yxtay commented 6 years ago

Hi @surangak apologies for the late reply as I am not actively monitoring issues on this repository.

The short answer is to train the model till the loss doesn't decrease further. One way to do this is to monitor it on TensorBoard and stop the training once the loss flattens. You can also use the generated sample outputs to get a sense of whether the model is sufficiently trained.

A more proper way to assess whether training is complete is to have a holdout validation data on which you calculate the loss every epoch and stop training when validation loss is not decreasing significantly. Using training loss tend to lead to overfitting.

The current version of the code does not apply any form of early stopping. It will carry on with the training until the number of specified epoch is reached even if learning is stagnant. Hence, I do not recommend setting too large a value.