Open stefan-it opened 4 years ago
Thanks for running this on German @stefan-it! I haven't done experiments on different languages yet and it's great to see it at least hold!
Unfortunately I have run into similar issues in the past re: NaNs at around 15 or 20 epochs on the enwik8
data which works out to around 2 epochs on your larger German dataset.
I still haven't tracked down exactly what it might be but I do know the random seed can impact it. My guess is that there might be an issue with dropout over the attention window or similar.
I'll be making a new cleaner codebase and ensuring that issues like this don't occur will be a top priority. As a temporary fix if you're curious to continue investigating you could save the model once every N iterations if the loss hasn't NaN'ed out and restart as you've done with a different random seed. That's admittedly not a great solution however.
I'm glad you're enjoying the generated text! I ran it through Google Translate and it at least produces something I can read lol. I'll note that for this model the more context you can seed it with the better!
Hi @Smerity ,
thanks for open sourcing the code for that great project :heart:
I trained a character-based model for German on ~1GB of text (mainly from OPUS). It worked well for two epochs, but then the following error message is thrown:
Then I resumed training with a lower learning rate (pretty much the same parameters as stated in the main readme) and the same error is thrown after one epochs.
Do you know how this can be prevented 🤔
However, the generated text is very interesting 😅