search

nyu-dl / dl4mt-tutorial

BSD 3-Clause "New" or "Revised" License
618 stars 249 forks source link

NaN detected #64

Open amirj opened 8 years ago

amirj commented 8 years ago

After some iterations the following error is occurred:

... 128 samples computed 256 samples computed 384 samples computed 462 samples computed Valid 2.78936 Epoch 0 Update 88100 Cost 0.587470054626 UD 1.63904905319 NaN detected

Reloading the model using reload=True parameter, continue the learning process. It maybe related to this issue.

valentinkoe commented 7 years ago

Have you tried @orhanf 's suggestions in the issue you linked? For my use case it seems like applying gradient clipping (setting clip-c to 1.0) solved the problem of NaNs.