search

stanfordnlp / GloVe

Software in C and data files for the popular GloVe model for distributed word representations, a.k.a. word vectors or embeddings
Apache License 2.0
6.86k stars 1.51k forks source link

cost: -nan #99

Closed zhou1xiang2 closed 6 years ago

zhou1xiang2 commented 6 years ago

I'm trying to train a model on a 59G Chinese corpus. After a few iterations, I got "cost: -nan". And every vector element in vectors.txt is '-nan'. It looks like the training did not converge. What should I do to avoid this situation? Any replies will be appreciated.

TRAINING MODEL Read 7517324041 lines. Initializing parameters...done. vector size: 300 vocab size: 9285835 x_max: 0.000000 alpha: 0.750000 iter: 001, cost: 0.755250 iter: 002, cost: 0.630120 iter: 003, cost: 0.578230 iter: 004, cost: 0.545477 iter: 005, cost: 0.524861 iter: 006, cost: 0.509154 iter: 007, cost: 0.496371 iter: 008, cost: 0.486092 iter: 009, cost: -nan iter: 010, cost: -nan iter: 011, cost: -nan iter: 012, cost: -nan

sidak commented 6 years ago

Try decreasing the learning rate (referred to eta here). From my experience, when I was training on Toronto book corpus with 40 threads, I got something very similar to you. I decrease eta from 0.05 to 0.005, after which I didn't face the issue.

zhou1xiang2 commented 6 years ago

Thanks dude, it works.