Hi. After processing a large (30G) input file, I tried training Glove on it but I ran into a few problems. Every epoch outputs -nan as the cost. I've pasted all the relevant output below:
(train-file.sh is just a modified version of demo.sh using my input file instead of text8)
Closing this as these errors went away when I ran the program on a machine with more memory and more disk space (there are many temporary files written to disk during the preprocessing tasks).
Hi. After processing a large (30G) input file, I tried training Glove on it but I ran into a few problems. Every epoch outputs
-nan
as the cost. I've pasted all the relevant output below:(
train-file.sh
is just a modified version ofdemo.sh
using my input file instead oftext8
)Things I've noticed:
after calling
shuffle
, 0 lines processed:SHUFFLING COOCCURRENCES array size: 382520524 Shuffling by chunks: processed 0 lines.
after calling
glove
, again 0 lines read:TRAINING MODEL Read 0 lines
Other info:
vocab.txt
looks good, there are 1817927 lines (27 MB)vectors.txt
look good; it's not empty; there is a vector entry for every token, and nonan
s that I could find (845 MB)vectors.bin
looks good too, it's a 1.4 GB file.Both
cooccurrence.bin
andcoocurrence.shuf.bin
are zero-length files.Things I've tried
setting
x-max
to 100.0 instead of the default value of 10.0. It didn't help. I got the same behaviour as above.setting
min-count
to 50 to reduce the size of vectors. It didn't help. I got the same behaviour as above.Any thoughts about this?