Closed C-Dongbo closed 5 years ago
Hi. Can you manage to reproduce this with a small corpus? Is it crashing at the beginning or more in the middle?
Hi. I experimented with various small corpus. (10,000 lines, 100,000 lines, 1,000,000 lines) 10,000 and 100,000 lines small corpus didn't make error. But 1,000,000 lines small corpus make error in the middle ( 21012 Segmentaion fault).
sorry we can't reproduce it if you don't provide a working example of the case. maybe you have some extremely long lines in that larger corpus? maybe try with fasttext as well to see if it will also fail on the same corpus or not?
./fasttext cbow-c+w-ngrams -input ../data/new.txt -output ../model/new -minCount 100 -dim 300 -ws 10 -epoch 20 -lr 0.2 -wordNgrams 2 -loss ns -neg 10 -thread 20 -t 0.000005 -dropoutK 4 -minCountLabel 20 -bucket 2000000 -bucketChar 1000000 -lrUpdateRate 10000 -minn 3 -maxn 6
Read 6205M words Number of words: 291863 Number of labels: 0 Progress: 0.1% words/sec/thread: 85004 lr: 0.199755 loss: 21.799864 eta: 20h15m ./training_cbowngrams.sh: line 1: 10367 Segmentation fault
I don't know how to solve segmentation fault. Is there a good solution??