yandex / faster-rnnlm

Faster Recurrent Neural Network Language Modeling Toolkit with Noise Contrastive Estimation and Hierarchical Softmax
Other
561 stars 138 forks source link

Get stuck in text generation mode #16

Closed gorinars closed 7 years ago

gorinars commented 8 years ago

I'd like to generate text using the tool.

The network has 500 hidden units, trained on about 300K tokens encoded in UTF-8 (27.3K vocab). Tried both NCE and HS. Training shows no issues but when running with "-rnnlm $rnnmodel --generate-samples 1000000 --generate-temperature 1.0" the program gives the following output and seem to be freezing (memory and CPU is almost not occupied). Same behaviour with and without CUDA compilation and execution.

Read the vocabulary: 27360 words
Restoring existing nnet
Constructing RNN: layer_size=500, layer_type=sigmoid, layer_count=1, maxent_hash_size=999980640, maxent_order=3, vocab_size=27360, use_nce=0
Contructed HS: arity=2, height=19

Trying to find what is the issue from the code, but no success so far

gorinars commented 8 years ago

This seem to be solved. It was not clear from documentation that in order to generate sequence(s), we need to type the first unit in stdin

A possible solution is to manually add at the beginning of each line of the training data and explicitly define the buffer in words.cc

`
strcpy (buffer_,"");

// if (fgets(buffer_, MAX_LINESIZE, file) == NULL) {

// return false;

// } `

This looks more like a hack, so I keep it here without commit. Thanks for your tool. Speed-up in RNN training is just amazing.