search

senarvi / theanolm

TheanoLM is a recurrent neural network language modeling tool implemented using Theano
Apache License 2.0
81 stars 29 forks source link

Including end-of-sentence marker in vocabular causes numerical error in validation #18

Closed Waino closed 7 years ago

Waino commented 7 years ago

If the (class) vocabulary includes an entry for the end-of-sentence marker </s>, training fails during calculation of validation set perplexity:

File ".../theanolm/trainers/basictrainer.py", line 128, in train
theanolm.exceptions.NumberError: Validation set perplexity computation resulted in a numerical error.

Workaround: remove the end-of-sentence marker from the vocabulary file.