Entropy is nan when a vocabulary larger than 2 million words is used.

yandex / faster-rnnlm

Faster Recurrent Neural Network Language Modeling Toolkit with Noise Contrastive Estimation and Hierarchical Softmax

Other

561 stars 138 forks source link

Entropy is nan when a vocabulary larger than 2 million words is used. #24

Open mpatsis opened 8 years ago

mpatsis commented 8 years ago

Hi, When I am using a vocabulary that is larger than 2 million words (e.g., 2.2 million) the validation entropy is always nan. However, on the exact same data if I use a slightly smaller vocabulary (1937725 words) then entropy is calculated normally. The vocabulary is being limited by rare words from the vocabulary file.

Best regards, Rafael