Closed kharazi closed 7 years ago
Have you done any preprocessing for the data? Usually the punctuation is deleted or separated from the words. For example, you might want to replace
Go, get you home, you fragments!
with
Go , get you home , you fragments !
In your first example sentence you have the word glad.
instead of glad
, which does not appear in the training data. You might also want to replace all words with lower case, as you have such a small data set.
In the third sentence you have two words, glad
and meet
that actually should be in the training data. Have you provided any vocabulary for the train command? What is the command you used to train the model?
I don't have any pre-processing to this dataset but I think it shouldn't make problem. For example when I test it with a sentence that actually is in the dataset like glad to see
(4 occurrences in the dataset) I got this result:
log(p(I | <s>)) = -3.4169204235076904
log(p(am | <s> I)) = -2.3558216094970703
log(p(<unk> | <s> I am)) = -0.883741021156311
log(p(to | ... I am <unk>)) = -1.9496990442276
log(p(see | ... am <unk> to)) = -4.616444110870361
log(p(your | ... <unk> to see)) = -2.7698867321014404
log(p(<unk> | ... to see your)) = -0.06693568825721741
log(p(</s> | ... see your <unk>)) = -1.7577476501464844
Sentence perplexity: 9.273394881980112
My train command is theanolm train model.h5
with no extra options. Is there any way to check vocabulary?
You should also give a training file, and probably also a validation file (for determining when to stop training). Another option is to train a fixed number of epochs, e.g:
theanolm train model.h5 --training-set input.txt --max-epochs 5
I trained 5 epochs using above command, then wrote "glad to see" into test.txt
, and computed word probabilities using
theanolm score model.h5 test.txt --output word-scores
I got:
Using gpu device 0: Quadro K2000 (CNMeM is disabled, cuDNN not available)
Reading vocabulary from network state.
Number of words in vocabulary: 25673
Number of word classes: 25673
Building neural network.
Restoring neural network state.
Building text scorer.
Scoring text.
# Sentence 1
log(p(glad | <s>)) = -11.732762336730957
log(p(to | <s> glad)) = -3.53452730178833
log(p(see | <s> glad to)) = -4.968919277191162
log(p(</s> | ... glad to see)) = -7.588036060333252
Sentence perplexity: 1049.4917139680845
Number of sentences: 1
Number of words: 5
Number of tokens: 5
Number of predicted probabilities: 4
Number of excluded (OOV) words: 0
Cross entropy (base e): 6.956061244010925
Perplexity: 1049.4917139680845
One thing is that you can get some unexpected results if you forget that model.h5
already exists from some previous training, because TheanoLM tries to continue training from that existing model.
There's actually a way to look at the model file using h5dump. You can print the contents using h5dump --contents model.h5
and display the vocabulary using h5dump --dataset=/vocabulary/words model.h5
.
It seems that there was a problem while training. Thanks for your help!
I'm trying to train a language model using Tiny-Shakespeare dataset. It seems that language model is correctly trained and I got the following output for this file:
As you can see, there is word
glad
in the dataset, but while scoring the given text the text scorer labels it as<unk>
(I tested this for different words.). Why it happened?