Hi, I am currently working with recurrent-language-model.lua and I noticed that dl.text2tensor function behaves weird when you put some OOV words in either of PennTreebank files like "ptb.valid.txt" or "ptb.test.txt". I also added a custom OOV word to "ptb.train.txt" to see if dl.buildVocab handles this case accordingly but oov variable counter stayed 0. I hope I am not confusing this for intended behaviour.
If this is a bug, then recurrent-language-model.lua won't work with any data but ptb which is split so that neither of "valid" or "test" samples have oov words.
After looking closer at the code I see than one can make use of minfreq parameter to dl.buildVocab and set it to 2 for example. This would allow "valid" or "test" to have oov words.
Hi, I am currently working with
recurrent-language-model.lua
and I noticed thatdl.text2tensor
function behaves weird when you put some OOV words in either of PennTreebank files like "ptb.valid.txt" or "ptb.test.txt". I also added a custom OOV word to "ptb.train.txt" to see ifdl.buildVocab
handles this case accordingly butoov
variable counter stayed 0. I hope I am not confusing this for intended behaviour. If this is a bug, thenrecurrent-language-model.lua
won't work with any data but ptb which is split so that neither of "valid" or "test" samples have oov words.