Closed bottydim closed 8 years ago
By default the tokenizer replaces all words with frequency less than 10 with UNKOWN tokens which are represented by the symbol 2. You can change this by calling Tokenizer(min_df=1) instead.
If you use tokenizer.inverse_transform([[2, 2], [2, 2]]) it will return ["UNK UNK", "UNK UNK"].
Makes perfect sense, thank you very much for your help and time :)
It seems like the Tokenizer is broken since the following code snippet: train_text = ['hello world', 'foo bar']
tokenizer = Tokenizer() train_tokens = tokenizer.fit_transform(train_text)
results in:
[[2, 2], [2, 2]]