Punctuation options

[lmullen]

Some tokenizers have options to strip or retain punctuation and numbers. These options should also be present in the word and n-gram tokenizers.

• [x] tokenize_words()
• [ ] tokenize_ngrams()
• [ ] tokenize_skip_ngrams()

[lmullen]

The punct-options branch now has options for preserving punctuation in tokenize_words().

@kbenoit Does this do what you expect it to do?

[lmullen]

Not going to do this for n-grams. Works for word tokenizer.