search

diasks2 / pragmatic_tokenizer

A multilingual tokenizer to split a string into tokens
MIT License
90 stars 11 forks source link

options should (also) allow symbols #2

Closed maia closed 8 years ago

maia commented 8 years ago

I do think the values of the options hash should allow symbols (or: switch all strings to symbols).

This then would be valid:

PragmaticTokenizer::Tokenizer.new(text, punctuation: :none).tokenize
diasks2 commented 8 years ago

Good idea, this should be fixed now: https://github.com/diasks2/pragmatic_tokenizer/commit/ee21fcacce5071c6c4cacbc7f9fe5e76b39a04d9