Can this tokenizer be used for English Language also?

mideind / Tokenizer

A tokenizer for Icelandic text

Other

27 stars 6 forks source link

Closed Dhanachandra closed 4 years ago

Dhanachandra commented 4 years ago

If so, Why MICHAEL K . is tokenized as "MICHAEL" , "K." instead of "MICHAEL", "K", "."

sveinbjornt commented 4 years ago

No, this module only tokenizes Icelandic text.