ahmetaa / zemberek-nlp

NLP tools for Turkish.

Other

1.14k stars 207 forks source link

Tokenizer should work with some tokens with apostrophes. #205

Closed ahmetaa closed 5 years ago

ahmetaa commented 5 years ago

Cannot tokenize:

ali@gmail.com'u @kemal'in

foo'nun

(via Müge)