search

emorynlp / nlp4j-tokenization

Tokenize raw texts into tokens and sentences.
Other
6 stars 4 forks source link

Handle final "y" in english word tokenization #3

Closed spraynasal closed 8 years ago

spraynasal commented 8 years ago

Fix word tokenization of sentences like : "I did it my way." "Definitely not worth stopping by."

Where the last token will include the end period.

jdchoi77 commented 8 years ago

I merged your changes and modified the originally containsOnlyConsonants with your method. Thank you for the bugfix.

best,

Jinho