search

rth / vtext

Simple NLP in Rust with Python bindings
Apache License 2.0
147 stars 11 forks source link

Renamed `UnicodeSegmentTokenizer` to `UnicodeWordTokenizer`. #75

Closed rth closed 4 years ago

rth commented 4 years ago

Closes https://github.com/rth/vtext/issues/72

I think I prefer UnicodeWordTokenizer to UnicodeTokenizer to avoid confusion with UnicodeSentenceTokenizer..