Ensure consistent results around punctuation

rspeer / wordfreq

Access a database of word frequencies, in various natural languages.

Other

1.4k stars 101 forks source link

Closed rspeer closed 3 years ago

rspeer commented 4 years ago

Set minimum versions of dependencies, so that we get consistent tokenization in edge cases around punctuation
Add test cases that check for this consistent tokenization
Update langcodes to use closest_match