Fix the disconnected EOS problem - Githubissues

WorksApplications / SudachiPy

Python version of Sudachi, a Japanese tokenizer.

Apache License 2.0

392 stars 50 forks source link

Fix the disconnected EOS problem #109

Closed sorami closed 4 years ago

sorami commented 4 years ago

The problem as reported in #96.

The solution is the same as Java version, c.f. Fix #45 by kazuma-t · Pull Request #113 · WorksApplications/Sudachi

[x] Update the tokenizer to avoid the error with 阿Q[a-zA-Z]+ cases
[x] Update the Simple OOV Plugin: look at the possible word length
[x] Add test, update test dictionary (copied from Java Sudachi test)

sorami commented 4 years ago

The above Java fix PR also fixes the Simple OOV Plugin.