Currently there is no way to specify what full stop, half stop means for each scheme. Client libraries treat, .,; as word breaker characters for all the languages. This won't work well for Hindi because in Hindi, DEVANAGARI DANDA is used as full stop. DEVANAGARI DOUBLE DANDA is used for semicolon.
Proposal:
punctuation_marks ["।"]
When learning a word, all punctuation marks should be removed.
121 and PrathamBooks/spp#472 depends on this issue.
Currently there is no way to specify what full stop, half stop means for each scheme. Client libraries treat,
.,;as word breaker characters for all the languages. This won't work well for Hindi because in Hindi, DEVANAGARI DANDA is used as full stop. DEVANAGARI DOUBLE DANDA is used for semicolon.Proposal:
When learning a word, all punctuation marks should be removed.
121 and PrathamBooks/spp#472 depends on this issue.