Word segmenter for Far east Asia languages

[makotokato]

Actually, LSTM segmenter may be focused for East Asian Language such as Thai. But I am not sure for Japanese and Chinese. Since UAX#29 doesn't define word segmenter for Chinese and Japanese, ICU uses dictionary-based segmenter.

• Can LSTM use for CJ?
• Is there another segmenter implementations to use machine learning for CJ?
• Should we use dictionary-model for CJ like ICU? Dictionary-based implementation cannot covert all words.

[aethanyc]

@makotokato Another question is: Is there any customization rules or optimization for Chinese and Japanese, which are not in the spec, but are in current gecko's lwbrk that we want to support?

[makotokato]

Is there any customization rules or optimization for Chinese and Japanese, which are not in the spec,

Sentence segmentation. Sentence in Japanese depends on contextual, so it cannot define the rule.

in current gecko's lwbrk that we want to support?

lwbrk requires word and line only. For word segmentation, we need any way to implement it (dictionary? or ml?). Dictionary won't cover all word segmenter case in Japanese unfortunately.

[aethanyc]

We've tracked this issue in https://github.com/unicode-org/icu4x/issues/1033.