sentence segmentation

[ruslandoga]

first as on https://ichi.moe

• https://readevalprint.tumblr.com/post/97467849358/who-needs-graph-theory-anyway
• https://github.com/tshatrov/ichiran

or https://jisho.org

[ruslandoga]

• http://www.phontron.com/kytea/
• https://pypi.org/project/JapaneseTokenizer/
• https://github.com/ikeikeikeike/exkanji
• https://github.com/tex2e/mecab-elixir
• https://kairozu.github.io/updates/cleaning-jp-text
• https://kairozu.github.io/updates/japanese-wiki-corpus
• https://kairozu.github.io/updates/japanese-tokenization
• https://kairozu.github.io/updates/nltk-introduction
• https://www.nltk.org/howto/japanese.html
• https://github.com/facebookresearch/LASER
• https://github.com/google/sentencepiece 🔥
• https://github.com/wwwcojp/ja_sentence_segmenter
• https://aclanthology.org/C04-1067.pdf
• https://digitalorientalist.com/2021/05/11/basic-python-for-japanese-studies-using-fugashi-for-text-segmentation/

[ruslandoga]

• https://dev.to/patarapolw/how-do-mecab-kuromoji-and-kagome-japanese-text-analyzer-compare-and-which-dictionary-to-choose-mj1
• https://github.com/ikawaha/kagome