This pull request primarily focuses on updating the Kagome Japanese morphological analyzer in the Go programming language. The most significant changes include updating the Kagome library, modifying the tokenizer initialization, and adjusting token positions in tests.
Library update:
go.mod: Updated the Kagome library from github.com/ikawaha/kagome.ipadic v1.1.2 to github.com/ikawaha/kagome-dict v1.0.9, github.com/ikawaha/kagome-dict/ipa v1.0.10, and github.com/ikawaha/kagome/v2 v2.9.5.
Tokenizer initialization:
lang/ja/ja_morph_kagome.go: The NewKagomeMorphTokenizer and NewKagomeMorphTokenizerWithUserDic functions now return an error if the tokenizer fails to initialize. The tokenizer initialization now uses the ipa.DictShrink() function instead of tokenizer.SysDic().
Token positions in tests:
lang/ja/analyzer_ja_test.go and lang/ja/ja_morph_kagome_test.go: Token positions in tests have been adjusted to start from 0 instead of 1. [1][2][3]
This pull request primarily focuses on updating the Kagome Japanese morphological analyzer in the Go programming language. The most significant changes include updating the Kagome library, modifying the tokenizer initialization, and adjusting token positions in tests.
Library update:
go.mod: Updated the Kagome library fromgithub.com/ikawaha/kagome.ipadic v1.1.2togithub.com/ikawaha/kagome-dict v1.0.9,github.com/ikawaha/kagome-dict/ipa v1.0.10, andgithub.com/ikawaha/kagome/v2 v2.9.5.Tokenizer initialization:
lang/ja/ja_morph_kagome.go: TheNewKagomeMorphTokenizerandNewKagomeMorphTokenizerWithUserDicfunctions now return an error if the tokenizer fails to initialize. The tokenizer initialization now uses theipa.DictShrink()function instead oftokenizer.SysDic().Token positions in tests:
lang/ja/analyzer_ja_test.goandlang/ja/ja_morph_kagome_test.go: Token positions in tests have been adjusted to start from 0 instead of 1. [1] [2] [3]