Closed terasakisatoshi closed 2 years ago
This PR resolves #1
julia> tokenize(Tokenizer(), "アイス") 1-element Vector{Tuple{String, String}}: ("アイス", "名詞,一般,*,*,*,*,アイス,アイス,アイス") julia> tokenize(Tokenizer(), "不思議の国のアリス") 5-element Vector{Tuple{String, String}}: ("不思議", "名詞,形容動詞語幹,*,*,*,*,不思議,フシギ,フシギ") ("の", "助詞,連体化,*,*,*,*,の,ノ,ノ") ("国", "名詞,一般,*,*,*,*,国,クニ,クニ") ("の", "助詞,連体化,*,*,*,*,の,ノ,ノ") ("アリス", "名詞,固有名詞,人名,名,*,*,アリス,アリス,アリス") julia> tokenize(Tokenizer(), "すもももももももものうち") 7-element Vector{Tuple{String, String}}: ("すもも", "名詞,一般,*,*,*,*,すもも,スモモ,スモモ") ("も", "助詞,係助詞,*,*,*,*,も,モ,モ") ("もも", "名詞,一般,*,*,*,*,もも,モモ,モモ") ("も", "助詞,係助詞,*,*,*,*,も,モ,モ") ("もも", "名詞,一般,*,*,*,*,もも,モモ,モモ") ("の", "助詞,連体化,*,*,*,*,の,ノ,ノ") ("うち", "名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ") julia>
Thanks
This PR resolves #1