Analyze the correspondence between Kuromoji tags and Wiktionary tags. Implement the lookup process.

[x-ji]

IPADIC information

https://hayashibe.jp/tr/mecab/dictionary/ipadic

品詞 / 品詞細分類1

• 名詞
• 動詞
• 形容詞
• 副詞
• 接続詞
• 感動詞
• 助詞
• 助動詞
• 連体詞
• 接頭詞
• 記号: This is just the punctuation marks.
• フィラー: Filler?
• その他: 間投

Wiktionary Information

https://en.wiktionary.org/wiki/Wiktionary:About_Japanese

They claim their methods conform to the methods in modern kokugo texts.

• Noun 名詞, N
• Verbs 動詞, V...
• Adjectives 形容詞, A
• Adjectival nouns (形容動詞) This is classified as a sub-category of Noun, 名詞 - 形容動詞語幹, under the IPADIC scheme
• Adverbs (副詞), ADV
• Conjunctions (接続詞), CNJ
• Interjections (感動詞), ITJ
• Particles (助詞), PRT
• Auxiliaries (助動詞) Strangely, seems to also have SFX as type. Maybe it's because its usage is similar to SFX?
• Adnominals (連体詞), DET?
• Counter words (助数詞), CNT This is classified as a sub-sub category of noun, as 名詞 - 接尾 - 助数詞, under the IPADIC scheme.
• Prefixes (接頭辞), ??? couldn't find an example of such a word in the dump yet. An example on Wiktionary would be https://en.wiktionary.org/wiki/%E8%B6%85#Japanese
• Suffixes (接尾辞), SFX Seems to be classified under three different places, as subcategories of noun, verb and adj respectively.

[verenablaschke]

• Kuromoji uses some hierarchy for the POS tags associated with each token (their Token class, sample output).
• Here's a manual for IPAdic that includes a section on POS tagging. I'm not sure how closely kuromoji's output tagset matches this one.

[verenablaschke]

From the Wiktionary dump, I additionally get:

• subsets of N:
  • named entities: NE
  • numerals: NUM
  • pronouns: PRN

[verenablaschke]

I'm not sure if we should include this in this issue or create a new one, but we will have to merge some of the kuromoji tokens; at least those for inflected verbs/adjectives. Example:

• input: この動物は日本語で何と言いますか。
• kuromoji output: この　動物　は　日本語　で　何　と　言い　ます　か。
• what we expect: この　動物　は　日本語　で　何　と　言います　か。

[x-ji]

Now there is a first implementation. Merging of inflected verbs is still not implemented yet.

[verenablaschke]

We should probably also add a convertPOSTag method for Wiktionary tokens, where we map all the adjectives to a A and all the verb tags to V (we might also need to merge the noun-related tags). That way, we can establish a better link between Kuromoji tags and Wiktionary tags while still being able to display the additional information that comes with the Wiktionary tags (inflection paradigm, transitivity).

[x-ji]

Created a new issue for merging tokens to look for potential inflections.