BigBoyBarney
commented
7 months ago Thank you for this! I agree that UniDic is the best choice, however, bundling the dictionary by default would most likely be an unwelcome change for most users. There is no need to host UniDic dictionaries in / by Memento, as they're actively maintained by NINJAL. Perhaps there could be a button to fetch the dictionary from NINJAL's site, if the user wishes to do so.
How MeCab works
Just to make sure that everybody reading this issue is on the same page regarding how MeCab works, I will try to concisely explain it:
Mecab segments a text based on the inflections, stems and word forms provided by a dictionary. This means that different dictionaries can segment differently. Additionally, since natural languages aren't perfectly rule based, a text string can have multiple, grammatically correct segmentations. MeCab segments every possible permutation and only returns the first one by default. The number of segmentations returned can be controlled with the -N[X] flag. E.g.: -N3 to return the first 3 results. MeCab only truncates, so returning more results incurs 0 performance cost.
Different dictionaries structure their output differently, so whatever Memento implements will have to be dictionary specific. Ipadic is by far the worst one available. Jumandic works most of the time out of the box, Unidic can be made to work every time, but it is currently not a drop in replacement.
Let's look at 2 examples with 感じたんだ:
Jumandic
echo 感じたんだ | mecab -N2 --dicdir /usr/lib64/mecab/dic/jumandic
感じた 動詞,*,ザ変動詞,タ形,感ずる,かんじた,代表表記:感ずる
んだ 助動詞,*,ナ形容詞,基本形,んだ,んだ,*
EOS
感じた 動詞,*,母音動詞,タ形,感じる,かんじた,補文ト 代表表記:感じる
んだ 助動詞,*,ナ形容詞,基本形,んだ,んだ,*
EOSThe first result from Jumandic returns 感ずる, the second returns 感じる. It is obviously possible to find both of these forms in JMdict just by looking up the stem, but that is currently besides the point. Accounting for multiple results from MeCab at index 4 gives us the verbs, along with the inflections.
UniDic
echo 感じたんだ | mecab -N2 --dicdir /usr/lib64/mecab/dic/unidic_spoken
感じ 動詞,一般,*,*,サ行変格,連用形-一般,カンズル,感ずる,感じ,カンジ,感ずる,カンズル,混,*,*,*,*,*,*,用,カンジ,カンズル,カンジ,カンズル,0,C2,M4@1,2106681492382337,7664
た 助動詞,*,*,*,助動詞-タ,連体形-一般,タ,た,た,タ,た,タ,和,*,*,*,*,*,*,助動,タ,タ,タ,タ,*,"動詞%F2@1,形容詞%F4@-2",*,5948916285711041,21642
ん 助詞,準体助詞,*,*,*,*,ノ,の,ん,ン,ん,ン,和,*,*,*,*,*,*,準助,ン,ン,ン,ン,*,"動詞%F2@0,形容詞%F2@-1",*,7968727735869952,28990
だ 助動詞,*,*,*,助動詞-ダ,終止形-一般,ダ,だ,だ,ダ,だ,ダ,和,*,*,*,*,*,*,助動,ダ,ダ,ダ,ダ,*,名詞%F1,*,6299110739157675,22916
EOS
感じ 動詞,一般,*,*,上一段-ザ行,連用形-一般,カンズル,感ずる,感じ,カンジ,感じる,カンジル,混,*,*,*,*,*,*,用,カンジ,カンジル,カンジ,カンジル,0,C2,M4@1,2106672902447745,7664
た 助動詞,*,*,*,助動詞-タ,連体形-一般,タ,た,た,タ,た,タ,和,*,*,*,*,*,*,助動,タ,タ,タ,タ,*,"動詞%F2@1,形容詞%F4@-2",*,5948916285711041,21642
ん 助詞,準体助詞,*,*,*,*,ノ,の,ん,ン,ん,ン,和,*,*,*,*,*,*,準助,ン,ン,ン,ン,*,"動詞%F2@0,形容詞%F2@-1",*,7968727735869952,28990
だ 助動詞,*,*,*,助動詞-ダ,終止形-一般,ダ,だ,だ,ダ,だ,ダ,和,*,*,*,*,*,*,助動,ダ,ダ,ダ,ダ,*,名詞%F1,*,6299110739157675,22916
EOSUniDic seems overly verbose and scary at first, but it is very well documented. Columns of particular interest are index 7 and 10. 語彙素 and 書字形基本形 respectively. Once again, while it would be possible to find 感じる in JMdict from the first result alone, but the second result gives it directly, with the appropriate inflections.
UniDic has formatting issues that currently crashes Memento
This was possibly solved sometime in the past, because setting UniDic as my system dictionary and changing:
#define WORD_INDEX 6to
#define WORD_INDEX 7in dictionary.cpp line 269 makes Memento use UniDic, and it works mostly correctly. (Although as I mentioned earlier, since it only looks at column 7, it only finds 感ずる and not 感じる. Important to note that this is not a shortcoming of Mecab, it is simply not fully implemented yet.
With the current MeCab implementation, Jumandic with #define WORD_INDEX 4 is a drop in upgrade to ipadic in every way, since it is a more complete dictionary. Even though Unidic is far more extensive, the way it orders the results happens to be less compatible with the current MeCab implementation in Memento.
However, for a proper grammar implementation, I suggest basing it on Unidic's format, as it has numerous dictionaries for both modern and classical Japanese.
My C++ knowledge is somewhat limited, and I'm almost entirely unfamiliar with how Memento currently works, but I would gladly help with parsing the output.
EDIT: I can confirm that with your example subtitle of
〝民間南極観測隊
３年ぶりに派遣決まる〞and Unidic, Memento does indeed crash. Removing 〝 from the subtitles fixed this issue. I'm not entirely sure why it happens, as Unidic itself handles 〝 just fine. However, sanitising the subtitles is a reasonable solution.
ripose-jp
Calvin-Xu
Given the recent discussion in https://github.com/ripose-jp/Memento/issues/109 touching on how UniDic may offer better tokenization and possibly morphological analysis (offerings https://github.com/ripose-jp/Memento/pull/210 but Japanese school grammar), I wonder if it might to good to consider supporting UniDic in Memento again. For a overview of the different tokenizer dictionaries, see https://www.dampfkraft.com/nlp/japanese-tokenizer-dictionaries.html. tldr:
The previous discussion about supporting UniDic were in https://github.com/ripose-jp/Memento/issues/35 and in https://github.com/ripose-jp/Memento/issues/109. The main issues are:
@ripose-jp has said that
If just using UniDic can improve the experience, it may be beneficial for UniDic specifically to be supported since it seems be the only thing worth using over IPADic. (edit: to clarify I don’t mean it should be bundled / default due to the large size).
As a side note on things newer than MeCab (not maintained): Sudachi's Apache 2.0 license is incompatible with GPLv2 of this project.