asfimport
commented
12 years ago Christian Moen (@cmoen) (migrated from JIRA)
These are very interesting questions, Robert. Please find my comments below.
should these parameters continue to be static-final, or configurable?
It's perhaps possible to make these configurable, but I think we'd be exposing configuration that is most likely to confuse most users rather than help them.
The values currently uses have been found using some analysis and experimentation, and they can probably be improved both in terms of tuning and with added heuristics – in particular for katakana compounds (more below).
However, changing and improving this requires quite detailed analysis and testing, though. I think the major case for exposing them is as a means for easily tuning them rather than these parameters being generally useful to users.
should POS also play a role in the algorithm (can/should we refine exactly what we decompound)?
Very good question and an interesting idea.
In the case of long kanji words such as 関西国際空港 (Kansai International Airport), which is a known noun, we can possible use POS info as a hint for applying the Viterbi penalty. In the case of unknown kanji, Kuromoji unigrams them. (関西国際空港 becomes 関西 国際 空港 (Kansai International Airport) using search mode.)
Katakana compounds such as シニアソフトウェアエンジニア (senior software engineer) becomes one token without search mode, but when search mode is used, we get three tokens シニア ソフトウェア エンジニア as you would expect. It's also the case that シニアソフトウェアエンジニア is an unknown word, but its constituents become known and get the correct POS after search mode.
In general, unknown words get a noun-POS (名詞) so the idea of using POS here should be fine.
There are some problems with the katakana decompounding in search mode. For example, コニカミノルタホールディングス (Konika Minolta Holdings) becomes コニカ ミノルタ ホール ディングス (Konika Minolta horu dings), where we get the additional token ホール (also means hall, in Japanese).
To sum up, I think we can potentially use the noun-POS as a hint when doing the decompounding in search mode, but I'm not sure how much we will benefit from it, but I like the idea. I think we'll benefit most from an improved heuristic for non-kanji to improve katakana decompounding.
Let me have a tinker and see how I can improve this.
is the Tokenizer the best place to do this, or should we do it in a tokenfilter? or both?
Interesting idea and good point regarding IDF.
In order do the decompoundning, we'll need access to the lattice and add entries to it before we run the Viterbi. If we do normal segmentation first then run a decompounding filter, I think we'll need to run the Viterbi twice in order to get the desired results. (Optimizations are possible, though.)
I'm thinking a possibility could be to expose possible decompounds as part of Kuromoji's Token interface. We can potentially have something like
/**
* Returns a list of possible decompounds for this token found by a heuristic
*
* `@return` a list of candidate decompounds or null of none is found
*/
List<Token> getDecompounds() {
// ...
}In the case of シニアソフトウェアエンジニア, the current token would have surface form シニアソフトウェアエンジニア, but with tokens シニア, ソフトウェア and エンジニア accessible using getDecompounds().
As a general notice, I should point our that how well the heuristics performs depends on the dictionary/statistical model used (i.e. IPADIC) and if we might want to make different heuristics for each of those we support as needed.
Kuromoji supports an option to segment text in a way more suitable for search, by preventing long compound nouns as indexing terms.
In general 'how you segment' can be important depending on the application (see http://nlp.stanford.edu/pubs/acl-wmt08-cws.pdf for some studies on this in chinese)
The current algorithm punishes the cost based on some parameters (SEARCH_MODE_PENALTY, SEARCH_MODE_LENGTH, etc) for long runs of kanji.
Some questions (these can be separate future issues if any useful ideas come out):
Either way, I think as a start we should turn on what we have by default: its likely a very easy win.
Migrated from LUCENE-3726 by Robert Muir (@rmuir), resolved Feb 05 2012 Attachments: kuromojieval.tar.gz, LUCENE-3726.patch (versions: 3)