Training Sudachi on new data

[ardakdemir]

If I understand correctly, Sudachi is a lattice-based tokenizer and uses the occurrence probabilities and left-right probabilities (costs) for finding the best token sequence.

We would like to know whether we could customize these cost values. I imagine that in a niche domain like biomedicine with many unknown bacteria/disease names, we need domain-specific values to have the best tokenizer.

[katsutan]

Unfortunately, transition cost learning is not implemented in Sudachi.

Since SudachiDict is based on UniDic, you can use unidic-mecab to train costs and convert to Sudachi format yourself. However, we do not recommend it because it is complicated.

[sorami]

FYI, you can add words using a user dictionary, and there you can set arbitrary costs. This could be an easy ad-hoc method that may be helpful in some situation, without training the CRF using corpora.

To be more pricise, in user dict, you can set "occurance cost" and "left/right connection IDs". You cannot directly set the connection costs as they are defined seprately in the trained connection table matrix.def.

There is an official documentation about user dictionary, however, it is currently only in Japnese; https://github.com/WorksApplications/Sudachi/blob/develop/docs/user_dict.md