About Preprocessing in `Juman.apply_to_sentence`

[tealgreen0503]

It appears that some preprocessing takes place when performing morphological analysis with Juman.apply_to_sentence. For example, half-width spaces are replaced with full-width spaces, and line breaks are removed.

import rhoknp
juman = rhoknp.Jumanpp()
text = " これは半角スペースです。"
print([morpheme.surf for morpheme in juman.apply_to_sentence(text).morphemes])
# ['\u3000', 'これ', 'は', '半角', 'スペース', 'です', '。']
text = "\nこれは改行です。"
print([morpheme.surf for morpheme in juman.apply_to_sentence(text).morphemes])
# ['これ', 'は', '改行', 'です', '。']

Are there other such preprocessings?

[hkiyomaru]

The preprocessing steps performed by Jumanpp.apply_to_sentence include:

1. Replacing half-width spaces with full-width spaces.
2. Replacing straight double quotation marks (") with curved double quotation marks (”).
3. Removing line breaks.
4. Removing carriage returns.

[hkiyomaru]

It's important to note that sentences beginning with # are considered comments and are not parsed. The Juman++ developer has proposed a workaround to address this, which can be found in this Github issue. It's worth mentioning that rhoknp does not perform this workaround as a pre-processing step. If you require this functionality, you will need to implement the workaround yourself.

[hkiyomaru]

Let's carry on this discussion on https://github.com/ku-nlp/jumanpp/discussions/154.

[hkiyomaru]

https://github.com/ku-nlp/rhoknp/pull/123 will fix the handling of half-width spaces and straight double quotation marks.