Open reckart opened 8 years ago
I agree that it might be more useful to split ]。
into ]
and 。
, but this is actually how the dictionary assets we are using have been designed, but perhaps it might make sense to change some of this. I have some ideas I'd like to try out...
Just jumping in to say that this outputs highlights a problem with Mecab-ipadic - symbols such as the [] here are treated as 名詞・サ変接続.
see fix for problem here: https://github.com/taku910/mecab/pull/37
Thanks. We could also do this using a user-defined unk definition...
Looks like the segmenter does not work properly if there are square brackets, e.g.:
or