Elasticsearch version: 7.13.3 (tested on 7.15.1 too)
Plugins installed: [repository-s3, analysis-nori]
JVM version (java -version): Eclipse Adoptium/OpenJDK 64-Bit Server VM/17/17+35
OS version (uname -a if on a Unix-like system): Linux/4.9.184-linuxkit/amd64
Description of the problem including expected versus actual behavior:
The nori tokenizer change the token yield depending on punctuation. (even when discard punctuation is true). the decompound_mode has also no impact on the token yields.
I guess it is a bug because if discard punctuation is true, the output should be the same.
Elasticsearch version:
7.13.3
(tested on7.15.1
too)Plugins installed: [
repository-s3
,analysis-nori
]JVM version (
java -version
):Eclipse Adoptium/OpenJDK 64-Bit Server VM/17/17+35
OS version (
uname -a
if on a Unix-like system):Linux/4.9.184-linuxkit/amd64
Description of the problem including expected versus actual behavior:
The
nori tokenizer
change the token yield depending on punctuation. (even when discard punctuation is true). thedecompound_mode
has also no impact on the token yields. I guess it is a bug because if discard punctuation is true, the output should be the same.will have
[표현, 입, 니, 당]
will have
[표현, 입, 니당]
Steps to reproduce:
analysis-nori
.