Currently the DecompoundTokenFilter does not respect keywords that might come from any earlier
Keyword Marker Token Filter. With this PR you can enable this behavior in the configuration by setting "respect_keywords": true. This will prevent the DecompoundTokenFilter from decompounding any keywords.
Please note that I set the default value for respect_keywords to false so this change is completely backwards compatible. also all changes are covered by tests.
Currently the
DecompoundTokenFilter
does not respect keywords that might come from any earlier Keyword Marker Token Filter. With this PR you can enable this behavior in the configuration by setting"respect_keywords": true
. This will prevent theDecompoundTokenFilter
from decompounding any keywords.Please note that I set the default value for
respect_keywords
to false so this change is completely backwards compatible. also all changes are covered by tests.