Enable rule-based tokenization that regroups neighboring tokenized terms that logically belong together under the same entities. Think compound words or full names.
Objectives
Edit tokenization functions to allow a variable number of parameters to allow for specific rules and exceptions during tokenization.
Description
Enable rule-based tokenization that regroups neighboring tokenized terms that logically belong together under the same entities. Think compound words or full names.
Objectives