I want the tokenizer to make it match entire the word only - my expected output:
['chien', 'thanggggg', 'chien thang']
I have made a change in the source code of tokenize function - add the word boundaries in the logic of regex:
def tokenize(text, format=None, tag=False, use_character_normalize=True, use_token_normalize=True, fixed_words=[]):
"""
tokenize text for word segmentation
Args:
use_token_normalize: use token normalize or not
use_character_normalize: use character normalize or not
tag: return token with tag or not
format: format of result, default is None
"""
global recompile_regex_patterns
global patterns
if len(fixed_words) > 0:
compiled_fixed_words = [re.sub(' ', '\ ', fixed_word) for fixed_word in fixed_words]
fixed_words_pattern = "(?P<fixed_words>\\b" + "\\b|\\b".join(compiled_fixed_words) + "\\b)" # fix here, add the \b
merged_regex_patterns = [fixed_words_pattern] + regex_patterns
regex_patterns_combine = r"(" + "|".join(merged_regex_patterns) + ")"
patterns = re.compile(regex_patterns_combine, re.VERBOSE | re.UNICODE)
recompile_regex_patterns = True
if use_character_normalize:
text = normalize_characters_in_text(text)
matches = [m for m in re.finditer(patterns, text)]
tokens = [extract_match(m) for m in matches]
if tag:
return tokens
tokens = [token[0] for token in tokens]
if use_token_normalize:
tokens = [token_normalize(_, use_character_normalize=use_character_normalize) for _ in tokens]
if format == "text":
return " ".join(tokens)
return tokens
Then it works correctly in my cases, but I still wonder if this change breaks the logic of regex in other cases or not.
Can you guys enable this option for the word tokenizer? thank you, really appreciate your work
I have a script
The result:
I want the tokenizer to make it match entire the word only - my expected output:
I have made a change in the source code of tokenize function - add the word boundaries in the logic of regex:
Then it works correctly in my cases, but I still wonder if this change breaks the logic of regex in other cases or not.
Can you guys enable this option for the word tokenizer? thank you, really appreciate your work