Closed yc-lim-boop closed 1 year ago
Getting the same error, have you found a way to solve it?
Currently I'm just implementing my own version of word_tokenize
as shown above, that strips whitespace from the initial tokens.
My case has a little bit different from yours, this error emerges when I attempt to pass a fixed vocabulary, and I am looking for a "legal" solution but yeah, the error is solved when I strip my vocabulary.
underthesea
version: 6.2.0text_normalize
inserts a space for "Đaị", which causesword_tokenize
to panic:The space seems to be inserted by the
token_map
intext_normalizer
. Is this the expected behaviour?As a workaround, if whitespace is stripped from the tokens returned by
text_normalize
,word_tokenize
works: