Closed fetchadd closed 4 years ago
The tokenizing is special for Chinese, as 'word' is not sperated by whitespace. So there is a new "tokenize_chinese" function for tokenizing Chinese.
The tokenizing is special for Chinese, as 'word' is not sperated by whitespace. So there is a new "tokenize_chinese" function for tokenizing Chinese.