Fix NLTK tokenizer within RAG analyzer

infiniflow / infinity

The AI-native database built for LLM applications, providing incredibly fast hybrid search of dense vector, sparse vector, tensor (multi-vector), and full-text

https://infiniflow.org

Apache License 2.0

2.68k stars 275 forks source link

Fix NLTK tokenizer within RAG analyzer #2228

Closed yingfeng closed 1 week ago

yingfeng commented 1 week ago

What problem does this PR solve?

Stack overflow due to unicode string
Adjust working mode of NLTK, it will work only when alphabets will exceed 90% of the string

Issue link:#2226

Type of change

[x] Bug Fix (non-breaking change which fixes an issue)