feat: Support tokenizer configurations - Githubissues

tensorchord / pg_bestmatch.rs

Generate BM25 sparse vector inside PostgreSQL

Apache License 2.0

18 stars 4 forks source link

feat: Support tokenizer configurations #5

Open VoVAllen opened 1 month ago

VoVAllen commented 1 month ago

https://medium.com/@emitchellh/extending-bm25-with-subwords-30b334728ebd

VoVAllen commented 5 days ago

Chinese: https://github.com/messense/jieba-rs
Tiktoken (OpenAI GPT's tokenizer): https://github.com/zurawiki/tiktoken-rs
BPE tokenizer setting use tokenizers library (Roberta vocabulary list?)
(Optional) Allow user to choose Chinese vocab https://huggingface.co/google-bert/bert-base-chinese/raw/main/tokenizer.json