Optional: Tokenizer with n-Grams

XeniaRieger / Modern-Search-Engines

0 stars 0 forks source link

Closed XeniaRieger closed 2 months ago

XeniaRieger commented 2 months ago

Some functions like detect language and relevance need simple tokens, not n-grams. Maybe two tokenizers?