search

meilisearch / charabia

Library used by Meilisearch to tokenize queries and documents
MIT License
258 stars 89 forks source link

Add Math symbols in the default separator list #300

Closed ManyTheFish closed 3 months ago

ManyTheFish commented 3 months ago

In search engines, math symbols are mostly ignored, so it would be better to consider them as separators by default.

related to https://www.compart.com/en/unicode/category/Sm