jolicode / emoji-search

:smile: Emoji synonyms to build your own emoji-capable search engine (elasticsearch, solr, OpenSearch)

https://jolicode.com/blog/elasticsearch-icu-now-understands-emoji

MIT License

218 stars 64 forks source link

An Emoji capable tokenizer #3

Closed damienalexandre closed 7 years ago

damienalexandre commented 8 years ago

We need a specific tokenizer to handle emoji properly in Elasticsearch, here is the specification.

Emoji Tokenizer

Should be based on the standard tokenizer to keep the grammar logic and Unicode Text Segmentation implementation.

But add support for symbols / emoji as specified in tr51.

Test cases

Input	Tokens
Get some donut's.	`Get` `some` `donut's`
Get some 🍩's.	`Get` `some` `🍩's`
"🍩"	`🍩`

damienalexandre commented 8 years ago

Currently working on a ICU 58 based tokenizer with additional custom rules to consider emoji & their combinaisons as word/token. Elasticsearch's ICU plugin is out of date (ICU 54), Lucene ICU too (ICU 56), so I will have to ship my own ICU with the plugin, my only concern is about compatibility...

:yum:

gibrown commented 7 years ago

@damienalexandre did you manage to build an ICU tokenizer that supports emoji tokenization by any chance?

damienalexandre commented 7 years ago

Yes! The plugin documentation is here: https://github.com/jolicode/emoji-search/tree/master/esplugin

gibrown commented 7 years ago

Looks really cool! Thanks! Adding to our project list to try it out.