Finally finished this. It looks like quite a lot of changes, but apart from the multi-person skintones, I mostly just moved code around and deleted unused code:
The logic from demojize() is moved to two separate private function tokenize() and filter_tokens() in a new file emoji/tokenizer.py
demojize(), replace_emoji(), emoji_list() and analyze() all use the same logic, that's why I moved it into the new private function tokenize() and filter_tokens(). Otherwise there would have been duplicated code.
Also the logic for the search tree is moved to the emoji/tokenizer.py file.
A new public function analyze() is available and that supports the multi-person skintones. It's similar to emoji_list(). It returns Token and EmojiMatch objects that provide more information than emoji_list() and also allow to split up ZWJ-emoji into the sub-emoji.
The handling of the multi-person skintones in demojize() and replace_emoji can be controlled by the new emoji.config class, which is a static class that works as a module-wide configuration:
I have removed support for Python 2, 3.4, 3.5, because I used some features of Python 3.6. I removed lots of things that were specific to Python 2, especially all the u'strings' .
Finally finished this. It looks like quite a lot of changes, but apart from the multi-person skintones, I mostly just moved code around and deleted unused code:
The logic from
demojize()
is moved to two separate private functiontokenize()
andfilter_tokens()
in a new fileemoji/tokenizer.py
demojize()
,replace_emoji()
,emoji_list()
andanalyze()
all use the same logic, that's why I moved it into the new private functiontokenize()
andfilter_tokens()
. Otherwise there would have been duplicated code. Also the logic for the search tree is moved to theemoji/tokenizer.py
file.A new public function
analyze()
is available and that supports the multi-person skintones. It's similar toemoji_list()
. It returnsToken
andEmojiMatch
objects that provide more information thanemoji_list()
and also allow to split up ZWJ-emoji into the sub-emoji.The handling of the multi-person skintones in
demojize()
andreplace_emoji
can be controlled by the newemoji.config
class, which is a static class that works as a module-wide configuration:I have removed support for Python 2, 3.4, 3.5, because I used some features of Python 3.6. I removed lots of things that were specific to Python 2, especially all the
u'strings'
.Fixes #204 Fixes #256