Closed ertugrul-dmr closed 2 years ago
You are calling a private function (_tokenize
) which you shouldn't when you use WordTokenizer
class (or subclass) as an user because it is there for internal usage (you can of course can use when you understand the consequences) What you need is
it = toker(mention=True)
tokens_pred = it(text) # calling __call__ function on WordTokenizer class
Please do check my commit :) I have already done.
Yeah, thank you for update. What I did was just little fixes where I missed in the first place :)
can you ensure that Text2Doc also works properly with the exception rules and close the pull request if it is not required.
Self note: Going to fix issue with Text2Doc where it's not updating hashtag, mention etc. after first call; then will add test cases. Both steps under this PR.
Files changed:
tests/test_tokenizer_flags.py:
To-do:
Going to add test cases for Text2Doc with @onatyap and enrich test cases for different options.