While working on some preprocessing steps (#270) we noticed that arguments for tokenizers (including Text2Doc) has no effect on the outcome. Basically enable/disable options are not working.
For this issue we might need to create test cases and find the root cause of the problem.
To replicate this you can basically follow these steps:
from sadedegel.bblock.word_tokenizer import ICUTokenizer
tokenizer = ICUTokenizer(hashtag=True, mention=True, emoji=True)
text = 'bu #bir @metindir, 🍰'
print(tokenizer._tokenize(text))
Will result same with False or True options. Same goes for Text2Doc example below:
While working on some preprocessing steps (#270) we noticed that arguments for tokenizers (including Text2Doc) has no effect on the outcome. Basically enable/disable options are not working.
For this issue we might need to create test cases and find the root cause of the problem.
To replicate this you can basically follow these steps:
Will result same with False or True options. Same goes for Text2Doc example below: