search

sinaahmadi / klpt

The Kurdish Language Processing Toolkit
https://sinaahmadi.github.io/klpt/
Other
91 stars 11 forks source link

in_separator in mwe_tokenize doesn't work properly #18

Open sinaahmadi opened 1 year ago

sinaahmadi commented 1 year ago

The argument in_separator seems to be overwritten at the end of the mwe_tokenize function. The argument is not accessible through word_tokenizer either.