Closed kant2002 closed 1 year ago
https://github.com/lang-uk/tokenize-uk Seems to be proper path forward for tokenization, but I cannot run it's tests locally yet, so let's have temporary tokenizer meanwhile.
https://github.com/lang-uk/tokenize-uk Seems to be proper path forward for tokenization, but I cannot run it's tests locally yet, so let's have temporary tokenizer meanwhile.