search

aboSamoor / polyglot

Multilingual text (NLP) processing toolkit
http://polyglot-nlp.com
Other
2.28k stars 337 forks source link

Is it possible to detokenize in polygot? #238

Open ajesujoba opened 3 years ago

ajesujoba commented 3 years ago

I have used polygot to tokenize sentences in Yoruba (word tokenization not sentence segmentation). Is there a way to detokenize?