search

alpheios-project / tokenizer

Alpheios Tokenizer Service
1 stars 0 forks source link

Additional language dependencies #34

Closed irina060981 closed 3 years ago

irina060981 commented 3 years ago

For the issue https://github.com/alpheios-project/tokenizer/issues/33

Added support for

Russian: pymorphy2 Ukrainian: pymorphy2 Thai: pythainlp Vietnamese: Pyvi

irina060981 commented 3 years ago

Can you please also fix the 3 char code for chinese per my comments in #31 ? (zho instead of zhu ?) Thanks!

Fixed

balmas commented 3 years ago

@irina060981 and @monzug this is deployed now. Hopefully the tokenizer will work a little better now for some of these languages.

irina060981 commented 3 years ago

Thank you, @balmas, for all your help with it :)