Consider dropping NLTK, consider adding sacreMoses for tokenization

ELITR / SLTev

SLTev is a tool for comprehensive evaluation of (simultaneous) spoken language translation.

8 stars 3 forks source link

Consider dropping NLTK, consider adding sacreMoses for tokenization #5

Closed obo closed 3 years ago

obo commented 3 years ago

NLTK is quite a heavy dependence. All we want from it is tokenization. We would probably be better off with sacreMoses (which also adds some useless bells and whistles like the progress bar). Please discuss here.

mohammad2928 commented 3 years ago

Hi Ondrej, I have dropped NLTK in the commit by the name " NLTK dropped in quality calculation". Also, I have used mosestokenizer for tokenization.

obo commented 3 years ago

NLTK is still in requirements.txt