Consider python-only moses-like-tokenizes in sacremoses instead of mosestokenizer

ELITR / SLTev

SLTev is a tool for comprehensive evaluation of (simultaneous) spoken language translation.

8 stars 3 forks source link

Consider python-only moses-like-tokenizes in sacremoses instead of mosestokenizer #40

Closed obo closed 3 years ago

obo commented 3 years ago

The pip module mosestokenizer is sometimes unreliably installed. sacremoses (https://github.com/alvations/sacremoses) could be easier. Consider using sacremoses.

mohammad2928 commented 3 years ago

improved