Add sanity check of mosestokenizer

ELITR / SLTev

SLTev is a tool for comprehensive evaluation of (simultaneous) spoken language translation.

8 stars 3 forks source link

Add sanity check of mosestokenizer #38

Closed obo closed 3 years ago

obo commented 3 years ago

The BrokenPipe problem referenced in #22 is due to malfunctioning mosestokenizer in my installation. Please add a diagnosis: Before the first run of mosestokenizer (or after its installation during STLev installation?), please test if:

tokenize = MosesTokenizer('en')
tokenize('Hello World!')

returns ['Hello', 'World', '!']

If not, exit, explaining that mosestokenizer is badly installed. Do it in try-catch clause.

Ideally find a fix for mosestokenizer...

mohammad2928 commented 3 years ago

I will apply in the next version (version 1.0.7).

mohammad2928 commented 3 years ago

improved.