Open fmichaelkunz opened 3 years ago
Hi @fmichaelkunz thanks for pointing this out.
When you know the language, you can always provide the source_lang parameter. In that case, no automatic language detection is performed.
Otherwise you can overwrite the lang detector:
def my_lang_detection(text):
lang = model.language_detection_fasttext(text)
if lang in ['yue', 'wuu', 'min']:
lang = 'zh'
return lang
model = EasyNMT(...)
model._lang_detectors = [my_lang_detection]
EasyNMT uses fasttext to identify language. Some chinese phrases can be misidentified into chinese variants, like 'yue' or 'wuu'. This will cause easyNMT to fail. Can you map Chinese language variants to Chinese. 'yue', 'wuu', 'min' to 'zh'?