search

Helsinki-NLP / OPUS-CAT

OPUS-CAT is a collection of software which make it possible to OPUS-MT neural machine translation models in professional translation. OPUS-CAT includes a local offline MT engine and a collection of CAT tool plugins.
MIT License
70 stars 11 forks source link

CJK characters modified by translation (even in input tab) #83

Open seelebrn opened 1 year ago

seelebrn commented 1 year ago

Hello ! As per title, when trying to use a model (here, opus+bt-2021-04-30, multilingual) to translate a sentence (e.g. : "【测试】哎呀？我的台本哪里去了？我现在应该说啥？") to English, I noticed some characters (【 and 】) are modified even in the input tab.

【测试】哎呀？我的台本哪里去了？我现在应该说啥？

... becomes :

[测试]哎呀?我的台本哪里去了?我现在应该说啥?

So I suppose there's no way the translation is going to be exact. And indeed, it outputs :

[Test] Oh? Where is my script? What should I say now?

Is it possible to fix this so, for example, 【 and 】 are not transformed before generation ?

Thanks !

TommiNieminen commented 1 year ago

Hi,

It seems the lenticular brackets are converted to standard brackets by the OPUS MT model preprocessing, so the model just treats them as standard square brackets. This is probably an error, since at least according to Wikipedia, the lenticular brackets denote headings etc., i.e. they are not equivalent to standard brackets.

This can be fixed only by retraining the models, so I'll make a note of this and hopefully we can modify the preprocessing script for the next training run.

-Tommi

seelebrn commented 1 year ago

Thanks for answering ! I hope this will be possible indeed.