search

argosopentech / argos-translate

Open-source offline translation library written in Python
https://www.argosopentech.com
MIT License
3.87k stars 282 forks source link

Potential regression after updating to the latest version #121

Closed rushter closed 3 years ago

rushter commented 3 years ago

I've noticed different output from LibreTranslate and the latest version of argos-translate for Chinese language. LibreTranslate uses argostranslate==1.1.4.

I've tested this problem locally and got the same results. It looks like a regression to me. Both versions are using the same model.

Original sentence (in English, translated using GoogleTranslate): 

Hi there, this is a Chinese sentence. How are you doing today?

argostranslate==1.3.0

 echo "嗨，这是中文句子。 你今天过得怎么样?" | argos-translate --from-lang zh --to-lang en
Pierre, the Chinese sentence. What you have done today?

LibreTranslate (argostranslate==1.1.4):

This is the Chinese sentence. What is you doing today?
PJ-Finlay commented 3 years ago

That's strange the only significant change to the translation pipeline since then has been multiple hypotheses but in this case I don't think that would change the way CTranslate2 is called. We may have changed some of the CTranslate2 parameters since then but I'm not sure.

Overall though I don't have any reason to believe that translations have gotten worse.

rushter commented 3 years ago

The behavior changes after a few commits:

https://github.com/rushter/argos-translate/commit/528c63a5861a20938a1d62277a835f2431119d48 https://github.com/rushter/argos-translate/commit/ab57727a7b4a5f34e925f5aeb8739a3a0462cc5f

rushter commented 3 years ago

This is the parameter that changes the behavior: https://github.com/rushter/argos-translate/blob/03dcbcf53960c9248e699d212e16c2023de90def/argostranslate/translate.py#L396

Using beam_size=2 and num_hypotheses=1 gives the old bahaviour.

PJ-Finlay commented 3 years ago

That makes sense, beam_size would change the output sometimes. I think this should generally improve translations though, just not in this specific case.

PJ-Finlay commented 3 years ago

Actually maybe the issue is that when num_hypotheses is 1 it reduces beam_size which could make translations worse.

PJ-Finlay commented 3 years ago

I set a min beam size of 4. Hopefully this improves translations some, speed didn't change noticeably.

PJ-Finlay commented 3 years ago

https://forum.opennmt.net/t/training-convergence-and-beam-size-impact/3842/4