search

facebookresearch / fairseq

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
MIT License
30.13k stars 6.36k forks source link

Translation Quality of NLLB-200(Dense, 3.3B) model is worse as compared to all other models for Japanese & English. Can anyone suggest why? #4641

Open suraj143rosy opened 2 years ago

suraj143rosy commented 2 years ago

Hi,

I have used all the different NLLB models for Japanese to English, and English to Japanese translations. I have observed that the translation quality of NLLB-200(Dense, 3.3B) is very bad when compared to all other models. I wanted to know the reason for this. Can Can someone suggest a reason for this?

vedanuj commented 2 years ago

Which models have you compared NLLB-200 3.3B dense with?

suraj143rosy commented 2 years ago

I have compared NLLB-200 3.3B dense with NLLB-200 54.5 B MoE, NLLB-200 1.3B dense and NLLB-200-Distilled 600M dense models.



gmryu commented 2 years ago

@suraj143rosy Are you comparing NLLB (multilingual model) to a only japanese-english model?

While this is not an absolute reason, it definetly impacts: https://github.com/facebookresearch/fairseq/issues/4560 So I believe a lot of characters are unk to the model.

suraj143rosy commented 2 years ago

No, I am comparing these NLLB models with each other.

vince62s commented 1 year ago

can you post your BLEU scores for all models ? did you see the same for Chinese or Korean ?