Add more models

[Bachstelze]

In some (english-centric or low ressource) cases other models could yield better results, e.g. mBART or SMaLL100: https://huggingface.co/facebook/mbart-large-50-many-to-one-mmt https://huggingface.co/alirezamsh/small100 has another tokenizer Could we include them?

[ikergarcia1996]

Hi @Bachstelze, both of them are already supported. You can use them with the following command. I will update the README.md to clarify it.

python3 translate.py \
--sentences_path sample_text/en.txt \
--output_path sample_text/en2es.translation.small100.txt \
--source_lang en \
--target_lang es \
--model_name alirezamsh/small100

python3 translate.py \
--sentences_path sample_text/en.txt \
--output_path sample_text/en2es.translation.mbart.txt \
--source_lang hi_IN \
--target_lang fr_XX \
--model_name facebook/mbart-large-50-many-to-many-mmt

[ikergarcia1996]

Sorry, I just saw that small100 needs a custom tokenizer.py file. I will try to update Easy-Translate to support it.

[ikergarcia1996]

The last commit adds support for Small100 You can use the example scripts here

• examples/small100.sh
• examples/mbart.sh

I tested both of them

• Small 100 output
• Small 100 evaluation result
• Mbart-50-large-m2m output
• Mbart-50-large-m2m evaluation result

Let me know if have any problems running those models