Bitextor MT Models

This repository contains all you need to train models used for bitextor training in the HPLT project.

Structure

Configuration files are stored per language pair, i.e. the top level of this directory is a bunch of language pair directories.

Configuration files could include:

• OPUS-filter configurations
• OPUS-cleaner configurations (per dataset)
• bergamot pipeline configurations
• Just notes about which OPUS model you're distilling, using which datasets.

Download models:

v2

https://object.pouta.csc.fi/hplt_bitextor_models/bat-eng_tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/heb-eng_tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/kor-eng_tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/slk-eng_tiny.zip

v1

https://object.pouta.csc.fi/hplt_bitextor_models/ara_base.tar.gz \ https://object.pouta.csc.fi/hplt_bitextor_models/ara_tiny.tar.gz \ https://object.pouta.csc.fi/hplt_bitextor_models/ca-en_exported_base.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/ca-en_exported_tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/eus_base.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/eus_tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/gl-en_exported_base.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/gl-en_exported_tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/hin_base.tar.gz \ https://object.pouta.csc.fi/hplt_bitextor_models/hin_tiny.tar.gz \ https://object.pouta.csc.fi/hplt_bitextor_models/jpn-eng.base.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/jpn-eng.tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/sw-en_exported_base.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/sw-en_exported_tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/vie-eng.base.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/vie-eng.tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/zho_hans.base.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/zho_hans.tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/zho_hant.base.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/zho_hant.tiny.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/zho_joint.base.zip \ https://object.pouta.csc.fi/hplt_bitextor_models/zho_joint.tiny.zip