Fixes all current issues with FLORES V1 - Githubissues

facebookresearch / flores

Facebook Low Resource (FLoRes) MT Benchmark

Other

705 stars 123 forks source link

Fixes all current issues with FLORES V1 #40

Open mnoukhov opened 2 years ago

mnoukhov commented 2 years ago

A working branch of FLORES v1

Tested reproduce.sh neen and got 12.5 BLEU on devtest after 2 iterations of BT (compared to README's 15.9). I used 1 RTX8000 and the full pipeline ran in ~100 hours (after adjusting max_tokens to 16000 to eliminate unnecessary update_freq)

Download Issues:

Fixes GlobalVoices download link #20
Corrects download path #33
Fixes download_indic.sh git command #22
Removes download of en-hidataset as it is now manual download only (https://www.cfilt.iitb.ac.in/iitb_parallel/dataset.html)

Other issues:

fixes old fairseq-train args e.g. min-lr
converts old args to omegaconf cfg
skips inputs that are greater than 1024 tokens when doing backtranslation

mnoukhov commented 2 years ago

@guzmanhe thanks! I rebased + merged the two previous PRs so if they are accepted I should have no merge conflicts but let me know if there are issues