Can you provide the method to train using our own corpora using your version of fairseq ?

jerinphilip / ilmulti

Tooling to play around with multilingual machine translation for Indian Languages.

http://preon.iiit.ac.in/~jerin/bhasha

MIT License

21 stars 4 forks source link

Can you provide the method to train using our own corpora using your version of fairseq ? #8

Open vishnu3741 opened 3 years ago

vishnu3741 commented 3 years ago

I normally use indicnlp to tokenize and moses to train the MT but your model is giving better accuracy and can you give an insight into the amount or corpus used to train the model? Thank you.

jerinphilip commented 3 years ago

Perhaps the paper linked below will answer the corpus used.

Revisiting Low Resource Status of Indian Languages in Machine Translation.

Regarding the data/training:

fairseq/data/cvit/corpora.py has dataset with a tag based inclusion, which I specify through a configuration file (example). An example training script in our cluster looks like this.

vishnu3741 commented 3 years ago

hey, Is there way to add vocabulary (I mean words) to the model instead of retraining the entire model? can we edit the files in mm-all-iter1 to do this?

jerinphilip commented 3 years ago

This paper might have some useful information, I think. I'd just retrain with new vocabulary, turnaround is approximately 1 day or something on 4 GPUs to start getting reasonable numbers. This one used 1080Tis or 2080Tis.