Incrementally adding new languages to multilingual checkpoint

[vnikouliNLE]

Reproduce the same experiments as done by @yongzx on English checkpoint with multilingual checkpoint: https://huggingface.co/bigscience/tr5a-1B3-multilingual-mt5tok-checkpoints/tree/global_step30000 Related issues:

• https://github.com/bigscience-workshop/multilingual-modeling/pull/3

• https://github.com/bigscience-workshop/multilingual-modeling/issues/2

  TODOs

  • [ ] zero-shot evaluation of multilingual checkpoint on German and Korean
  • [ ] finetune embedding layer on the German/Korean Oscar dataset
  • [ ] train adapter layer on German/Korean dataset

[yongzx]

Thanks @vnikouliNLE! For the adapter layer training, I think I should work on this instead of GPT-2, since now we have this model.

[yongzx]

Here's the link to the repo for incremental-addition-of-languages: https://github.com/bigscience-workshop/multilingual-modeling/tree/+lang