Closed vnikouliNLE closed 2 years ago
Thanks @vnikouliNLE! For the adapter layer training, I think I should work on this instead of GPT-2, since now we have this model.
Here's the link to the repo for incremental-addition-of-languages: https://github.com/bigscience-workshop/multilingual-modeling/tree/+lang
Reproduce the same experiments as done by @yongzx on English checkpoint with multilingual checkpoint: https://huggingface.co/bigscience/tr5a-1B3-multilingual-mt5tok-checkpoints/tree/global_step30000 Related issues:
https://github.com/bigscience-workshop/multilingual-modeling/issues/2
TODOs