Closed Markfryazino closed 3 years ago
Thank you! First was only our typos. It is fixed and renamed. Try set parameter --load-openai for loading transformers checkpoint if you want to fit model on megatron. you can also download model from hugginface transformers.
I am totally confused about the details of GPT3 pretraining. First of all, file
pretrain_ruGPT3Large.sh
tries to run an unexisting python scriptpretrain_gpt2.py
. However, in readmepretrain_gpt2.py
is replaced withpretrain_megatron.py
.Secondly, there are some strange things about GPT3 checkpointing. The
pretrain_megatron.py
script operates model checkpoints as directories with .pt dumps, meanwhile,generate_ruGPT3Large.py
takes as input a directory with model.bin, vocab.json, etc. Also, the directory of this format (and not .pt file) can be downloaded from Google Drive.So, how should I finetune ruGPT3Large? It is obvious that just calling
pretrain_megatron.py
is not the correct way - at least because it starts from random due to mismatch of checkpoint files.