Open mynewstart opened 12 months ago
I have met the same problem and my solution is to use deepspeed zero2 instead of zero3
My solution is to save checkpoints by myself or you can use zero_to_fp32
My solution is to save checkpoints by myself or you can use zero_to_fp32
@mynewstart I found my converted ckpt global_step_xxx
only contains meaningful *optim_states.pt
but only empty *model_states.pt
. Any clues on this? Thanks.
Required prerequisites
Questions
Hi, I used the code to continue pretrain the model and used zero3 for model training. But I found my checkpoint file zero_pprank*_mp_rank_00_model_states.pt is empty, the file only has model parameters name and shape, don't have the weights. Have you ever met this problem and how to fix?
Thanks!
Checklist