Closed MatejUlcar closed 1 year ago
Hmmm. We haven’t really been using ZeRO 3, as our testing indicates that it’s a big step up in complexity for a minimal increase in performance. It’s possible we broke something without realizing it.
I’ll have to train a model and play with it a bit, can you post the config file? Alternatively, would you be able to share the trained model by any chance? Hopefully we can get this resolved.
I've figured it's less bother to train from scratch without ZeRO 3, seeing how GPUs were under-utilized in the original run. Either way, I can share the trained model https://drive.google.com/drive/folders/13Z4g4eGFd33yhI2HWj4EM4G6lGw2W24D?usp=sharing The config files included in the archive, the model config here:
{
# parallelism settings ( you will want to change these based on your cluster setup, ideally scheduling pipeline stages
# across the node boundaries )
"pipe-parallel-size": 0,
"model-parallel-size": 1,
# model settings
"num-layers": 12,
"hidden-size": 768,
"num-attention-heads": 12,
"seq-length": 2048,
"max-position-embeddings": 2048,
"norm": "layernorm",
"pos-emb": "rotary",
"no-weight-tying": true,
# this should provide some speedup but takes a while to build, set to true if desired
"scaled-upper-triang-masked-softmax-fusion": false,
"train-iters": 80000,
# optimizer settings
"optimizer": {
"type": "Adam",
"params": {
"lr": 1.0e-5,
#"freeze_step": 5000,
"betas": [0.9, 0.999],
#"cuda_aware": false,
#"comm_backend_name": "nccl"
}
},
"zero_optimization": {
"stage": 3,
"allgather_partitions": True,
"allgather_bucket_size": 100000000,
"overlap_comm": True,
"reduce_scatter": True,
"reduce_bucket_size": 100000000,
"contiguous_gradients": True,
"cpu_offload": False
},
"zero_allow_untested_optimizer": true,
# batch / data settings
"train_micro_batch_size_per_gpu": 8,
"gradient_accumulation_steps": 4,
"data-impl": "mmap",
"split": "949,50,1",
# activation checkpointing
"checkpoint-activations": true,
"checkpoint-num-layers": 1,
"partition-activations": true,
"synchronize-each-layer": true,
# regularization
"gradient_clipping": 1.0,
"weight-decay": 0.05,
"hidden-dropout": 0.1,
"attention-dropout": 0.1,
# precision settings
"fp16": {
"enabled": true,
"loss_scale": 0,
"loss_scale_window": 500,
"hysteresis": 2,
"min_loss_scale": 1,
},
# lr decay settings
"lr-decay-iters": 80000,
"lr-decay-style": "cosine",
"warmup": 0.01,
# misc. training settings
"distributed-backend": "nccl",
#"save-interval": 10000,
#"eval-interval": 1000,
"save-interval": 500,
"eval-interval": 100,
"eval-iters": 10,
# logging
#"log-interval": 100,
"log-interval": 10,
"steps_per_print": 10,
"keep-last-n-checkpoints": 4,
"wall_clock_breakdown": true,
# sparse attention
#"attention_config": [[["local", "global"], "all"]],
}
We do not currently support ZeRO 3, which seems to be the core source of your issue. Closing for now.
Describe the bug Unable to convert a custom gpt-neox model (with zero stage 3) checkpoints using zero_to_fp32.py script.
To Reproduce Train a model with zero stage 3, pp=0, mp=1 (haven't attempted other combinations). Save checkpoint. Run the zero_to_fp32.py script, attached to the saved checkpoint.
Environment (please complete the following information):
Attempted so far Run as is:
Limit env to 1 GPU:
Adding map_location argument to torch.load as per above error
I have also attempted to convert with deepspeed's latest script, but I got an error that it's not a model state checkpoint, so I figure the differences between current deeperspeed and latest deepspeed are too great to be of any use. Please advice on how I could salvage, ie. actually use the model for inference/evaluation other than training again from scratch.