Open lxd551326 opened 3 months ago
I have the same question in deepspeed stage3 but for the shape in current model is torch.Size([0]), please someone help us. T_T
@lxd551326, it seems you seeing two different issues.
For both above cases, it would be very helpful if you provide repro steps?
- DeepSpeed for a model that works with pure pytorch is very stran
Have you solved the problem? I meet it too. The shape is correct in my program without using deepspeed.
@lhyscau, @DavidYanAnDe, and @lxd551326 are you able to provide repro steps?
@lhyscau, @DavidYanAnDe, and @lxd551326 are you able to provide repro steps?
I comment the zero_optimizer param in the ds.config file, then the error doesn't happen.
Describe the bug i can only use pytorch to training model with Qwen1.5-7B. but when i use deepSpeed i got a problem with CUDA out of memory
my config with zeRo2 is:
{
"fp16": {
"enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
},
"bf16": {
"enabled": "auto"
},
"optimizer": {
"type": "AdamW",
"params": {
"lr": "auto",
"betas": "auto",
"eps": "auto",
"weight_decay": "auto"
}
},
} then i had change param with "offload_optimizer": { "device": "cpu", "pin_memory": true }, then it has an other problem like this: