[72B-Chat LoRA微调报错] AssertionError: {'id': 643, 'status': 'NOT_AVAILABLE', 'numel': 0, 'ds_numel': 0, 'shape': (0,), 'ds_shape': (0,), 'requires_grad': True, 'grad_shape': None, 'persist': True, 'active_sub_modules': {7}, 'ds_tensor.shape': torch.Size([0])} - Githubissues

QwenLM / Qwen

The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud.

Apache License 2.0

13.59k stars 1.11k forks source link

[72B-Chat LoRA微调报错] AssertionError: {'id': 643, 'status': 'NOT_AVAILABLE', 'numel': 0, 'ds_numel': 0, 'shape': (0,), 'ds_shape': (0,), 'requires_grad': True, 'grad_shape': None, 'persist': True, 'active_sub_modules': {7}, 'ds_tensor.shape': torch.Size([0])} #987

Closed chenyzh28 closed 8 months ago

chenyzh28 commented 8 months ago

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

[X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

[X] 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

训练报错

期望行为 | Expected Behavior

正常训练

复现方法 | Steps To Reproduce

运行finetune_lora_ds.sh，deepspeed，zero3模式，LoRA训练Qwen-72B-Chat。

运行环境 | Environment

- OS: Ubuntu 20.04
- Python: 3.8
- Transformers: 4.32.0
- PyTorch: 2.0.1
- CUDA 11.4

备注 | Anything else?

No response