CPM Bee 微调时设置 half 出现 CUDA 报错，不设置 half 则 assert 报错

[YingLaiLin]

CPM 使用微调脚本训练， 不开启 --use-delta 这一选项，则出现如下错误： Traceback (most recent call last): File "finetune_cpm_bee.py", line 503, in main() File "finetune_cpm_bee.py", line 499, in main finetune(args, tokenizer, model, optimizer, lr_scheduler, optim_manager) File "finetune_cpm_bee.py", line 364, in finetune optim_manager.step() File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/bmtrain/optim/optim_manager.py", line 131, in step optimizer.step(scale=self.loss_scale) File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/torch/optim/optimizer.py", line 109, in wrapper return func(*args, *kwargs) File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(args, **kwargs) File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/bmtrain/optim/adam_offload.py", line 77, in step state["_grad_fp16"] = torch.empty(p.size(), dtype=torch.float16, pin_memory=True) # on host RuntimeError: CUDA error: OS call failed or operation not supported on this OS CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.

[YingLaiLin]

CPM 微调脚本训练，不开启 --use-delta， 并且设置配置文件中 half 为 false，则出现如下错误： Traceback (most recent call last): File "finetune_cpm_bee.py", line 503, in main() File "finetune_cpm_bee.py", line 499, in main finetune(args, tokenizer, model, optimizer, lr_scheduler, optim_manager) File "finetune_cpm_bee.py", line 352, in finetune loss = loss_func(logits.view(-1, logits.size(-1)), targets.view(-1)) File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/bmtrain/loss/cross_entropy.py", line 192, in forward ret = OpFusedCrossEntropy.apply(input, target.int(), self.ignore_index) # return float tensor File "/ms_test2/miniconda3/envs/ms1.11/lib/python3.7/site-packages/bmtrain/loss/cross_entropy.py", line 18, in forward ignore_index, RuntimeError: input.dtype() == torch::kHalf INTERNAL ASSERT FAILED at "/tmp/pip-install-clhfk_l1/bmtrain_fe0a61bb02844d4b85067c24e12d4e87/csrc/cross_entropy_loss.cpp":25, please report a bug to PyTorch. input must be a half tensor

并且无法 /tmp 目录下找到该文件。

[YingLaiLin]

这个是否是由于开启 cpu offload 导致的， bmtrain 是否有相关开关控制呢？

[gongbaitao]

您好，这是由于此前loss_func算子仅支持半精度，现已修复