[BUG/Help] RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

THUDM / ChatGLM2-6B

ChatGLM2-6B: An Open Bilingual Chat LLM | 开源双语对话语言模型

Other

15.68k stars 1.85k forks source link

[BUG/Help] RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)` #590

Open Alan-JW opened 10 months ago

Alan-JW commented 10 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

@duzx16 , 多卡运行为微调运行ptuning文件夹中的bash train.sh文件，设置NUM_GPUS=5，报错 17f2c109a9dc4eb16d6dbb13d2ca200

Expected Behavior

No response

Steps To Reproduce

1、bash train.sh

Environment

- OS:Ubuntu 20.04.4 LTS
- Python: 3.8.10
- Transformers:4.30.2
- PyTorch:2.0.1+cud117
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) : True

Anything else?

No response

Li-jiaxian commented 10 months ago

我也遇到了同样的问题，请问你有解决吗