cuda error

[x1597275]

03/16/2021 16:09:53 - INFO - transformers.modeling_utils - loading weights file /home/lab/Desktop/xf_event_extraction2020Top1-master/bert/torch_roberta_wwm/pytorch_model.bin 03/16/2021 16:16:01 - INFO - src_final.utils.functions_utils - Use single gpu in: ['1'] 03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Running training 03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Num Examples = 7416 03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Num Epochs = 6 03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Total training batch size = 16 03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Total optimization steps = 2784 03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Save model in 464 steps; Eval model in 464 steps Traceback (most recent call last): File "train.py", line 223, in training(args) File "train.py", line 152, in training train_base(opt, info_dict, train_examples, dev_info) File "train.py", line 44, in train_base train(opt, model, train_dataset) File "/home/lab/Desktop/xf_event_extraction2020Top1-master/src_final/utils/trainer.py", line 136, in train loss = model(batch_data)[0] File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(*input, *kwargs) File "/home/lab/Desktop/xf_event_extraction2020Top1-master/src_final/utils/model_utils.py", line 239, in forward token_type_ids=token_type_ids File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(input, kwargs) File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/transformers/modeling_bert.py", line 734, in forward encoder_attention_mask=encoder_extended_attention_mask, File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(*input, kwargs) File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/transformers/modeling_bert.py", line 407, in forward hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(*input, *kwargs) File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/transformers/modeling_bert.py", line 368, in forward self_attention_outputs = self.attention(hidden_states, attention_mask, head_mask) File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(input, kwargs) File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/transformers/modeling_bert.py", line 314, in forward hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(*input, *kwargs) File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/transformers/modeling_bert.py", line 216, in forward mixed_query_layer = self.query(hidden_states) File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(input, **kwargs) File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward return F.linear(input, self.weight, self.bias) File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/functional.py", line 1612, in linear output = input.matmul(weight.t()) RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc) 你好，我在运行train.sh 一直报这个错误，我看了一下我的cuda正常啊，batchsize也调整了，也找了相关的资料，还是报错，不知道为什么，可以请您帮忙解答一下吗

[WuHuRestaurant]

显卡型号是什么；cuda版本要对应我给的版本；还有再Linux上跑，不要再windows上运行

[x1597275]

问题解决了，我用的是一块显卡跑的，改成两块就不报错了

[x1597275]

03/16/2021 16:31:40 - INFO - transformers.modeling_utils - loading weights file /home/lab/Desktop/xf_event_extraction2020Top1-master/bert/torch_roberta_wwm/pytorch_model.bin 03/16/2021 16:36:53 - INFO - src_final.utils.functions_utils - Use multi gpus in: ['0', '1'] 03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Running training 03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Num Examples = 7416 03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Num Epochs = 6 03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Total training batch size = 16 03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Total optimization steps = 2784 03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Save model in 464 steps; Eval model in 464 steps terminate called after throwing an instance of 'std::runtime_error' what(): NCCL Error 1: unhandled cuda error ./script/final/train.sh: line 38: 6386 Aborted (core dumped) python train.py --gpu_ids=$GPU_IDS --mode=$MODE --raw_data_dir=$RAW_DATA_DIR --mid_data_dir=$MID_DATA_DIR --aux_data_dir=$AUX_DATA_DIR --bert_dir=$BERT_DIR --output_dir=$OUTPUT_DIR --bert_type=$BERT_TYPE --task_type=$TASK_TYPE --max_seq_len=320 --train_epochs=6 --train_batch_size=16 --lr=2e-5 --other_lr=2e-4 --attack_train="pgd" --swa_start=4 --eval_model --enhance_data --use_trigger_distance 现在报这个错，还是cuda问题？？我是按照项目里面进行配置的，也是在linux下跑的

[x1597275]

我直接在虚拟环境中安装了cuda10.1，但还是报上面的错误

[x1597275]

我的显卡型号是GeForce RTX 3080

[WuHuRestaurant]

3080在pytorch环境上一直有bug，现在应该是还没有修复