RUCKBReasoning / RESDSQL

The Pytorch implementation of RESDSQL (AAAI 2023).
https://arxiv.org/abs/2302.05965
MIT License
243 stars 59 forks source link

Training cross-coder error #69

Open zhihui-shao opened 8 months ago

zhihui-shao commented 8 months ago

Traceback (most recent call last): File "schema_item_classifier.py", line 463, in <module> _train(opt) File "schema_item_classifier.py", line 271, in _train model_outputs = model( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/szh/RESDSQL/utils/classifier_model.py", line 185, in forward = self.table_column_cls( File "/home/shaozhihui/szh/RESDSQL/utils/classifier_model.py", line 106, in table_column_cls encoder_output = self.plm_encoder( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/.local/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 852, in forward encoder_outputs = self.encoder( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/.local/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 527, in forward layer_outputs = layer_module( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/.local/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 411, in forward self_attention_outputs = self.attention( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/.local/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 338, in forward self_outputs = self.self( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/.local/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 234, in forward attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2)) RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when callingcublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)``

What's the cause, please?

lihaoyang-ruc commented 7 months ago

This could be due to issues within your Python environment.