search

google-research / electra

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Apache License 2.0
2.34k stars 354 forks source link

failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED #136

Open EJDU21 opened 1 year ago

EJDU21 commented 1 year ago

How can I save this problem? I use 3080ti /tensorflow 1.15 / python 3.7.10 / cuda 10.1

2022-12-27 15:27:14.672772: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED ERROR:tensorflow:Error recorded from training_loop: 2 root error(s) found. (0) Internal: Blas GEMM launch failed : a.shape=(4096, 2), b.shape=(2, 128), m=4096, n=128, k=2 [[node electra/embeddings_1/MatMul (defined at /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] [[add_10/_9743]] (1) Internal: Blas GEMM launch failed : a.shape=(4096, 2), b.shape=(2, 128), m=4096, n=128, k=2 [[node electra/embeddings_1/MatMul (defined at /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] 0 successful operations. 0 derived errors ignored.

martinpolacek commented 1 year ago

Hey, you have to use NVIDIA version of TF, which supports training on RTX cards.

https://github.com/NVIDIA/tensorflow#install