ResourceExhaustedError: OOM when allocating tensor with shape[501153,768] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Mul] #76
I am training a model using bert and TF2.3. I have loaded both language-agnostic Bert Sentence Encoder model (laBSE) and another multilingual model from TF hub or bert_multi_cased_L-12_H-768_A-12, and I get the following OOM error.
ResourceExhaustedError: OOM when allocating tensor with shape[501153,768] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Mul]
I have tried to reduce batch size as low as 8 then even 1, also lowered max_seq_length to 20, I have 4 GPUs of 16 GB, but I still get this error. What is the solution? Please help, thank you!
I am training a model using bert and TF2.3. I have loaded both language-agnostic Bert Sentence Encoder model (laBSE) and another multilingual model from TF hub or bert_multi_cased_L-12_H-768_A-12, and I get the following OOM error.
ResourceExhaustedError: OOM when allocating tensor with shape[501153,768] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Mul]
I have tried to reduce batch size as low as 8 then even 1, also lowered max_seq_length to 20, I have 4 GPUs of 16 GB, but I still get this error. What is the solution? Please help, thank you!
ResourceExhaustedError Traceback (most recent call last)