ResourceExhaustedError: OOM when allocating tensor with shape[3,16,1024,1024]

openai / gpt-2

Code for the paper "Language Models are Unsupervised Multitask Learners"

https://openai.com/blog/better-language-models/

Other

22.58k stars 5.53k forks source link

ResourceExhaustedError: OOM when allocating tensor with shape[3,16,1024,1024] #173

Open loretoparisi opened 5 years ago

loretoparisi commented 5 years ago

I'm getting this error when training the model 345M on a dual GTX 1080i - 8GB RAM.

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[3,16,1024,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

MrKrzYch00 commented 5 years ago

Doesn't it require the same amount of memory for each? On CPU it took ~11GB to train 345M so I guess without 24GB Titan it's no go?