ParikhKadam
commented
5 years ago I have faced similar issues. The only thing you can do is create tensors that can fit in your memory. This error can solved by reducing text vector dimension. But that will lead towards low accuracy of your model. You can try reducing batch size which will not affect the model much and will throw this error.
For more help, you need to share your code/model architecture so that one can understand how much memory you are actually allocating to tensors and where we can reduce the memory usage.
Aravinviju
msymp
I am getting a ResourceExhaustedError: OOM - When Doing an embedding usinfg google's trasformer Architecture which embeds the text into a 512 dimensional vectors.
The data I'm trying to embed has 5000 records which adds up to 40MB of data. GPU used: Tesla k80 in a GCP instance. CPUs : 4 (15mb RAM) Tensorflow: tensorflow-gpu (3.0.1)
HERE IS THE CODE SNIPPET:
with tf.Session() as session: session.run([tf.global_variables_initializer(), tf.tables_initializer()]) message_embeddings = session.run(embed(test_cleansed_data))Here is the log:
ResourceExhaustedError Traceback (most recent call last) ~/anaconda3/envs/cluster_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, args) 1326 try: -> 1327 return fn(args) 1328 except errors.OpError as e:
~/anaconda3/envs/cluster_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata) 1311 return self._call_tf_sessionrun( -> 1312 options, feed_dict, fetch_list, target_list, run_metadata) 1313
~/anaconda3/envs/cluster_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata) 1419 self._session, options, feed_dict, fetch_list, target_list, -> 1420 status, run_metadata) 1421
~/anaconda3/envs/cluster_gpu/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py in exit(self, type_arg, value_arg, traceback_arg) 515 compat.as_text(c_api.TF_Message(self.status.status)), --> 516 c_api.TF_GetCode(self.status.status)) 517 # Delete the underlying status object from memory otherwise it stays alive
ResourceExhaustedError: OOM when allocating tensor with shape[4096000,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: module_apply_default_5/Encoder_en/Transformer/TransformerEncodeFast/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Softmax = SoftmaxT=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
ResourceExhaustedError Traceback (most recent call last)