search

mandarjoshi90 / coref

BERT for Coreference Resolution
Apache License 2.0
440 stars 92 forks source link

Resource exhausted: OOM when allocating tensor with shape... #68

Open wanxinxie opened 3 years ago

wanxinxie commented 3 years ago

Hi when I run Jonathan K. Kummerfeld's notebook. When run the "Run prediction" section, line: ! GPU=0 python predict.py $CHOSEN_MODEL sample.in.json sample.out.txt

it shows the following:

2020-10-08 16:48:11.126861: W tensorflow/core/common_runtime/bfcallocator.cc:319] ****____***__** 2020-10-08 16:48:11.134264: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at tile_ops.cc:220 : Resource exhausted: OOM when allocating tensor with shape[202893,17445] and type int32 on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call return fn(*args) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: OOM when allocating tensor with shape[202893,17445] and type int32 on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node Tile_1}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[TopKV2/_1997]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[202893,17445] and type int32 on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node Tile_1}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations. 0 derived errors ignored.

jkkummerfeld commented 3 years ago

There was some software dependency issues that have now been fixed. You may want to try again.