Closed Sohaib90 closed 2 years ago
Hi @Sohaib90 , Thank you for your interest in code2seq!
Can you please provide:
--predict
instead of the --release
flag?CUDA_VISIBLE_DEVICES='' python3 -u code2seq.py ...
?Best, Uri
Create a
tf.sparse.SparseTensorand use
tf.sparse.to_dense` instead.
2021-08-17 11:43:51.175193: W tensorflow/core/framework/allocator.cc:122] Allocation of 806102784 exceeds 10% of system memory.
2021-08-17 11:44:19.267276: W tensorflow/core/framework/allocator.cc:122] Allocation of 806102784 exceeds 10% of system memory.
2021-08-17 11:44:52.016404: W tensorflow/core/framework/allocator.cc:122] Allocation of 806102784 exceeds 10% of system memory.
2021-08-17 11:44:56.527500: W tensorflow/core/framework/allocator.cc:122] Allocation of 806102784 exceeds 10% of system memory.
2021-08-17 11:45:01.210316: W tensorflow/core/framework/allocator.cc:122] Allocation of 806102784 exceeds 10% of system memory.
2021-08-17 11:51:22.784047: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 12.01GiB. Current allocation summary follows.
2021-08-17 11:51:22.784129: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 19, Chunks in use: 19. 4.8KiB allocated for chunks. 4.8KiB in use in bin. 76B client-requested in use in bin.
2021-08-17 11:51:22.784147: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784160: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2021-08-17 11:51:22.784175: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 2, Chunks in use: 2. 4.0KiB allocated for chunks. 4.0KiB in use in bin. 4.0KiB client-requested in use in bin.
2021-08-17 11:51:22.784188: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 1, Chunks in use: 1. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin.
2021-08-17 11:51:22.784199: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784211: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784223: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784235: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784249: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784273: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 1, Chunks in use: 1. 400.0KiB allocated for chunks. 400.0KiB in use in bin. 400.0KiB client-requested in use in bin.
2021-08-17 11:51:22.784287: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 1, Chunks in use: 1. 640.0KiB allocated for chunks. 640.0KiB in use in bin. 640.0KiB client-requested in use in bin.
2021-08-17 11:51:22.784298: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784310: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784321: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784333: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784347: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 1, Chunks in use: 1. 26.24MiB allocated for chunks. 26.24MiB in use in bin. 26.24MiB client-requested in use in bin.
2021-08-17 11:51:22.784358: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784370: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784381: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784394: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 10.16GiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-08-17 11:51:22.784406: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 12.01GiB was 256.00MiB, Chunk State:
2021-08-17 11:51:22.784423: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 10.16GiB | Requested Size: 0B | in_use: 0, prev: Size: 26.24MiB | Requested Size: 26.24MiB | in_use: 1
2021-08-17 11:51:22.784437: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000000 of size 256
2021-08-17 11:51:22.784447: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000100 of size 256
2021-08-17 11:51:22.784456: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000200 of size 256
2021-08-17 11:51:22.784465: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000300 of size 256
2021-08-17 11:51:22.784474: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000400 of size 256
2021-08-17 11:51:22.784484: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000500 of size 256
2021-08-17 11:51:22.784493: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000600 of size 256
2021-08-17 11:51:22.784502: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000700 of size 256
2021-08-17 11:51:22.784511: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000800 of size 256
2021-08-17 11:51:22.784520: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000900 of size 256
2021-08-17 11:51:22.784529: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000a00 of size 256
2021-08-17 11:51:22.784538: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000b00 of size 256
2021-08-17 11:51:22.784547: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da000c00 of size 2048
2021-08-17 11:51:22.784557: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da001400 of size 256
2021-08-17 11:51:22.784565: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da001500 of size 2048
2021-08-17 11:51:22.784574: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da001d00 of size 256
2021-08-17 11:51:22.784583: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da001e00 of size 256
2021-08-17 11:51:22.784592: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da001f00 of size 256
2021-08-17 11:51:22.784602: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da002000 of size 256
2021-08-17 11:51:22.784611: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da002100 of size 5120
2021-08-17 11:51:22.784620: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da003500 of size 256
2021-08-17 11:51:22.784630: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da003600 of size 256
2021-08-17 11:51:22.784639: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da003700 of size 1280
2021-08-17 11:51:22.784649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da003c00 of size 409600
2021-08-17 11:51:22.784658: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da067c00 of size 655360
2021-08-17 11:51:22.784668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fb0da107c00 of size 27517440
2021-08-17 11:51:22.784678: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fb0dbb45e00 of size 10904553472
2021-08-17 11:51:22.784687: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size:
2021-08-17 11:51:22.784698: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 19 Chunks of size 256 totalling 4.8KiB
2021-08-17 11:51:22.784709: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2021-08-17 11:51:22.784719: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2048 totalling 4.0KiB
2021-08-17 11:51:22.784728: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 5120 totalling 5.0KiB
2021-08-17 11:51:22.784739: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 409600 totalling 400.0KiB
2021-08-17 11:51:22.784749: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 655360 totalling 640.0KiB
2021-08-17 11:51:22.784759: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 27517440 totalling 26.24MiB
2021-08-17 11:51:22.784770: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 27.27MiB
2021-08-17 11:51:22.784783: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit: 10933151335
InUse: 28597760
MaxInUse: 28597760
NumAllocs: 26
MaxAllocSize: 275174402021-08-17 11:51:22.784808: W tensorflow/core/common_runtime/bfc_allocator.cc:271] ___ 2021-08-17 11:51:22.784885: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[25190712,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "/opt/dl/anaconda3/envs/tf112/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(args) File "/opt/dl/anaconda3/envs/tf112/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/opt/dl/anaconda3/envs/tf112/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[25190712,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node model/NODES_VOCAB/Initializer/random_uniform/RandomUniform}} = RandomUniformT=DT_INT32, _class=["loc:@model/NODES_VOCAB/Assign"], dtype=DT_FLOAT, seed=239, seed2=1058, _device="/job:localhost/replica:0/task:0/device:GPU:0"]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.`
Hi @Sohaib90 ,
It seems that you are running out of GPU memory.
Which GPU are you using, and how much memory does it have?
Can you share a screenshot of running:
nvidia-smi
?
Hello @urialon
Yes I think so too. GPU memory is 10.92GB and I think it runs out of memory when loading a pretrained model. Specifications are:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582 totalMemory: 10.92GiB freeMemory: 10.78GiB
Can you share a screenshot of runningnvidia-smi
:
Closing due to inactivity
First of all thank you so much for code2seq.
I have a rather odd problem, which might be because of environment issues but I still need a second opinion. When I train code2seq model, whilst also loading a pretrained model, it works fine. But, when I do not provide a training dataset and just load the model (for example to evaluate or releasing the model in a smaller size) the program aborts after giving the error
2021-08-10 11:44:55.713136: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1 Aborted (core dumped)
At first I thought maybe its an environment issue, which i reinstalled but it did not help
tensorflow-gpu 1.14.0 h0d30ee6_0 defaults cudatoolkit 10.1.243 h6bb024c_0 defaults cudnn 7.6.5 cuda10.1_0 defaults
Has anyone else faced the same problem
Example command on which the problem arises python3 -u code2seq.py --load path_to_model/model_iter29 --release