Closed Juuustin closed 4 years ago
WeijingShi
commented
4 years ago Hi, I think you run out of GPU memory during the test. You can try to increase GPU memory, downsampling, cropping the point cloud, etc. If you do not mind little accuracy loss, you can also modify the config file and replace the content in "runtime_graph_gen_kwargs" with the same as in "graph_gen_kwargs". This usually boosts the speed and reduces the memory requirement a lot with little accuracy loss.
Juuustin
commented
4 years ago Hi, thank you for your reply, and it works well!
zjx99
commented
3 years ago @Juuustin, Hi, have you got promising results? I am also trying to do the test on my own data. But the problem is that I don't have images in accordance with lidar data, so the obtained files are all empty when using the KITTI images since they don't match with my lidar data. How do you deal with this issue? Do you have your own images? Really appreciate if you can provide some advice. Thanks.
Dear,
I tried to test the Point-GNN on my own Lidar data, sometimes, there are errors, as is shown below. Could anyone tell me the reason for those errors? I am really appreciative for any advices!
`(py37) objloc@deeplearning:~/OBJLOC/Point-GNN$ python3 run.py checkpoints/car_auto_T3_trainval/ --test --dataset_root_dir pointGNN_dataset_small --output_dir OUTPUT --level 2 /home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) @ level 0 Graph, Add layer: layer1, type: scatter_max_point_set_pooling WARNING:tensorflow:From /home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. Feature Dim:300 @ level 1 Graph, Add layer: layer2, type: scatter_max_graph_auto_center_net Feature Dim:300 @ level 1 Graph, Add layer: layer3, type: scatter_max_graph_auto_center_net Feature Dim:300 @ level 1 Graph, Add layer: layer4, type: scatter_max_graph_auto_center_net Feature Dim:300 Final Feature Dim:300 Prediction 4 classes 2020-03-31 17:26:28.163067: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-03-31 17:26:28.185768: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3397755000 Hz 2020-03-31 17:26:28.186787: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55d32ffca7f0 executing computations on platform Host. Devices: 2020-03-31 17:26:28.186831: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0):,
2020-03-31 17:26:28.324051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:01:00.0
totalMemory: 11.91GiB freeMemory: 11.07GiB
2020-03-31 17:26:28.324083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-03-31 17:26:28.324847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-31 17:26:28.324860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2020-03-31 17:26:28.324867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2020-03-31 17:26:28.324936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10772 MB memory) -> physical GPU (device: 0, name: TITAN X (Pascal), pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-03-31 17:26:28.326513: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55d3306e2920 executing computations on platform CUDA. Devices:
2020-03-31 17:26:28.326531: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): TITAN X (Pascal), Compute Capability 6.1
Restore from checkpoint checkpoints/car_auto_T3_trainval/model-1400000
WARNING:tensorflow:From /home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
0%| | 0/21 [00:00<?, ?it/s]2020-03-31 17:26:29.195655: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
Configure the viewpoint as you want and press [q]
5%|████▌ | 1/21 [00:04<01:24, 4.22s/it]Configure the viewpoint as you want and press [q]
10%|█████████▏ | 2/21 [00:05<01:03, 3.37s/it]Configure the viewpoint as you want and press [q]
14%|█████████████▋ | 3/21 [00:06<00:49, 2.74s/it]Configure the viewpoint as you want and press [q]
19%|██████████████████▎ | 4/21 [00:08<00:38, 2.26s/it]Configure the viewpoint as you want and press [q]
24%|██████████████████████▊ | 5/21 [00:09<00:31, 1.94s/it]Configure the viewpoint as you want and press [q]
29%|███████████████████████████▍ | 6/21 [00:10<00:25, 1.73s/it]Configure the viewpoint as you want and press [q]
33%|████████████████████████████████ | 7/21 [00:11<00:21, 1.57s/it]Configure the viewpoint as you want and press [q]
38%|████████████████████████████████████▌ | 8/21 [00:12<00:18, 1.44s/it]Configure the viewpoint as you want and press [q]
43%|█████████████████████████████████████████▏ | 9/21 [00:14<00:17, 1.49s/it]Configure the viewpoint as you want and press [q]
48%|█████████████████████████████████████████████▏ | 10/21 [00:15<00:15, 1.42s/it]Configure the viewpoint as you want and press [q]
52%|█████████████████████████████████████████████████▊ | 11/21 [00:16<00:13, 1.34s/it]Configure the viewpoint as you want and press [q]
57%|██████████████████████████████████████████████████████▎ | 12/21 [00:29<00:43, 4.79s/it]Qt: Cannot set locale modifiers:
Configure the viewpoint as you want and press [q]
62%|██████████████████████████████████████████████████████████▊ | 13/21 [00:33<00:36, 4.54s/it]Configure the viewpoint as you want and press [q]
67%|███████████████████████████████████████████████████████████████▎ | 14/21 [00:59<01:15, 10.82s/it]Configure the viewpoint as you want and press [q]
71%|███████████████████████████████████████████████████████████████████▊ | 15/21 [01:08<01:02, 10.35s/it]2020-03-31 17:27:47.639016: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.30GiB. Current allocation summary follows.
2020-03-31 17:27:47.639090: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 93, Chunks in use: 93. 23.2KiB allocated for chunks. 23.2KiB in use in bin. 7.3KiB client-requested in use in bin.
2020-03-31 17:27:47.639113: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 6, Chunks in use: 6. 3.8KiB allocated for chunks. 3.8KiB in use in bin. 3.8KiB client-requested in use in bin.
2020-03-31 17:27:47.639134: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 36, Chunks in use: 36. 46.8KiB allocated for chunks. 46.8KiB in use in bin. 44.2KiB client-requested in use in bin.
2020-03-31 17:27:47.639153: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639170: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639188: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 1, Chunks in use: 1. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin.
2020-03-31 17:27:47.639208: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 4, Chunks in use: 4. 64.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-03-31 17:27:47.639228: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 1, Chunks in use: 1. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
2020-03-31 17:27:47.639249: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 9, Chunks in use: 9. 721.5KiB allocated for chunks. 721.5KiB in use in bin. 721.4KiB client-requested in use in bin.
2020-03-31 17:27:47.639269: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 1, Chunks in use: 1. 150.0KiB allocated for chunks. 150.0KiB in use in bin. 150.0KiB client-requested in use in bin.
2020-03-31 17:27:47.639289: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 15, Chunks in use: 13. 5.43MiB allocated for chunks. 4.58MiB in use in bin. 4.47MiB client-requested in use in bin.
2020-03-31 17:27:47.639309: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 1, Chunks in use: 1. 555.0KiB allocated for chunks. 555.0KiB in use in bin. 355.1KiB client-requested in use in bin.
2020-03-31 17:27:47.639327: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 1, Chunks in use: 0. 1.36MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639345: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639361: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639381: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 4, Chunks in use: 3. 42.66MiB allocated for chunks. 34.49MiB in use in bin. 27.42MiB client-requested in use in bin.
2020-03-31 17:27:47.639402: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 3, Chunks in use: 2. 75.98MiB allocated for chunks. 47.18MiB in use in bin. 46.70MiB client-requested in use in bin.
2020-03-31 17:27:47.639419: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639436: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639453: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 128.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639472: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 6, Chunks in use: 1. 8.25GiB allocated for chunks. 2.28GiB in use in bin. 2.28GiB client-requested in use in bin.
2020-03-31 17:27:47.639491: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 2.30GiB was 256.00MiB, Chunk State:
2020-03-31 17:27:47.639511: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 256.00MiB | Requested Size: 174.22MiB | in_use: 0
2020-03-31 17:27:47.639530: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.00GiB | Requested Size: 408.34MiB | in_use: 0
2020-03-31 17:27:47.639547: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.00GiB | Requested Size: 827.68MiB | in_use: 0
2020-03-31 17:27:47.639566: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.72GiB | Requested Size: 1.36MiB | in_use: 0, prev: Size: 2.28GiB | Requested Size: 2.28GiB | in_use: 1
2020-03-31 17:27:47.639583: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 2.00GiB | Requested Size: 1.01GiB | in_use: 0
2020-03-31 17:27:47.639601: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faada000000 of size 2448264192
2020-03-31 17:27:47.639615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fab6bed8c00 of size 1846703104
2020-03-31 17:27:47.639628: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fabe0000000 of size 2147483648
2020-03-31 17:27:47.639641: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7facc8000000 of size 1073741824
2020-03-31 17:27:47.639654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fad08000000 of size 1073741824
2020-03-31 17:27:47.639667: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fad48000000 of size 268435456
2020-03-31 17:27:47.639680: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fad58000000 of size 134217728
2020-03-31 17:27:47.639693: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fad60000000 of size 12432128
2020-03-31 17:27:47.639708: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fad60bdb300 of size 24482816
2020-03-31 17:27:47.639721: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fad62334700 of size 30193920
2020-03-31 17:27:47.639734: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fadd8000000 of size 1427456
2020-03-31 17:27:47.639748: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadd815c800 of size 15349760
2020-03-31 17:27:47.639761: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fadd9000000 of size 8564224
2020-03-31 17:27:47.639774: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadd982ae00 of size 24990208
2020-03-31 17:27:47.639788: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800000 of size 256
2020-03-31 17:27:47.639801: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800100 of size 256
2020-03-31 17:27:47.639814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800200 of size 256
2020-03-31 17:27:47.639827: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800300 of size 256
2020-03-31 17:27:47.639839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800400 of size 256
2020-03-31 17:27:47.639852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800500 of size 256
2020-03-31 17:27:47.639865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800600 of size 256
2020-03-31 17:27:47.639878: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800700 of size 256
2020-03-31 17:27:47.639891: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800800 of size 256
2020-03-31 17:27:47.639903: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800900 of size 256
2020-03-31 17:27:47.639916: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800a00 of size 256
2020-03-31 17:27:47.639929: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800b00 of size 256
2020-03-31 17:27:47.639941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800c00 of size 256
2020-03-31 17:27:47.639954: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800d00 of size 256
2020-03-31 17:27:47.639967: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800e00 of size 256
2020-03-31 17:27:47.639980: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800f00 of size 256
2020-03-31 17:27:47.639992: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801000 of size 256
2020-03-31 17:27:47.640005: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801100 of size 256
2020-03-31 17:27:47.640018: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801200 of size 256
2020-03-31 17:27:47.640031: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801300 of size 256
2020-03-31 17:27:47.640044: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801400 of size 256
2020-03-31 17:27:47.640056: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801500 of size 256
2020-03-31 17:27:47.640069: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801600 of size 256
2020-03-31 17:27:47.640082: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801700 of size 256
2020-03-31 17:27:47.640094: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801800 of size 256
2020-03-31 17:27:47.640107: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801900 of size 256
2020-03-31 17:27:47.640120: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801a00 of size 256
2020-03-31 17:27:47.640132: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801b00 of size 256
2020-03-31 17:27:47.640145: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801c00 of size 256
2020-03-31 17:27:47.640158: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801d00 of size 256
2020-03-31 17:27:47.640170: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801e00 of size 256
2020-03-31 17:27:47.640183: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801f00 of size 256
2020-03-31 17:27:47.640196: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802000 of size 256
2020-03-31 17:27:47.640209: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802100 of size 256
2020-03-31 17:27:47.640222: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802200 of size 256
2020-03-31 17:27:47.640235: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802300 of size 256
2020-03-31 17:27:47.640248: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802400 of size 256
2020-03-31 17:27:47.640260: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802500 of size 256
2020-03-31 17:27:47.640273: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802600 of size 256
2020-03-31 17:27:47.640287: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802700 of size 1280
2020-03-31 17:27:47.640300: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802c00 of size 256
2020-03-31 17:27:47.640312: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802d00 of size 256
2020-03-31 17:27:47.640325: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802e00 of size 1280
2020-03-31 17:27:47.640338: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde803300 of size 256
2020-03-31 17:27:47.640351: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde803400 of size 1280
2020-03-31 17:27:47.640364: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde803900 of size 256
2020-03-31 17:27:47.640377: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde803a00 of size 1280
2020-03-31 17:27:47.640390: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde803f00 of size 256
2020-03-31 17:27:47.640403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804000 of size 256
2020-03-31 17:27:47.640416: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804100 of size 256
2020-03-31 17:27:47.640429: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804200 of size 256
2020-03-31 17:27:47.640442: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804300 of size 256
2020-03-31 17:27:47.640455: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804400 of size 1280
2020-03-31 17:27:47.640467: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804900 of size 256
2020-03-31 17:27:47.640480: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804a00 of size 1280
2020-03-31 17:27:47.640493: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804f00 of size 256
2020-03-31 17:27:47.640506: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805000 of size 1280
2020-03-31 17:27:47.640519: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805500 of size 256
2020-03-31 17:27:47.640532: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805600 of size 1280
2020-03-31 17:27:47.640545: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805b00 of size 256
2020-03-31 17:27:47.640558: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805c00 of size 256
2020-03-31 17:27:47.640570: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805d00 of size 256
2020-03-31 17:27:47.640584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805e00 of size 256
2020-03-31 17:27:47.640597: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805f00 of size 256
2020-03-31 17:27:47.640610: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde806000 of size 1280
2020-03-31 17:27:47.640622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde806500 of size 256
2020-03-31 17:27:47.640635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde806600 of size 1280
2020-03-31 17:27:47.640649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde806b00 of size 256
2020-03-31 17:27:47.640665: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde806c00 of size 1280
2020-03-31 17:27:47.640676: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807100 of size 256
2020-03-31 17:27:47.640687: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807200 of size 1280
2020-03-31 17:27:47.640700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807700 of size 256
2020-03-31 17:27:47.640724: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807800 of size 256
2020-03-31 17:27:47.640737: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807900 of size 1280
2020-03-31 17:27:47.640749: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807e00 of size 256
2020-03-31 17:27:47.640762: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807f00 of size 256
2020-03-31 17:27:47.640775: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808000 of size 512
2020-03-31 17:27:47.640788: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808200 of size 256
2020-03-31 17:27:47.640801: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808300 of size 256
2020-03-31 17:27:47.640814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808400 of size 256
2020-03-31 17:27:47.640827: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808500 of size 256
2020-03-31 17:27:47.640839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808600 of size 256
2020-03-31 17:27:47.640852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808700 of size 256
2020-03-31 17:27:47.640865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808800 of size 256
2020-03-31 17:27:47.640877: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808900 of size 1280
2020-03-31 17:27:47.640890: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808e00 of size 256
2020-03-31 17:27:47.640903: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808f00 of size 1280
2020-03-31 17:27:47.640915: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde809400 of size 1280
2020-03-31 17:27:47.640929: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde809900 of size 1792
2020-03-31 17:27:47.640941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde80a000 of size 256
2020-03-31 17:27:47.640955: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde80a100 of size 16384
2020-03-31 17:27:47.640968: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde80e100 of size 256
2020-03-31 17:27:47.640981: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde80e200 of size 76800
2020-03-31 17:27:47.640994: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde820e00 of size 256
2020-03-31 17:27:47.641006: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde820f00 of size 1792
2020-03-31 17:27:47.641019: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde821600 of size 256
2020-03-31 17:27:47.641032: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde821700 of size 16384
2020-03-31 17:27:47.641045: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde825700 of size 256
2020-03-31 17:27:47.641057: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde825800 of size 76800
2020-03-31 17:27:47.641071: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde838400 of size 256
2020-03-31 17:27:47.641084: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde838500 of size 1792
2020-03-31 17:27:47.641097: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde838c00 of size 256
2020-03-31 17:27:47.641110: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde838d00 of size 16384
2020-03-31 17:27:47.641123: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde83cd00 of size 256
2020-03-31 17:27:47.641136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde83ce00 of size 76800
2020-03-31 17:27:47.641148: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde84fa00 of size 256
2020-03-31 17:27:47.641161: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde84fb00 of size 1792
2020-03-31 17:27:47.641174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde850200 of size 256
2020-03-31 17:27:47.641186: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde850300 of size 16384
2020-03-31 17:27:47.641199: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde854300 of size 256
2020-03-31 17:27:47.641211: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde854400 of size 76800
2020-03-31 17:27:47.641224: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde867000 of size 256
2020-03-31 17:27:47.641237: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde867100 of size 1024
2020-03-31 17:27:47.641250: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde867500 of size 256
2020-03-31 17:27:47.641263: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde867600 of size 76800
2020-03-31 17:27:47.641275: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde87a200 of size 256
2020-03-31 17:27:47.641289: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde87a300 of size 768
2020-03-31 17:27:47.641301: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde87a600 of size 256
2020-03-31 17:27:47.641314: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde87a700 of size 76800
2020-03-31 17:27:47.641327: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde88d300 of size 256
2020-03-31 17:27:47.641340: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde88d400 of size 470016
2020-03-31 17:27:47.641353: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddec00000 of size 1280
2020-03-31 17:27:47.641366: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddec00500 of size 363776
2020-03-31 17:27:47.641379: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddec59200 of size 1280
2020-03-31 17:27:47.641392: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddec59700 of size 360192
2020-03-31 17:27:47.641405: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddecb1600 of size 1280
2020-03-31 17:27:47.641418: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddecb1b00 of size 360192
2020-03-31 17:27:47.641431: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded09a00 of size 1280
2020-03-31 17:27:47.641444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded09f00 of size 768
2020-03-31 17:27:47.641456: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded0a200 of size 256
2020-03-31 17:27:47.641469: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded0a300 of size 76800
2020-03-31 17:27:47.641482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded1cf00 of size 256
2020-03-31 17:27:47.641495: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded1d000 of size 360192
2020-03-31 17:27:47.641507: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded74f00 of size 1280
2020-03-31 17:27:47.641520: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded75400 of size 568320
2020-03-31 17:27:47.641534: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddee00000 of size 1280
2020-03-31 17:27:47.641546: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddee00500 of size 360192
2020-03-31 17:27:47.641559: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddee58400 of size 1280
2020-03-31 17:27:47.641572: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddee58900 of size 360192
2020-03-31 17:27:47.641584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeeb0800 of size 1280
2020-03-31 17:27:47.641597: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeeb0d00 of size 768
2020-03-31 17:27:47.641610: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeeb1000 of size 256
2020-03-31 17:27:47.641622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeeb1100 of size 76800
2020-03-31 17:27:47.641662: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeec3d00 of size 256
2020-03-31 17:27:47.641678: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeec3e00 of size 360192
2020-03-31 17:27:47.641690: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddef1bd00 of size 1280
2020-03-31 17:27:47.641703: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddef1c200 of size 363776
2020-03-31 17:27:47.641716: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddef74f00 of size 1280
2020-03-31 17:27:47.641728: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddef75400 of size 360192
2020-03-31 17:27:47.641741: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddefcd300 of size 1280
2020-03-31 17:27:47.641754: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddefcd800 of size 360192
2020-03-31 17:27:47.641766: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf025700 of size 1280
2020-03-31 17:27:47.641779: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf025c00 of size 153600
2020-03-31 17:27:47.641792: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf04b400 of size 1280
2020-03-31 17:27:47.641805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf04b900 of size 32768
2020-03-31 17:27:47.641818: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf053900 of size 512
2020-03-31 17:27:47.641831: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf053b00 of size 8192
2020-03-31 17:27:47.641843: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf055b00 of size 256
2020-03-31 17:27:47.641856: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf055c00 of size 512
2020-03-31 17:27:47.641869: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf055e00 of size 256
2020-03-31 17:27:47.641882: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf055f00 of size 360192
2020-03-31 17:27:47.641894: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf0ade00 of size 1280
2020-03-31 17:27:47.641907: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf0ae300 of size 360192
2020-03-31 17:27:47.641920: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf106200 of size 1280
2020-03-31 17:27:47.641932: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7faddf106700 of size 429568
2020-03-31 17:27:47.641946: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf16f500 of size 124416
2020-03-31 17:27:47.641958: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7faddf18db00 of size 468224
2020-03-31 17:27:47.641972: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf800000 of size 8388608
2020-03-31 17:27:47.641984: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size:
2020-03-31 17:27:47.642001: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 93 Chunks of size 256 totalling 23.2KiB
2020-03-31 17:27:47.642017: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2020-03-31 17:27:47.642032: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 768 totalling 2.2KiB
2020-03-31 17:27:47.642047: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1024 totalling 1.0KiB
2020-03-31 17:27:47.642063: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 31 Chunks of size 1280 totalling 38.8KiB
2020-03-31 17:27:47.642078: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1792 totalling 7.0KiB
2020-03-31 17:27:47.642093: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 8192 totalling 8.0KiB
2020-03-31 17:27:47.642109: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 16384 totalling 64.0KiB
2020-03-31 17:27:47.642124: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 32768 totalling 32.0KiB
2020-03-31 17:27:47.642140: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 8 Chunks of size 76800 totalling 600.0KiB
2020-03-31 17:27:47.642156: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 124416 totalling 121.5KiB
2020-03-31 17:27:47.642172: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 153600 totalling 150.0KiB
2020-03-31 17:27:47.642187: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 360192 totalling 3.43MiB
2020-03-31 17:27:47.642203: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 363776 totalling 710.5KiB
2020-03-31 17:27:47.642219: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 470016 totalling 459.0KiB
2020-03-31 17:27:47.642234: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 568320 totalling 555.0KiB
2020-03-31 17:27:47.642249: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 8388608 totalling 8.00MiB
2020-03-31 17:27:47.642266: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12432128 totalling 11.86MiB
2020-03-31 17:27:47.642282: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 15349760 totalling 14.64MiB
2020-03-31 17:27:47.642297: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 24482816 totalling 23.35MiB
2020-03-31 17:27:47.642313: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 24990208 totalling 23.83MiB
2020-03-31 17:27:47.642328: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2448264192 totalling 2.28GiB
2020-03-31 17:27:47.642343: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 2.37GiB
2020-03-31 17:27:47.642361: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit: 11295313101
InUse: 2540349952
MaxInUse: 2540474368
NumAllocs: 1316
MaxAllocSize: 2448264192
2020-03-31 17:27:47.642408: W tensorflow/core/common_runtime/bfc_allocator.cc:271] ***___* 2020-03-31 17:27:47.642457: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at concat_op.cc:153 : Resource exhausted: OOM when allocating tensor with shape[2040220,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(args) File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2040220,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node layer2/concat}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "run.py", line 260, in
results = sess.run(fetches, feed_dict=feed_dict)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2040220,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node layer2/concat (defined at /home/objloc/OBJLOC/Point-GNN/models/gnn.py:352) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'layer2/concat', defined at: File "run.py", line 140, in
t_is_training)
File "/home/objloc/OBJLOC/Point-GNN/models/models.py", line 146, in predict
layer_kwargs)
File "/home/objloc/OBJLOC/Point-GNN/models/gnn.py", line 352, in apply_regular
axis=-1)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, *kwargs)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py", line 1256, in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1149, in concat_v2
"ConcatV2", values=values, axis=axis, name=name)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(args, kwargs)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[2040220,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node layer2/concat (defined at /home/objloc/OBJLOC/Point-GNN/models/gnn.py:352) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
71%|███████████████████████████████████████████████████████████████████▊ | 15/21 [01:18<00:31, 5.25s/it] (py37) objloc@deeplearning:~/OBJLOC/Point-GNN$ `