search

WeijingShi / Point-GNN

Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud, CVPR 2020.
MIT License
523 stars 114 forks source link

How to decrease batch size during testing? #34

Closed sarimmehdi closed 3 years ago

sarimmehdi commented 3 years ago

Hello, I get memory error half-way through testing your code. Is there any way to reduce the batch size during testing? This is the error I get:

2020-07-30 05:35:53.904913: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.02GiB (rounded to 1091328512).  Current allocation summary follows.
2020-07-30 05:35:53.905024: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (256):   Total Chunks: 93, Chunks in use: 93. 23.2KiB allocated for chunks. 23.2KiB in use in bin. 7.3KiB client-requested in use in bin.
2020-07-30 05:35:53.905071: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (512):   Total Chunks: 6, Chunks in use: 6. 3.8KiB allocated for chunks. 3.8KiB in use in bin. 3.8KiB client-requested in use in bin.
2020-07-30 05:35:53.905102: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1024):  Total Chunks: 36, Chunks in use: 36. 46.8KiB allocated for chunks. 46.8KiB in use in bin. 44.2KiB client-requested in use in bin.
2020-07-30 05:35:53.905130: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2048):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-07-30 05:35:53.905157: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4096):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-07-30 05:35:53.905187: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8192):  Total Chunks: 1, Chunks in use: 1. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin.
2020-07-30 05:35:53.905220: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16384):         Total Chunks: 4, Chunks in use: 4. 64.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-07-30 05:35:53.905252: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (32768):         Total Chunks: 2, Chunks in use: 2. 92.5KiB allocated for chunks. 92.5KiB in use in bin. 92.5KiB client-requested in use in bin.
2020-07-30 05:35:53.905284: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (65536):         Total Chunks: 7, Chunks in use: 7. 525.0KiB allocated for chunks. 525.0KiB in use in bin. 525.0KiB client-requested in use in bin.
2020-07-30 05:35:53.905317: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (131072):        Total Chunks: 2, Chunks in use: 2. 299.0KiB allocated for chunks. 299.0KiB in use in bin. 225.0KiB client-requested in use in bin.
2020-07-30 05:35:53.905347: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (262144):        Total Chunks: 14, Chunks in use: 13. 4.81MiB allocated for chunks. 4.48MiB in use in bin. 4.47MiB client-requested in use in bin.
2020-07-30 05:35:53.905379: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (524288):        Total Chunks: 2, Chunks in use: 1. 1.15MiB allocated for chunks. 562.5KiB in use in bin. 351.6KiB client-requested in use in bin.
2020-07-30 05:35:53.905410: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1048576):       Total Chunks: 2, Chunks in use: 0. 2.39MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-07-30 05:35:53.905439: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2097152):       Total Chunks: 1, Chunks in use: 1. 3.43MiB allocated for chunks. 3.43MiB in use in bin. 3.43MiB client-requested in use in bin.
2020-07-30 05:35:53.905469: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 6.80MiB allocated for chunks. 6.80MiB in use in bin. 3.43MiB client-requested in use in bin.
2020-07-30 05:35:53.905503: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8388608):       Total Chunks: 3, Chunks in use: 3. 31.98MiB allocated for chunks. 31.98MiB in use in bin. 26.52MiB client-requested in use in bin.
2020-07-30 05:35:53.905530: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-07-30 05:35:53.905562: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (33554432):      Total Chunks: 1, Chunks in use: 0. 43.39MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-07-30 05:35:53.905589: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-07-30 05:35:53.905617: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-07-30 05:35:53.905646: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (268435456):     Total Chunks: 3, Chunks in use: 1. 3.00GiB allocated for chunks. 1.01GiB in use in bin. 1.01GiB client-requested in use in bin.
2020-07-30 05:35:53.905678: I tensorflow/core/common_runtime/bfc_allocator.cc:885] Bin for 1.02GiB was 256.00MiB, Chunk State: 
2020-07-30 05:35:53.905719: I tensorflow/core/common_runtime/bfc_allocator.cc:891]   Size: 1019.84MiB | Requested Size: 358.83MiB | in_use: 0 | bin_num: 20, prev:   Size: 1.01GiB | Requested Size: 1.01GiB | in_use: 1 | bin_num: -1
2020-07-30 05:35:53.905750: I tensorflow/core/common_runtime/bfc_allocator.cc:891]   Size: 1.00GiB | Requested Size: 76.55MiB | in_use: 0 | bin_num: 20
2020-07-30 05:35:53.905772: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 2149908480
2020-07-30 05:35:53.905797: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fec12000000 next 168 of size 1080523264
2020-07-30 05:35:53.905820: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fec52677a00 next 18446744073709551615 of size 1069385216
2020-07-30 05:35:53.905841: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1073741824
2020-07-30 05:35:53.905862: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fec98000000 next 18446744073709551615 of size 1073741824
2020-07-30 05:35:53.905883: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 67108864
2020-07-30 05:35:53.905906: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed32000000 next 178 of size 10805248
2020-07-30 05:35:53.905927: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed32a4e000 next 169 of size 10805248
2020-07-30 05:35:53.905947: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fed3349c000 next 18446744073709551615 of size 45498368
2020-07-30 05:35:53.905967: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 8388608
2020-07-30 05:35:53.905988: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fed46600000 next 167 of size 1254400
2020-07-30 05:35:53.906008: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed46732400 next 18446744073709551615 of size 7134208
2020-07-30 05:35:53.906042: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 16777216
2020-07-30 05:35:53.906060: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fed9c000000 next 171 of size 1254400
2020-07-30 05:35:53.906078: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9c132400 next 182 of size 3601920
2020-07-30 05:35:53.906097: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9c4a1a00 next 18446744073709551615 of size 11920896
2020-07-30 05:35:53.906115: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1048576
2020-07-30 05:35:53.906133: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da00000 next 1 of size 256
2020-07-30 05:35:53.906153: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da00100 next 2 of size 256
2020-07-30 05:35:53.906171: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da00200 next 3 of size 256
2020-07-30 05:35:53.906188: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da00300 next 4 of size 256
2020-07-30 05:35:53.906207: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da00400 next 5 of size 1280
2020-07-30 05:35:53.906225: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da00900 next 6 of size 256
2020-07-30 05:35:53.906242: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da00a00 next 7 of size 256
2020-07-30 05:35:53.906260: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da00b00 next 8 of size 256
2020-07-30 05:35:53.906277: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da00c00 next 9 of size 256
2020-07-30 05:35:53.906296: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da00d00 next 10 of size 1280
2020-07-30 05:35:53.906313: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01200 next 11 of size 256
2020-07-30 05:35:53.906331: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01300 next 12 of size 256
2020-07-30 05:35:53.906348: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01400 next 13 of size 256
2020-07-30 05:35:53.906366: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01500 next 14 of size 256
2020-07-30 05:35:53.906382: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01600 next 15 of size 256
2020-07-30 05:35:53.906399: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01700 next 16 of size 256
2020-07-30 05:35:53.906416: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01800 next 17 of size 256
2020-07-30 05:35:53.906434: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01900 next 18 of size 256
2020-07-30 05:35:53.906451: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01a00 next 19 of size 256
2020-07-30 05:35:53.906468: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01b00 next 20 of size 256
2020-07-30 05:35:53.906486: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01c00 next 21 of size 256
2020-07-30 05:35:53.906504: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01d00 next 22 of size 512
2020-07-30 05:35:53.906521: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da01f00 next 23 of size 256
2020-07-30 05:35:53.906539: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da02000 next 24 of size 256
2020-07-30 05:35:53.906556: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da02100 next 25 of size 256
2020-07-30 05:35:53.906574: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da02200 next 26 of size 256
2020-07-30 05:35:53.906591: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da02300 next 27 of size 1280
2020-07-30 05:35:53.906609: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da02800 next 28 of size 256
2020-07-30 05:35:53.906626: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da02900 next 29 of size 256
2020-07-30 05:35:53.906643: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da02a00 next 30 of size 256
2020-07-30 05:35:53.906661: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da02b00 next 31 of size 256
2020-07-30 05:35:53.906678: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da02c00 next 32 of size 256
2020-07-30 05:35:53.906696: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da02d00 next 33 of size 1280
2020-07-30 05:35:53.906713: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da03200 next 34 of size 256
2020-07-30 05:35:53.906730: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da03300 next 35 of size 256
2020-07-30 05:35:53.906747: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da03400 next 36 of size 256
2020-07-30 05:35:53.906765: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da03500 next 37 of size 1280
2020-07-30 05:35:53.906782: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da03a00 next 38 of size 256
2020-07-30 05:35:53.906800: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da03b00 next 39 of size 256
2020-07-30 05:35:53.906817: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da03c00 next 40 of size 256
2020-07-30 05:35:53.906834: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da03d00 next 41 of size 256
2020-07-30 05:35:53.906852: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da03e00 next 42 of size 256
2020-07-30 05:35:53.906869: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da03f00 next 43 of size 256
2020-07-30 05:35:53.906887: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da04000 next 44 of size 1280
2020-07-30 05:35:53.906904: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da04500 next 45 of size 256
2020-07-30 05:35:53.906921: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da04600 next 46 of size 256
2020-07-30 05:35:53.906950: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da04700 next 47 of size 256
2020-07-30 05:35:53.906970: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da04800 next 48 of size 256
2020-07-30 05:35:53.906985: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da04900 next 49 of size 1280
2020-07-30 05:35:53.907000: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da04e00 next 50 of size 256
2020-07-30 05:35:53.907016: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da04f00 next 51 of size 1280
2020-07-30 05:35:53.907033: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da05400 next 52 of size 256
2020-07-30 05:35:53.907050: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da05500 next 53 of size 256
2020-07-30 05:35:53.907078: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da05600 next 54 of size 256
2020-07-30 05:35:53.907095: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da05700 next 55 of size 1280
2020-07-30 05:35:53.907113: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da05c00 next 56 of size 256
2020-07-30 05:35:53.907129: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da05d00 next 57 of size 256
2020-07-30 05:35:53.907175: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da05e00 next 58 of size 256
2020-07-30 05:35:53.907195: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da05f00 next 59 of size 256
2020-07-30 05:35:53.907215: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da06000 next 60 of size 1280
2020-07-30 05:35:53.907232: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da06500 next 61 of size 1280
2020-07-30 05:35:53.907250: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da06a00 next 62 of size 256
2020-07-30 05:35:53.907267: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da06b00 next 63 of size 256
2020-07-30 05:35:53.907286: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da06c00 next 64 of size 1280
2020-07-30 05:35:53.907303: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07100 next 65 of size 256
2020-07-30 05:35:53.907321: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07200 next 66 of size 1280
2020-07-30 05:35:53.907338: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07700 next 67 of size 256
2020-07-30 05:35:53.907356: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07800 next 68 of size 256
2020-07-30 05:35:53.907373: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07900 next 69 of size 256
2020-07-30 05:35:53.907391: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07a00 next 70 of size 256
2020-07-30 05:35:53.907408: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07b00 next 71 of size 256
2020-07-30 05:35:53.907426: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07c00 next 72 of size 256
2020-07-30 05:35:53.907444: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07d00 next 73 of size 256
2020-07-30 05:35:53.907461: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07e00 next 74 of size 256
2020-07-30 05:35:53.907479: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da07f00 next 75 of size 256
2020-07-30 05:35:53.907496: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da08000 next 76 of size 256
2020-07-30 05:35:53.907513: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da08100 next 77 of size 256
2020-07-30 05:35:53.907531: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da08200 next 78 of size 1280
2020-07-30 05:35:53.907548: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da08700 next 79 of size 256
2020-07-30 05:35:53.907566: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da08800 next 80 of size 256
2020-07-30 05:35:53.907584: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da08900 next 81 of size 256
2020-07-30 05:35:53.907601: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da08a00 next 82 of size 256
2020-07-30 05:35:53.907619: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da08b00 next 83 of size 1280
2020-07-30 05:35:53.907636: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da09000 next 84 of size 256
2020-07-30 05:35:53.907654: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da09100 next 85 of size 256
2020-07-30 05:35:53.907671: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da09200 next 86 of size 256
2020-07-30 05:35:53.907689: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da09300 next 87 of size 256
2020-07-30 05:35:53.907707: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da09400 next 88 of size 1280
2020-07-30 05:35:53.907726: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da09900 next 89 of size 76800
2020-07-30 05:35:53.907744: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da1c500 next 90 of size 1792
2020-07-30 05:35:53.907763: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da1cc00 next 91 of size 1280
2020-07-30 05:35:53.907781: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da1d100 next 92 of size 1024
2020-07-30 05:35:53.907799: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da1d500 next 93 of size 360192
2020-07-30 05:35:53.907819: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da75400 next 94 of size 153600
2020-07-30 05:35:53.907836: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9da9ac00 next 95 of size 76800
2020-07-30 05:35:53.907854: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9daad800 next 96 of size 1280
2020-07-30 05:35:53.907873: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9daadd00 next 97 of size 8192
2020-07-30 05:35:53.907890: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9daafd00 next 98 of size 256
2020-07-30 05:35:53.907908: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9daafe00 next 99 of size 32768
2020-07-30 05:35:53.907927: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dab7e00 next 100 of size 512
2020-07-30 05:35:53.907944: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dab8000 next 101 of size 512
2020-07-30 05:35:53.907962: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dab8200 next 102 of size 16384
2020-07-30 05:35:53.907980: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dabc200 next 103 of size 256
2020-07-30 05:35:53.907998: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dabc300 next 104 of size 16384
2020-07-30 05:35:53.908016: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac0300 next 107 of size 256
2020-07-30 05:35:53.908034: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac0400 next 108 of size 1280
2020-07-30 05:35:53.908052: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac0900 next 109 of size 256
2020-07-30 05:35:53.908069: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac0a00 next 110 of size 1792
2020-07-30 05:35:53.908087: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac1100 next 111 of size 768
2020-07-30 05:35:53.908105: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac1400 next 112 of size 256
2020-07-30 05:35:53.908123: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac1500 next 113 of size 1792
2020-07-30 05:35:53.908140: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac1c00 next 114 of size 1280
2020-07-30 05:35:53.908158: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac2100 next 116 of size 16384
2020-07-30 05:35:53.908175: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac6100 next 117 of size 1280
2020-07-30 05:35:53.908193: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac6600 next 118 of size 256
2020-07-30 05:35:53.908210: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac6700 next 119 of size 1792
2020-07-30 05:35:53.908228: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac6e00 next 120 of size 256
2020-07-30 05:35:53.912712: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac6f00 next 121 of size 1280
2020-07-30 05:35:53.912733: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac7400 next 123 of size 1280
2020-07-30 05:35:53.912740: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac7900 next 124 of size 256
2020-07-30 05:35:53.912746: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dac7a00 next 125 of size 76800
2020-07-30 05:35:53.912751: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dada600 next 126 of size 256
2020-07-30 05:35:53.912756: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dada700 next 127 of size 1280
2020-07-30 05:35:53.912762: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9dadac00 next 18446744073709551615 of size 152576
2020-07-30 05:35:53.912768: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 2097152
2020-07-30 05:35:53.912773: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9de00000 next 106 of size 360192
2020-07-30 05:35:53.912779: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9de57f00 next 115 of size 363776
2020-07-30 05:35:53.912784: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9deb0c00 next 122 of size 360192
2020-07-30 05:35:53.912791: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9df08b00 next 128 of size 76800
2020-07-30 05:35:53.912797: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9df1b700 next 129 of size 360192
2020-07-30 05:35:53.912803: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9df73600 next 18446744073709551615 of size 576000
2020-07-30 05:35:53.912809: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 4194304
2020-07-30 05:35:53.912814: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fc00000 next 131 of size 76800
2020-07-30 05:35:53.912820: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fc12c00 next 132 of size 1280
2020-07-30 05:35:53.912825: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fc13100 next 133 of size 360192
2020-07-30 05:35:53.912831: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fc6b000 next 134 of size 1280
2020-07-30 05:35:53.912836: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fc6b500 next 135 of size 256
2020-07-30 05:35:53.912842: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fc6b600 next 136 of size 76800
2020-07-30 05:35:53.912847: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fc7e200 next 137 of size 360192
2020-07-30 05:35:53.912852: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fcd6100 next 138 of size 1280
2020-07-30 05:35:53.912856: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fcd6600 next 139 of size 256
2020-07-30 05:35:53.912873: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fcd6700 next 140 of size 76800
2020-07-30 05:35:53.912877: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fce9300 next 141 of size 256
2020-07-30 05:35:53.912900: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fce9400 next 142 of size 256
2020-07-30 05:35:53.912906: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fce9500 next 143 of size 256
2020-07-30 05:35:53.913188: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fce9600 next 144 of size 256
2020-07-30 05:35:53.913196: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fce9700 next 145 of size 768
2020-07-30 05:35:53.913201: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fce9a00 next 146 of size 256
2020-07-30 05:35:53.913205: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fce9b00 next 147 of size 363776
2020-07-30 05:35:53.913210: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fd42800 next 148 of size 256
2020-07-30 05:35:53.913214: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fd42900 next 149 of size 1280
2020-07-30 05:35:53.913218: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fd42e00 next 150 of size 256
2020-07-30 05:35:53.913223: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fd42f00 next 151 of size 768
2020-07-30 05:35:53.913227: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fd43200 next 152 of size 256
2020-07-30 05:35:53.913231: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fd43300 next 153 of size 360192
2020-07-30 05:35:53.913236: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fd9b200 next 154 of size 360192
2020-07-30 05:35:53.913240: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fdf3100 next 155 of size 16384
2020-07-30 05:35:53.913244: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fdf7100 next 156 of size 360192
2020-07-30 05:35:53.913249: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fe4f000 next 157 of size 1280
2020-07-30 05:35:53.913253: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fe4f500 next 158 of size 1280
2020-07-30 05:35:53.913258: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fe4fa00 next 159 of size 256
2020-07-30 05:35:53.913262: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fe4fb00 next 160 of size 360192
2020-07-30 05:35:53.913266: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9fea7a00 next 161 of size 363776
2020-07-30 05:35:53.913271: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9ff00700 next 162 of size 256
2020-07-30 05:35:53.913275: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9ff00800 next 163 of size 256
2020-07-30 05:35:53.913280: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9ff00900 next 164 of size 1280
2020-07-30 05:35:53.913284: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fed9ff00e00 next 183 of size 349952
2020-07-30 05:35:53.913289: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fed9ff56500 next 170 of size 61952
2020-07-30 05:35:53.913293: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fed9ff65700 next 18446744073709551615 of size 633088
2020-07-30 05:35:53.913298: I tensorflow/core/common_runtime/bfc_allocator.cc:914]      Summary of in-use Chunks by size: 
2020-07-30 05:35:53.913308: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 93 Chunks of size 256 totalling 23.2KiB
2020-07-30 05:35:53.913314: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 512 totalling 1.5KiB
2020-07-30 05:35:53.914050: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 768 totalling 2.2KiB
2020-07-30 05:35:53.914069: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1024 totalling 1.0KiB
2020-07-30 05:35:53.914089: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 31 Chunks of size 1280 totalling 38.8KiB
2020-07-30 05:35:53.914094: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 1792 totalling 7.0KiB
2020-07-30 05:35:53.914117: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 8192 totalling 8.0KiB
2020-07-30 05:35:53.914123: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 16384 totalling 64.0KiB
2020-07-30 05:35:53.914129: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 32768 totalling 32.0KiB
2020-07-30 05:35:53.914134: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 61952 totalling 60.5KiB
2020-07-30 05:35:53.914140: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 7 Chunks of size 76800 totalling 525.0KiB
2020-07-30 05:35:53.914147: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 152576 totalling 149.0KiB
2020-07-30 05:35:53.914152: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 153600 totalling 150.0KiB
2020-07-30 05:35:53.914159: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 10 Chunks of size 360192 totalling 3.43MiB
2020-07-30 05:35:53.914165: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 363776 totalling 1.04MiB
2020-07-30 05:35:53.914170: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 576000 totalling 562.5KiB
2020-07-30 05:35:53.914175: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 3601920 totalling 3.43MiB
2020-07-30 05:35:53.914180: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 7134208 totalling 6.80MiB
2020-07-30 05:35:53.914185: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 10805248 totalling 20.61MiB
2020-07-30 05:35:53.914203: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 11920896 totalling 11.37MiB
2020-07-30 05:35:53.914207: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1080523264 totalling 1.01GiB
2020-07-30 05:35:53.914212: I tensorflow/core/common_runtime/bfc_allocator.cc:921] Sum Total of in-use chunks: 1.05GiB
2020-07-30 05:35:53.914216: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_: 3323265024 memory_limit_: 3323265024 available bytes: 0 curr_region_allocation_bytes_: 8589934592
2020-07-30 05:35:53.914224: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats: 
Limit:                  3323265024
InUse:                  1131147776
MaxInUse:               2198831872
NumAllocs:                   26656
MaxAllocSize:           1083862528

2020-07-30 05:35:53.914241: W tensorflow/core/common_runtime/bfc_allocator.cc:424] *********************************________________________________________________________________*_*
2020-07-30 05:35:53.916244: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at concat_op.cc:153 : Resource exhausted: OOM when allocating tensor with shape[900436,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
 85%|█████████████████████████████████████████████████████████████████████████████████████████████▌                | 377/443 [07:53<01:22,  1.26s/it]
Traceback (most recent call last):
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[900436,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node layer2/concat}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[ArgMax/_187]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[900436,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node layer2/concat}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 260, in <module>
    results = sess.run(fetches, feed_dict=feed_dict)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[900436,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node layer2/concat (defined at /home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[ArgMax/_187]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[900436,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node layer2/concat (defined at /home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

Original stack trace for 'layer2/concat':
  File "run.py", line 140, in <module>
    t_is_training)
  File "/home/sarim/PycharmProjects/trajectory_prediction/Point-GNN/models/models.py", line 146, in predict
    **layer_kwargs)
  File "/home/sarim/PycharmProjects/trajectory_prediction/Point-GNN/models/gnn.py", line 352, in apply_regular
    axis=-1)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 1420, in concat
    return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 1257, in concat_v2
    "ConcatV2", values=values, axis=axis, name=name)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/home/sarim/PycharmProjects/trajectory_prediction/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()
sumeshthakr commented 3 years ago

You can change the batch size in *_train_train_config file