search

thtrieu / darkflow

Translate darknet to tensorflow. Load trained weights, retrain/fine-tune using tensorflow, export constant graph def to mobile devices
GNU General Public License v3.0
6.13k stars 2.08k forks source link

I a getting this , can someone please help me. #564

Open shashi29 opened 6 years ago

shashi29 commented 6 years ago

Please help me in calculating the memory requirement .

2018-02-07 17:32:27.393509: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 84.50MiB. Current allocation summary follows. 2018-02-07 17:32:27.393621: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (256): Total Chunks: 94, Chunks in use: 94. 23.5KiB allocated for chunks. 23.5KiB in use in bin. 9.0KiB client-requested in use in bin. 2018-02-07 17:32:27.393663: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (512): Total Chunks: 19, Chunks in use: 18. 9.5KiB allocated for chunks. 9.0KiB in use in bin. 9.0KiB client-requested in use in bin. 2018-02-07 17:32:27.393699: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1024): Total Chunks: 25, Chunks in use: 25. 29.8KiB allocated for chunks. 29.8KiB in use in bin. 29.1KiB client-requested in use in bin. 2018-02-07 17:32:27.393730: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2048): Total Chunks: 18, Chunks in use: 18. 36.0KiB allocated for chunks. 36.0KiB in use in bin. 36.0KiB client-requested in use in bin. 2018-02-07 17:32:27.393761: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4096): Total Chunks: 35, Chunks in use: 35. 140.0KiB allocated for chunks. 140.0KiB in use in bin. 140.0KiB client-requested in use in bin. 2018-02-07 17:32:27.393789: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-02-07 17:32:27.393820: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16384): Total Chunks: 7, Chunks in use: 7. 126.0KiB allocated for chunks. 126.0KiB in use in bin. 126.0KiB client-requested in use in bin. 2018-02-07 17:32:27.393851: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (32768): Total Chunks: 2, Chunks in use: 2. 106.0KiB allocated for chunks. 106.0KiB in use in bin. 105.6KiB client-requested in use in bin. 2018-02-07 17:32:27.393882: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (65536): Total Chunks: 9, Chunks in use: 9. 749.2KiB allocated for chunks. 749.2KiB in use in bin. 748.9KiB client-requested in use in bin. 2018-02-07 17:32:27.393911: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (131072): Total Chunks: 7, Chunks in use: 7. 1.03MiB allocated for chunks. 1.03MiB in use in bin. 1.03MiB client-requested in use in bin. 2018-02-07 17:32:27.393940: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (262144): Total Chunks: 6, Chunks in use: 6. 1.69MiB allocated for chunks. 1.69MiB in use in bin. 1.69MiB client-requested in use in bin. 2018-02-07 17:32:27.393996: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-02-07 17:32:27.394031: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1048576): Total Chunks: 6, Chunks in use: 6. 6.75MiB allocated for chunks. 6.75MiB in use in bin. 6.75MiB client-requested in use in bin. 2018-02-07 17:32:27.394059: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-02-07 17:32:27.394089: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4194304): Total Chunks: 6, Chunks in use: 6. 27.00MiB allocated for chunks. 27.00MiB in use in bin. 27.00MiB client-requested in use in bin. 2018-02-07 17:32:27.394117: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-02-07 17:32:27.394149: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16777216): Total Chunks: 8, Chunks in use: 7. 171.66MiB allocated for chunks. 139.99MiB in use in bin. 139.99MiB client-requested in use in bin. 2018-02-07 17:32:27.394182: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (33554432): Total Chunks: 10, Chunks in use: 10. 386.63MiB allocated for chunks. 386.63MiB in use in bin. 386.63MiB client-requested in use in bin. 2018-02-07 17:32:27.394213: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (67108864): Total Chunks: 2, Chunks in use: 1. 161.40MiB allocated for chunks. 84.50MiB in use in bin. 84.50MiB client-requested in use in bin. 2018-02-07 17:32:27.394245: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (134217728): Total Chunks: 5, Chunks in use: 5. 845.00MiB allocated for chunks. 845.00MiB in use in bin. 845.00MiB client-requested in use in bin. 2018-02-07 17:32:27.394272: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-02-07 17:32:27.394300: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin for 84.50MiB was 64.00MiB, Chunk State: 2018-02-07 17:32:27.394338: I tensorflow/core/common_runtime/bfc_allocator.cc:649] Size: 76.90MiB | Requested Size: 0B | in_use: 0, prev: Size: 43.07MiB | Requested Size: 43.07MiB | in_use: 1 2018-02-07 17:32:27.394365: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc0000 of size 1280 2018-02-07 17:32:27.394388: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc0500 of size 256 2018-02-07 17:32:27.394409: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc0600 of size 1792 2018-02-07 17:32:27.394430: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc0d00 of size 256 2018-02-07 17:32:27.394450: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc0e00 of size 256 2018-02-07 17:32:27.394470: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc0f00 of size 256 2018-02-07 17:32:27.394490: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc1000 of size 256 2018-02-07 17:32:27.394511: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc1100 of size 18432 2018-02-07 17:32:27.394531: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc5900 of size 256 2018-02-07 17:32:27.394551: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc5a00 of size 256 2018-02-07 17:32:27.394571: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc5b00 of size 256 2018-02-07 17:32:27.394596: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc5c00 of size 256 2018-02-07 17:32:27.394633: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fc5d00 of size 73728 2018-02-07 17:32:27.394666: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fd7d00 of size 256 2018-02-07 17:32:27.394687: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fd7e00 of size 256 2018-02-07 17:32:27.394707: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fd7f00 of size 256 2018-02-07 17:32:27.394739: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fd8000 of size 512 2018-02-07 17:32:27.394778: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x501fd8200 of size 294912 2018-02-07 17:32:27.394810: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x502020200 of size 512 2018-02-07 17:32:27.394831: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x502020400 of size 512 2018-02-07 17:32:27.394851: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x502020600 of size 512 2018-02-07 17:32:27.394872: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x502020800 of size 1024 2018-02-07 17:32:27.394893: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x502020c00 of size 1179648 2018-02-07 17:32:27.394913: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x502140c00 of size 1024 2018-02-07 17:32:27.394932: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x502141000 of size 1024 2018-02-07 17:32:27.394957: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x502141400 of size 1024 2018-02-07 17:32:27.394995: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x502141800 of size 2048 2018-02-07 17:32:27.395032: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x502142000 of size 4718592 2018-02-07 17:32:27.395070: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5025c2000 of size 2048 2018-02-07 17:32:27.395108: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5025c2800 of size 2048 2018-02-07 17:32:27.395144: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5025c3000 of size 2048 2018-02-07 17:32:27.395182: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5025c3800 of size 4096 2018-02-07 17:32:27.395224: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5025c4800 of size 18874368 2018-02-07 17:32:27.395263: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5037c4800 of size 4096 2018-02-07 17:32:27.395302: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5037c5800 of size 4096 2018-02-07 17:32:27.395340: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5037c6800 of size 4096 2018-02-07 17:32:27.395373: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5037c7800 of size 4096 2018-02-07 17:32:27.395410: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5037c8800 of size 37748736 2018-02-07 17:32:27.395448: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bc8800 of size 4096 2018-02-07 17:32:27.395486: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bc9800 of size 4096 2018-02-07 17:32:27.395524: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bca800 of size 4096 2018-02-07 17:32:27.395562: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bcb800 of size 256 2018-02-07 17:32:27.395603: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bcb900 of size 143360 2018-02-07 17:32:27.395642: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bee900 of size 256 2018-02-07 17:32:27.395680: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505beea00 of size 256 2018-02-07 17:32:27.395717: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505beeb00 of size 1792 2018-02-07 17:32:27.395754: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bef200 of size 1792 2018-02-07 17:32:27.395829: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bef900 of size 256 2018-02-07 17:32:27.395890: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505befa00 of size 256 2018-02-07 17:32:27.395950: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505befb00 of size 256 2018-02-07 17:32:27.396010: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505befc00 of size 256 2018-02-07 17:32:27.396048: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505befd00 of size 18432 2018-02-07 17:32:27.396071: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bf4500 of size 18432 2018-02-07 17:32:27.396091: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bf8d00 of size 256 2018-02-07 17:32:27.396112: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bf8e00 of size 256 2018-02-07 17:32:27.396132: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bf8f00 of size 256 2018-02-07 17:32:27.396152: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bf9000 of size 256 2018-02-07 17:32:27.396172: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505bf9100 of size 73728 2018-02-07 17:32:27.396192: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505c0b100 of size 73728 2018-02-07 17:32:27.396212: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505c1d100 of size 256 2018-02-07 17:32:27.396231: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505c1d200 of size 256 2018-02-07 17:32:27.396250: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505c1d300 of size 512 2018-02-07 17:32:27.396270: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505c1d500 of size 512 2018-02-07 17:32:27.396290: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505c1d700 of size 294912 2018-02-07 17:32:27.396311: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505c65700 of size 294912 2018-02-07 17:32:27.396330: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505cad700 of size 512 2018-02-07 17:32:27.396349: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505cad900 of size 512 2018-02-07 17:32:27.396369: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505cadb00 of size 1024 2018-02-07 17:32:27.396388: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505cadf00 of size 1024 2018-02-07 17:32:27.396408: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505cae300 of size 1179648 2018-02-07 17:32:27.396428: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505dce300 of size 1179648 2018-02-07 17:32:27.396448: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505eee300 of size 1024 2018-02-07 17:32:27.396467: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505eee700 of size 1024 2018-02-07 17:32:27.396486: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505eeeb00 of size 2048 2018-02-07 17:32:27.396506: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505eef300 of size 2048 2018-02-07 17:32:27.396526: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x505eefb00 of size 4718592 2018-02-07 17:32:27.396546: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50636fb00 of size 4718592 2018-02-07 17:32:27.396566: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5067efb00 of size 2048 2018-02-07 17:32:27.396585: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5067f0300 of size 2048 2018-02-07 17:32:27.396606: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5067f0b00 of size 4096 2018-02-07 17:32:27.396625: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5067f1b00 of size 4096 2018-02-07 17:32:27.396645: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5067f2b00 of size 18874368 2018-02-07 17:32:27.396665: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5079f2b00 of size 18874368 2018-02-07 17:32:27.396684: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x508bf2b00 of size 4096 2018-02-07 17:32:27.396704: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x508bf3b00 of size 4096 2018-02-07 17:32:27.396723: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x508bf4b00 of size 4096 2018-02-07 17:32:27.396742: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x508bf5b00 of size 4096 2018-02-07 17:32:27.396766: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x508bf6b00 of size 37748736 2018-02-07 17:32:27.396799: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50aff6b00 of size 37748736 2018-02-07 17:32:27.396830: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d3f6b00 of size 4096 2018-02-07 17:32:27.396850: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d3f7b00 of size 4096 2018-02-07 17:32:27.396871: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d3f8b00 of size 256 2018-02-07 17:32:27.396891: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d3f8c00 of size 256 2018-02-07 17:32:27.396912: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d3f8d00 of size 143360 2018-02-07 17:32:27.396931: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d41bd00 of size 143360 2018-02-07 17:32:27.396951: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d43ed00 of size 256 2018-02-07 17:32:27.396970: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d43ee00 of size 1792 2018-02-07 17:32:27.396990: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d43f500 of size 256 2018-02-07 17:32:27.397009: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d43f600 of size 256 2018-02-07 17:32:27.397028: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d43f700 of size 256 2018-02-07 17:32:27.397048: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d43f800 of size 256 2018-02-07 17:32:27.397068: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d43f900 of size 18432 2018-02-07 17:32:27.397087: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d444100 of size 256 2018-02-07 17:32:27.397106: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d444200 of size 256 2018-02-07 17:32:27.397126: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d444300 of size 256 2018-02-07 17:32:27.397145: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d444400 of size 256 2018-02-07 17:32:27.397164: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d444500 of size 73728 2018-02-07 17:32:27.397184: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d456500 of size 256 2018-02-07 17:32:27.397203: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d456600 of size 256 2018-02-07 17:32:27.397222: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d456700 of size 256 2018-02-07 17:32:27.397241: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d456800 of size 512 2018-02-07 17:32:27.397261: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d456a00 of size 294912 2018-02-07 17:32:27.397281: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d49ea00 of size 512 2018-02-07 17:32:27.397300: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d49ec00 of size 512 2018-02-07 17:32:27.397319: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d49ee00 of size 512 2018-02-07 17:32:27.397339: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d49f000 of size 1024 2018-02-07 17:32:27.397358: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d49f400 of size 1179648 2018-02-07 17:32:27.397378: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d5bf400 of size 1024 2018-02-07 17:32:27.397398: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d5bf800 of size 1024 2018-02-07 17:32:27.397417: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d5bfc00 of size 1024 2018-02-07 17:32:27.397436: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d5c0000 of size 2048 2018-02-07 17:32:27.397456: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50d5c0800 of size 4718592 2018-02-07 17:32:27.397475: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50da40800 of size 2048 2018-02-07 17:32:27.397495: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50da41000 of size 2048 2018-02-07 17:32:27.397514: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50da41800 of size 2048 2018-02-07 17:32:27.397534: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50da42000 of size 4096 2018-02-07 17:32:27.397554: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50da43000 of size 18874368 2018-02-07 17:32:27.397573: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50ec43000 of size 4096 2018-02-07 17:32:27.397593: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50ec44000 of size 4096 2018-02-07 17:32:27.397612: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50ec45000 of size 4096 2018-02-07 17:32:27.397631: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50ec46000 of size 4096 2018-02-07 17:32:27.397651: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x50ec47000 of size 37748736 2018-02-07 17:32:27.397671: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511047000 of size 4096 2018-02-07 17:32:27.397690: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511048000 of size 4096 2018-02-07 17:32:27.397709: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511049000 of size 4096 2018-02-07 17:32:27.397729: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51104a000 of size 256 2018-02-07 17:32:27.397748: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51104a100 of size 143360 2018-02-07 17:32:27.397767: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51106d100 of size 256 2018-02-07 17:32:27.397787: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51106d200 of size 256 2018-02-07 17:32:27.397807: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51106d300 of size 1792 2018-02-07 17:32:27.397826: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51106da00 of size 1792 2018-02-07 17:32:27.397849: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51106e100 of size 256 2018-02-07 17:32:27.397881: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51106e200 of size 256 2018-02-07 17:32:27.397908: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51106e300 of size 256 2018-02-07 17:32:27.397928: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51106e400 of size 256 2018-02-07 17:32:27.397949: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51106e500 of size 18432 2018-02-07 17:32:27.397992: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511072d00 of size 18432 2018-02-07 17:32:27.398002: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511077500 of size 256 2018-02-07 17:32:27.398011: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511077600 of size 256 2018-02-07 17:32:27.398019: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511077700 of size 256 2018-02-07 17:32:27.398028: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511077800 of size 256 2018-02-07 17:32:27.398037: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511077900 of size 73728 2018-02-07 17:32:27.398045: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511089900 of size 73728 2018-02-07 17:32:27.398054: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51109b900 of size 256 2018-02-07 17:32:27.398063: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51109ba00 of size 256 2018-02-07 17:32:27.398071: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51109bb00 of size 512 2018-02-07 17:32:27.398080: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51109bd00 of size 512 2018-02-07 17:32:27.398089: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51109bf00 of size 294912 2018-02-07 17:32:27.398097: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5110e3f00 of size 294912 2018-02-07 17:32:27.398106: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51112bf00 of size 512 2018-02-07 17:32:27.398114: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51112c100 of size 512 2018-02-07 17:32:27.398123: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51112c300 of size 1024 2018-02-07 17:32:27.398132: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51112c700 of size 1024 2018-02-07 17:32:27.398141: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51112cb00 of size 1179648 2018-02-07 17:32:27.398149: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51124cb00 of size 1179648 2018-02-07 17:32:27.398158: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51136cb00 of size 1024 2018-02-07 17:32:27.398167: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51136cf00 of size 1024 2018-02-07 17:32:27.398175: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51136d300 of size 2048 2018-02-07 17:32:27.398184: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51136db00 of size 2048 2018-02-07 17:32:27.398192: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51136e300 of size 4718592 2018-02-07 17:32:27.398201: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5117ee300 of size 4718592 2018-02-07 17:32:27.398210: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511c6e300 of size 2048 2018-02-07 17:32:27.398218: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511c6eb00 of size 2048 2018-02-07 17:32:27.398227: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511c6f300 of size 4096 2018-02-07 17:32:27.398235: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511c70300 of size 4096 2018-02-07 17:32:27.398244: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x511c71300 of size 18874368 2018-02-07 17:32:27.398253: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x512e71300 of size 18874368 2018-02-07 17:32:27.398261: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x514071300 of size 4096 2018-02-07 17:32:27.398270: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x514072300 of size 4096 2018-02-07 17:32:27.398279: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x514073300 of size 4096 2018-02-07 17:32:27.398287: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x514074300 of size 4096 2018-02-07 17:32:27.398296: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x514075300 of size 37748736 2018-02-07 17:32:27.398305: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x516475300 of size 37748736 2018-02-07 17:32:27.398313: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518875300 of size 4096 2018-02-07 17:32:27.398322: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518876300 of size 4096 2018-02-07 17:32:27.398331: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518877300 of size 256 2018-02-07 17:32:27.398339: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518877400 of size 256 2018-02-07 17:32:27.398348: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518877500 of size 143360 2018-02-07 17:32:27.398357: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51889a500 of size 143360 2018-02-07 17:32:27.398365: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bd500 of size 256 2018-02-07 17:32:27.398374: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bd600 of size 256 2018-02-07 17:32:27.398383: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bd700 of size 256 2018-02-07 17:32:27.398392: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bd800 of size 512 2018-02-07 17:32:27.398400: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bda00 of size 256 2018-02-07 17:32:27.398409: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bdb00 of size 256 2018-02-07 17:32:27.398418: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bdc00 of size 256 2018-02-07 17:32:27.398426: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bdd00 of size 256 2018-02-07 17:32:27.398435: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bde00 of size 256 2018-02-07 17:32:27.398443: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bdf00 of size 256 2018-02-07 17:32:27.398452: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188be000 of size 256 2018-02-07 17:32:27.398460: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188be100 of size 1024 2018-02-07 17:32:27.398469: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188be500 of size 256 2018-02-07 17:32:27.398477: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188be600 of size 256 2018-02-07 17:32:27.398486: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188be700 of size 256 2018-02-07 17:32:27.398494: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188be800 of size 256 2018-02-07 17:32:27.398503: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188be900 of size 2048 2018-02-07 17:32:27.398512: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bf100 of size 256 2018-02-07 17:32:27.398521: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bf200 of size 256 2018-02-07 17:32:27.398529: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188bf300 of size 4096 2018-02-07 17:32:27.398538: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0300 of size 256 2018-02-07 17:32:27.398547: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0400 of size 256 2018-02-07 17:32:27.398555: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0500 of size 256 2018-02-07 17:32:27.398564: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0600 of size 256 2018-02-07 17:32:27.398572: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0700 of size 256 2018-02-07 17:32:27.398581: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0800 of size 256 2018-02-07 17:32:27.398589: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0900 of size 256 2018-02-07 17:32:27.398598: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0a00 of size 256 2018-02-07 17:32:27.398606: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0b00 of size 256 2018-02-07 17:32:27.398615: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0c00 of size 256 2018-02-07 17:32:27.398624: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0d00 of size 256 2018-02-07 17:32:27.398632: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0e00 of size 256 2018-02-07 17:32:27.398641: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c0f00 of size 256 2018-02-07 17:32:27.398649: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c1000 of size 256 2018-02-07 17:32:27.398658: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c1100 of size 256 2018-02-07 17:32:27.398666: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c1200 of size 256 2018-02-07 17:32:27.398675: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c1300 of size 256 2018-02-07 17:32:27.398684: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c1400 of size 256 2018-02-07 17:32:27.398693: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188c1500 of size 108288 2018-02-07 17:32:27.398703: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5188dbc00 of size 216320 2018-02-07 17:32:27.398712: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518910900 of size 54272 2018-02-07 17:32:27.398721: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51891dd00 of size 108288 2018-02-07 17:32:27.398729: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518938400 of size 256 2018-02-07 17:32:27.398738: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518938500 of size 108288 2018-02-07 17:32:27.398747: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518952c00 of size 54272 2018-02-07 17:32:27.398756: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518960200 of size 256 2018-02-07 17:32:27.398764: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518960300 of size 256 2018-02-07 17:32:27.398773: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x518960400 of size 18432 2018-02-07 17:32:27.398782: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51a910000 of size 2048 2018-02-07 17:32:27.398790: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51a910800 of size 256 2018-02-07 17:32:27.398799: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51a910900 of size 512 2018-02-07 17:32:27.398808: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51a910b00 of size 4096 2018-02-07 17:32:27.398816: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51a911b00 of size 256 2018-02-07 17:32:27.398825: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51a911c00 of size 256 2018-02-07 17:32:27.398834: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51a911d00 of size 1024 2018-02-07 17:32:27.398842: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51a912100 of size 4096 2018-02-07 17:32:27.398851: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51a913100 of size 33547008 2018-02-07 17:32:27.398861: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x51c911400 of size 177209344 2018-02-07 17:32:27.398870: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x527211400 of size 177209344 2018-02-07 17:32:27.398879: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x531b11400 of size 177209344 2018-02-07 17:32:27.398888: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x53c411400 of size 177209344 2018-02-07 17:32:27.398896: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x546d11400 of size 177209344 2018-02-07 17:32:27.398906: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x551611400 of size 44302336 2018-02-07 17:32:27.398915: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x554051400 of size 44302336 2018-02-07 17:32:27.398928: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x556a91400 of size 45158400 2018-02-07 17:32:27.398944: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x5595a2400 of size 88604672 2018-02-07 17:32:27.398955: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x55ea22400 of size 45158400 2018-02-07 17:32:27.398966: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x518960000 of size 512 2018-02-07 17:32:27.398976: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x518964c00 of size 33207296 2018-02-07 17:32:27.398985: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x561533400 of size 80635136 2018-02-07 17:32:27.398993: I tensorflow/core/common_runtime/bfc_allocator.cc:676] Summary of in-use Chunks by size: 2018-02-07 17:32:27.399007: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 94 Chunks of size 256 totalling 23.5KiB 2018-02-07 17:32:27.399018: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 18 Chunks of size 512 totalling 9.0KiB 2018-02-07 17:32:27.399029: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 18 Chunks of size 1024 totalling 18.0KiB 2018-02-07 17:32:27.399039: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1280 totalling 1.2KiB 2018-02-07 17:32:27.399049: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 1792 totalling 10.5KiB 2018-02-07 17:32:27.399060: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 18 Chunks of size 2048 totalling 36.0KiB 2018-02-07 17:32:27.399070: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 35 Chunks of size 4096 totalling 140.0KiB 2018-02-07 17:32:27.399080: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 7 Chunks of size 18432 totalling 126.0KiB 2018-02-07 17:32:27.399090: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 54272 totalling 106.0KiB 2018-02-07 17:32:27.399101: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 73728 totalling 432.0KiB 2018-02-07 17:32:27.399111: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 108288 totalling 317.2KiB 2018-02-07 17:32:27.399121: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 143360 totalling 840.0KiB 2018-02-07 17:32:27.399132: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 216320 totalling 211.2KiB 2018-02-07 17:32:27.399142: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 294912 totalling 1.69MiB 2018-02-07 17:32:27.399152: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 1179648 totalling 6.75MiB 2018-02-07 17:32:27.399162: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 4718592 totalling 27.00MiB 2018-02-07 17:32:27.399172: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 18874368 totalling 108.00MiB 2018-02-07 17:32:27.399183: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 33547008 totalling 31.99MiB 2018-02-07 17:32:27.399193: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 37748736 totalling 216.00MiB 2018-02-07 17:32:27.399204: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 44302336 totalling 84.50MiB 2018-02-07 17:32:27.399214: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 45158400 totalling 86.13MiB 2018-02-07 17:32:27.399224: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 88604672 totalling 84.50MiB 2018-02-07 17:32:27.399234: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 5 Chunks of size 177209344 totalling 845.00MiB 2018-02-07 17:32:27.399244: I tensorflow/core/common_runtime/bfc_allocator.cc:683] Sum Total of in-use chunks: 1.46GiB 2018-02-07 17:32:27.399257: I tensorflow/core/common_runtime/bfc_allocator.cc:685] Stats: Limit: 1680185753 InUse: 1566342656 MaxInUse: 1646960640 NumAllocs: 269 MaxAllocSize: 846242304

2018-02-07 17:32:27.399301: W tensorflow/core/common_runtime/bfcallocator.cc:277] *******__ 2018-02-07 17:32:27.399324: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[16,32,208,208] Traceback (most recent call last): File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call return fn(*args) File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn status, run_metadata) File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit__ c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[16,32,208,208] [[Node: 3-convolutional_2 = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Pad_1, 3-convolutional/kernel/read)]] [[Node: mul_17/_59 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2747_mul_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "flow", line 6, in cliHandler(sys.argv) File "/home/bots/shashi/YOLO/darkflow/cli.py", line 33, in cliHandler print('Enter training ...'); tfnet.train() File "/home/bots/shashi/YOLO/darkflow/net/flow.py", line 56, in train fetched = self.sess.run(fetches, feed_dict) File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run run_metadata_ptr) File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in _run feed_dict_tensor, options, run_metadata) File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run options, run_metadata) File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[16,32,208,208] [[Node: 3-convolutional_2 = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Pad_1, 3-convolutional/kernel/read)]] [[Node: mul_17/_59 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2747_mul_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op '3-convolutional_2', defined at: File "flow", line 6, in cliHandler(sys.argv) File "/home/bots/shashi/YOLO/darkflow/cli.py", line 26, in cliHandler tfnet = TFNet(FLAGS) File "/home/bots/shashi/YOLO/darkflow/net/build.py", line 75, in init self.build_forward() File "/home/bots/shashi/YOLO/darkflow/net/build.py", line 115, in build_forward state = op_create(args) File "/home/bots/shashi/YOLO/darkflow/net/ops/init.py", line 27, in op_create return op_types[layer_type](args) File "/home/bots/shashi/YOLO/darkflow/net/ops/baseop.py", line 42, in init self.forward() File "/home/bots/shashi/YOLO/darkflow/net/ops/convolution.py", line 71, in forward name = self.scope, strides = [1] + [self.lay.stride] * 2 + [1]) File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 631, in conv2d data_format=data_format, name=name) File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op op_def=op_def) File "/home/bots/shashi_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1470, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[16,32,208,208] [[Node: 3-convolutional_2 = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Pad_1, 3-convolutional/kernel/read)]] [[Node: mul_17/_59 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2747_mul_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

WegenPan commented 6 years ago

I meet the same problem with you. Whatever how small I change the batch_size, the problem can't be solved. If you solve this, please tell me how to do.

youyuge34 commented 6 years ago

@WegenPan If the training worked well under CPU mode, maybe your VRAM is too low to run the origin YOLOv2 model. I used the GTX970 which VRAM is 3.5G actually, the tiny-yolo-voc model is trained smoothly. But i can only train the yolo model with --batch 2 and --gpu 0.8. Otherwise the same error above occurs.