During the MultilayerPerceptron.py execution of the code, execution was interrupted. Problem could be buffer related, please try to resolve as soon as possible.
--model_21.csv
2018-08-28 19:44:21.882074: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1484] Adding visible gpu devices: 0
2018-08-28 19:44:21.886790: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-28 19:44:21.893262: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0
2018-08-28 19:44:21.899173: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0: N
2018-08-28 19:44:21.904082: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1409 MB memory) -> physical GPU (device: 0, name: GeForce 940M, pci bus id: 0000:01:00.0, compute capability: 5.0)
Epoch 0 completed out of 25 loss: 81610233872384.0
Epoch 1 completed out of 25 loss: 38165347827712.0
Epoch 2 completed out of 25 loss: 53055122833408.0
Epoch 3 completed out of 25 loss: 33106633424896.0
Epoch 4 completed out of 25 loss: 52647528759296.0
Epoch 5 completed out of 25 loss: 53092565385216.0
Epoch 6 completed out of 25 loss: 71293177167872.0
Epoch 7 completed out of 25 loss: 52137073573888.0
Epoch 8 completed out of 25 loss: 70791311917056.0
Epoch 9 completed out of 25 loss: 55155097600000.0
Epoch 10 completed out of 25 loss: 38832351215616.0
Epoch 11 completed out of 25 loss: 56168932179968.0
Epoch 12 completed out of 25 loss: 76183282647040.0
Epoch 13 completed out of 25 loss: 60526738538496.0
Epoch 14 completed out of 25 loss: 47243486822400.0
Epoch 15 completed out of 25 loss: 36883081986048.0
Epoch 16 completed out of 25 loss: 61538748596224.0
Epoch 17 completed out of 25 loss: 44601817694208.0
Epoch 18 completed out of 25 loss: 53942553673728.0
Epoch 19 completed out of 25 loss: 36598381019136.0
Epoch 20 completed out of 25 loss: 74899079036928.0
Epoch 21 completed out of 25 loss: 43831126917120.0
Epoch 22 completed out of 25 loss: 69689224986624.0
Epoch 23 completed out of 25 loss: 42242572025856.0
Epoch 24 completed out of 25 loss: 57911468359680.0
Accuracy: 0.65
--model_22.csv
2018-08-28 19:44:40.670096: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1484] Adding visible gpu devices: 0
2018-08-28 19:44:40.674933: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-28 19:44:40.678973: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0
2018-08-28 19:44:40.682352: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0: N
2018-08-28 19:44:40.684681: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1409 MB memory) -> physical GPU (device: 0, name: GeForce 940M, pci bus id: 0000:01:00.0, compute capability: 5.0)
2018-08-28 19:44:59.274131: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 18.92MiB. Current allocation summary follows.
2018-08-28 19:44:59.281530: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (256): Total Chunks: 61, Chunks in use: 61. 15.3KiB allocated for chunks. 15.3KiB in use in bin. 348B client-requested in use in bin.
2018-08-28 19:44:59.286182: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.291483: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.3KiB allocated for chunks. 1.3KiB in use in bin. 1.0KiB client-requested in use in bin.
2018-08-28 19:44:59.296321: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2048): Total Chunks: 26, Chunks in use: 26. 52.0KiB allocated for chunks. 52.0KiB in use in bin. 50.8KiB client-requested in use in bin.
2018-08-28 19:44:59.301640: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4096): Total Chunks: 26, Chunks in use: 26. 104.0KiB allocated for chunks. 104.0KiB in use in bin. 101.6KiB client-requested in use in bin.
2018-08-28 19:44:59.306595: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8192): Total Chunks: 53, Chunks in use: 53. 627.3KiB allocated for chunks. 627.3KiB in use in bin. 619.1KiB client-requested in use in bin.
2018-08-28 19:44:59.314851: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.321208: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.328013: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.331836: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.336335: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.344253: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.351021: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1048576): Total Chunks: 26, Chunks in use: 26. 49.59MiB allocated for chunks. 49.59MiB in use in bin. 49.59MiB client-requested in use in bin.
2018-08-28 19:44:59.360959: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.365184: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4194304): Total Chunks: 26, Chunks in use: 26. 198.36MiB allocated for chunks. 198.36MiB in use in bin. 198.36MiB client-requested in use in bin.
2018-08-28 19:44:59.370210: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.377306: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16777216): Total Chunks: 1, Chunks in use: 1. 17.86MiB allocated for chunks. 17.86MiB in use in bin. 13.62MiB client-requested in use in bin.
2018-08-28 19:44:59.381980: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (33554432): Total Chunks: 26, Chunks in use: 26. 1.12GiB allocated for chunks. 1.12GiB in use in bin. 1.12GiB client-requested in use in bin.
2018-08-28 19:44:59.387568: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.398573: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.410114: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-28 19:44:59.416478: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:646] Bin for 18.92MiB was 16.00MiB, Chunk State:
2018-08-28 19:44:59.424906: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000601180000 of size 1280
2018-08-28 19:44:59.428953: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000601180500 of size 46080000
2018-08-28 19:44:59.432808: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000603D72500 of size 8000000
2018-08-28 19:44:59.441259: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000604513700 of size 2000128
2018-08-28 19:44:59.443781: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006046FBC00 of size 8192
2018-08-28 19:44:59.446248: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006046FDC00 of size 16128
2018-08-28 19:44:59.448867: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000604701B00 of size 2048
2018-08-28 19:44:59.454883: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000604702300 of size 4096
2018-08-28 19:44:59.458079: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000604703300 of size 256
2018-08-28 19:44:59.460889: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000604703400 of size 256
2018-08-28 19:44:59.463495: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000604703500 of size 256
2018-08-28 19:44:59.477939: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000604703600 of size 46080000
2018-08-28 19:44:59.481783: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006072F5600 of size 46080000
2018-08-28 19:44:59.487376: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000609EE7600 of size 46080000
2018-08-28 19:44:59.490187: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000060CAD9600 of size 46080000
2018-08-28 19:44:59.493050: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000060F6CB600 of size 46080000
2018-08-28 19:44:59.497338: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006122BD600 of size 46080000
2018-08-28 19:44:59.501057: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000614EAF600 of size 46080000
2018-08-28 19:44:59.507032: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000617AA1600 of size 46080000
2018-08-28 19:44:59.511914: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000061A693600 of size 46080000
2018-08-28 19:44:59.516161: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000061D285600 of size 46080000
2018-08-28 19:44:59.519698: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000061FE77600 of size 46080000
2018-08-28 19:44:59.522671: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000622A69600 of size 46080000
2018-08-28 19:44:59.525766: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000062565B600 of size 46080000
2018-08-28 19:44:59.528407: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000062824D600 of size 46080000
2018-08-28 19:44:59.531405: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000062AE3F600 of size 46080000
2018-08-28 19:44:59.537531: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000062DA31600 of size 46080000
2018-08-28 19:44:59.541629: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000630623600 of size 46080000
2018-08-28 19:44:59.545819: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000633215600 of size 46080000
2018-08-28 19:44:59.551724: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000635E07600 of size 46080000
2018-08-28 19:44:59.556462: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006389F9600 of size 46080000
2018-08-28 19:44:59.560737: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000063B5EB600 of size 46080000
2018-08-28 19:44:59.564775: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000063E1DD600 of size 46080000
2018-08-28 19:44:59.571053: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000640DCF600 of size 46080000
2018-08-28 19:44:59.574430: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006439C1600 of size 46080000
2018-08-28 19:44:59.577171: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006465B3600 of size 46080000
2018-08-28 19:44:59.579965: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006491A5600 of size 8000000
2018-08-28 19:44:59.586850: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000649946800 of size 8000000
2018-08-28 19:44:59.590739: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064A0E7A00 of size 8000000
2018-08-28 19:44:59.593185: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064A888C00 of size 8000000
2018-08-28 19:44:59.595624: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064B029E00 of size 8000000
2018-08-28 19:44:59.599964: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064B7CB000 of size 8000000
2018-08-28 19:44:59.603323: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064BF6C200 of size 8000000
2018-08-28 19:44:59.606759: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064C70D400 of size 8000000
2018-08-28 19:44:59.609644: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064CEAE600 of size 8000000
2018-08-28 19:44:59.613921: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064D64F800 of size 8000000
2018-08-28 19:44:59.616956: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064DDF0A00 of size 8000000
2018-08-28 19:44:59.620136: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064E591C00 of size 8000000
2018-08-28 19:44:59.623734: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064ED32E00 of size 8000000
2018-08-28 19:44:59.626575: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064F4D4000 of size 8000000
2018-08-28 19:44:59.631006: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000064FC75200 of size 8000000
2018-08-28 19:44:59.633901: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000650416400 of size 8000000
2018-08-28 19:44:59.637192: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000650BB7600 of size 8000000
2018-08-28 19:44:59.640740: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000651358800 of size 8000000
2018-08-28 19:44:59.645452: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000651AF9A00 of size 8000000
2018-08-28 19:44:59.648159: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065229AC00 of size 8000000
2018-08-28 19:44:59.651642: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000652A3BE00 of size 8000000
2018-08-28 19:44:59.654714: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006531DD000 of size 8000000
2018-08-28 19:44:59.657878: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065397E200 of size 8000000
2018-08-28 19:44:59.662315: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065411F400 of size 8000000
2018-08-28 19:44:59.665283: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006548C0600 of size 8000000
2018-08-28 19:44:59.668466: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000655061800 of size 2000128
2018-08-28 19:44:59.672083: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000655249D00 of size 2000128
2018-08-28 19:44:59.676686: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000655432200 of size 2000128
2018-08-28 19:44:59.679755: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065561A700 of size 2000128
2018-08-28 19:44:59.682752: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000655802C00 of size 2000128
2018-08-28 19:44:59.686654: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006559EB100 of size 2000128
2018-08-28 19:44:59.689319: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000655BD3600 of size 2000128
2018-08-28 19:44:59.693844: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000655DBBB00 of size 2000128
2018-08-28 19:44:59.696719: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000655FA4000 of size 2000128
2018-08-28 19:44:59.699519: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065618C500 of size 2000128
2018-08-28 19:44:59.702894: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000656374A00 of size 2000128
2018-08-28 19:44:59.707795: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065655CF00 of size 2000128
2018-08-28 19:44:59.710518: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000656745400 of size 2000128
2018-08-28 19:44:59.713454: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065692D900 of size 2000128
2018-08-28 19:44:59.716189: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000656B15E00 of size 2000128
2018-08-28 19:44:59.719816: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000656CFE300 of size 2000128
2018-08-28 19:44:59.726207: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000656EE6800 of size 2000128
2018-08-28 19:44:59.729082: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006570CED00 of size 2000128
2018-08-28 19:44:59.731922: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006572B7200 of size 2000128
2018-08-28 19:44:59.735249: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065749F700 of size 2000128
2018-08-28 19:44:59.742590: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000657687C00 of size 2000128
2018-08-28 19:44:59.745406: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000657870100 of size 2000128
2018-08-28 19:44:59.748317: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000657A58600 of size 2000128
2018-08-28 19:44:59.751681: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000657C40B00 of size 2000128
2018-08-28 19:44:59.759476: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000657E29000 of size 2000128
2018-08-28 19:44:59.763347: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658011500 of size 8192
2018-08-28 19:44:59.770981: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658013500 of size 8192
2018-08-28 19:44:59.774383: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658015500 of size 8192
2018-08-28 19:44:59.776774: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658017500 of size 8192
2018-08-28 19:44:59.779511: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658019500 of size 8192
2018-08-28 19:44:59.782197: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065801B500 of size 8192
2018-08-28 19:44:59.788508: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065801D500 of size 8192
2018-08-28 19:44:59.793055: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065801F500 of size 8192
2018-08-28 19:44:59.796877: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658021500 of size 8192
2018-08-28 19:44:59.803189: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658023500 of size 8192
2018-08-28 19:44:59.807232: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658025500 of size 8192
2018-08-28 19:44:59.811927: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658027500 of size 8192
2018-08-28 19:44:59.818546: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658029500 of size 8192
2018-08-28 19:44:59.821480: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065802B500 of size 8192
2018-08-28 19:44:59.824245: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065802D500 of size 8192
2018-08-28 19:44:59.826712: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065802F500 of size 8192
2018-08-28 19:44:59.829427: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658031500 of size 8192
2018-08-28 19:44:59.834484: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658033500 of size 8192
2018-08-28 19:44:59.837715: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658035500 of size 8192
2018-08-28 19:44:59.841125: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658037500 of size 8192
2018-08-28 19:44:59.843907: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658039500 of size 8192
2018-08-28 19:44:59.848671: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065803B500 of size 8192
2018-08-28 19:44:59.852024: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065803D500 of size 8192
2018-08-28 19:44:59.855481: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065803F500 of size 8192
2018-08-28 19:44:59.858395: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658041500 of size 8192
2018-08-28 19:44:59.861195: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658043500 of size 16128
2018-08-28 19:44:59.865537: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658047400 of size 16128
2018-08-28 19:44:59.868880: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065804B300 of size 16128
2018-08-28 19:44:59.872419: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065804F200 of size 16128
2018-08-28 19:44:59.875201: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658053100 of size 16128
2018-08-28 19:44:59.879607: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658057000 of size 16128
2018-08-28 19:44:59.883012: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065805AF00 of size 16128
2018-08-28 19:44:59.886729: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065805EE00 of size 16128
2018-08-28 19:44:59.889462: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658062D00 of size 16128
2018-08-28 19:44:59.892124: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658066C00 of size 16128
2018-08-28 19:44:59.897298: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065806AB00 of size 16128
2018-08-28 19:44:59.900514: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065806EA00 of size 16128
2018-08-28 19:44:59.907605: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658072900 of size 16128
2018-08-28 19:44:59.910381: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658076800 of size 16128
2018-08-28 19:44:59.913525: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065807A700 of size 16128
2018-08-28 19:44:59.918287: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065807E600 of size 16128
2018-08-28 19:44:59.920924: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658082500 of size 16128
2018-08-28 19:44:59.923583: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658086400 of size 16128
2018-08-28 19:44:59.926613: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065808A300 of size 16128
2018-08-28 19:44:59.929173: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065808E200 of size 16128
2018-08-28 19:44:59.933812: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658092100 of size 16128
2018-08-28 19:44:59.936823: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658096000 of size 16128
2018-08-28 19:44:59.940187: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000658099F00 of size 16128
2018-08-28 19:44:59.942723: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000065809DE00 of size 16128
2018-08-28 19:44:59.945511: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580A1D00 of size 16128
2018-08-28 19:44:59.950300: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580A5C00 of size 2048
2018-08-28 19:44:59.953877: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580A6400 of size 2048
2018-08-28 19:44:59.957993: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580A6C00 of size 2048
2018-08-28 19:44:59.961749: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580A7400 of size 2048
2018-08-28 19:44:59.964237: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580A7C00 of size 2048
2018-08-28 19:44:59.967005: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580A8400 of size 2048
2018-08-28 19:44:59.974096: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580A8C00 of size 2048
2018-08-28 19:44:59.977467: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580A9400 of size 2048
2018-08-28 19:44:59.982210: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580A9C00 of size 2048
2018-08-28 19:44:59.986552: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580AA400 of size 2048
2018-08-28 19:44:59.994525: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580AAC00 of size 2048
2018-08-28 19:44:59.997363: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580AB400 of size 2048
2018-08-28 19:45:00.000748: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580ABC00 of size 2048
2018-08-28 19:45:00.003728: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580AC400 of size 2048
2018-08-28 19:45:00.007328: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580ACC00 of size 2048
2018-08-28 19:45:00.010079: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580AD400 of size 2048
2018-08-28 19:45:00.012976: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580ADC00 of size 2048
2018-08-28 19:45:00.022328: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580AE400 of size 2048
2018-08-28 19:45:00.027300: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580AEC00 of size 2048
2018-08-28 19:45:00.035469: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580AF400 of size 2048
2018-08-28 19:45:00.039335: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580AFC00 of size 2048
2018-08-28 19:45:00.042516: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B0400 of size 2048
2018-08-28 19:45:00.046175: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B0C00 of size 2048
2018-08-28 19:45:00.051119: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B1400 of size 2048
2018-08-28 19:45:00.053659: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B1C00 of size 2048
2018-08-28 19:45:00.056671: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B2400 of size 4096
2018-08-28 19:45:00.059086: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B3400 of size 4096
2018-08-28 19:45:00.061492: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B4400 of size 4096
2018-08-28 19:45:00.065738: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B5400 of size 4096
2018-08-28 19:45:00.069084: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B6400 of size 4096
2018-08-28 19:45:00.072326: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B7400 of size 4096
2018-08-28 19:45:00.076281: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B8400 of size 4096
2018-08-28 19:45:00.079045: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580B9400 of size 4096
2018-08-28 19:45:00.081950: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580BA400 of size 4096
2018-08-28 19:45:00.087246: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580BB400 of size 4096
2018-08-28 19:45:00.090510: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580BC400 of size 4096
2018-08-28 19:45:00.094554: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580BD400 of size 4096
2018-08-28 19:45:00.097382: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580BE400 of size 4096
2018-08-28 19:45:00.100498: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580BF400 of size 4096
2018-08-28 19:45:00.107244: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580C0400 of size 4096
2018-08-28 19:45:00.110838: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580C1400 of size 4096
2018-08-28 19:45:00.115533: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580C2400 of size 4096
2018-08-28 19:45:00.118690: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580C3400 of size 4096
2018-08-28 19:45:00.121689: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580C4400 of size 4096
2018-08-28 19:45:00.124300: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580C5400 of size 4096
2018-08-28 19:45:00.126700: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580C6400 of size 4096
2018-08-28 19:45:00.129251: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580C7400 of size 4096
2018-08-28 19:45:00.134864: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580C8400 of size 4096
2018-08-28 19:45:00.138586: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580C9400 of size 4096
2018-08-28 19:45:00.141645: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CA400 of size 4096
2018-08-28 19:45:00.144524: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CB400 of size 256
2018-08-28 19:45:00.148719: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CB500 of size 256
2018-08-28 19:45:00.152199: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CB600 of size 256
2018-08-28 19:45:00.157310: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CB700 of size 256
2018-08-28 19:45:00.160515: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CB800 of size 256
2018-08-28 19:45:00.163416: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CB900 of size 256
2018-08-28 19:45:00.168138: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CBA00 of size 256
2018-08-28 19:45:00.170846: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CBB00 of size 256
2018-08-28 19:45:00.173600: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CBC00 of size 256
2018-08-28 19:45:00.176601: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CBD00 of size 256
2018-08-28 19:45:00.179305: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CBE00 of size 256
2018-08-28 19:45:00.182133: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CBF00 of size 256
2018-08-28 19:45:00.185547: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CC000 of size 256
2018-08-28 19:45:00.188373: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CC100 of size 256
2018-08-28 19:45:00.195576: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CC200 of size 256
2018-08-28 19:45:00.199730: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CC300 of size 256
2018-08-28 19:45:00.204063: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CC400 of size 256
2018-08-28 19:45:00.209462: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CC500 of size 256
2018-08-28 19:45:00.212195: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CC600 of size 256
2018-08-28 19:45:00.215038: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CC700 of size 256
2018-08-28 19:45:00.218405: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CC800 of size 256
2018-08-28 19:45:00.226377: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CC900 of size 256
2018-08-28 19:45:00.229226: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CCA00 of size 256
2018-08-28 19:45:00.232058: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CCB00 of size 256
2018-08-28 19:45:00.235205: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CCC00 of size 256
2018-08-28 19:45:00.242939: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CCD00 of size 256
2018-08-28 19:45:00.246829: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CCE00 of size 256
2018-08-28 19:45:00.254083: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CCF00 of size 256
2018-08-28 19:45:00.259087: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CD000 of size 256
2018-08-28 19:45:00.262516: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CD100 of size 256
2018-08-28 19:45:00.265949: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CD200 of size 256
2018-08-28 19:45:00.273170: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CD300 of size 256
2018-08-28 19:45:00.276174: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CD400 of size 256
2018-08-28 19:45:00.278913: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CD500 of size 256
2018-08-28 19:45:00.281746: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CD600 of size 256
2018-08-28 19:45:00.296724: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CD700 of size 256
2018-08-28 19:45:00.301262: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CD800 of size 256
2018-08-28 19:45:00.304236: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CD900 of size 256
2018-08-28 19:45:00.306904: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CDA00 of size 256
2018-08-28 19:45:00.309346: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CDB00 of size 256
2018-08-28 19:45:00.313189: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CDC00 of size 256
2018-08-28 19:45:00.315568: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CDD00 of size 256
2018-08-28 19:45:00.318693: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CDE00 of size 256
2018-08-28 19:45:00.321870: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CDF00 of size 256
2018-08-28 19:45:00.324795: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CE000 of size 256
2018-08-28 19:45:00.329279: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CE100 of size 256
2018-08-28 19:45:00.331984: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CE200 of size 256
2018-08-28 19:45:00.335387: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CE300 of size 256
2018-08-28 19:45:00.338980: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CE400 of size 256
2018-08-28 19:45:00.343506: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CE500 of size 256
2018-08-28 19:45:00.346192: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CE600 of size 256
2018-08-28 19:45:00.348910: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CE700 of size 256
2018-08-28 19:45:00.352030: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CE800 of size 256
2018-08-28 19:45:00.355463: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CE900 of size 256
2018-08-28 19:45:00.359894: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CEA00 of size 256
2018-08-28 19:45:00.362533: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CEB00 of size 256
2018-08-28 19:45:00.365077: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CEC00 of size 256
2018-08-28 19:45:00.368579: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CED00 of size 256
2018-08-28 19:45:00.371864: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580CEE00 of size 9984
2018-08-28 19:45:00.376273: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000006580D1500 of size 18727936
2018-08-28 19:45:00.378879: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:671] Summary of in-use Chunks by size:
2018-08-28 19:45:00.381588: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 61 Chunks of size 256 totalling 15.3KiB
2018-08-28 19:45:00.384842: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.3KiB
2018-08-28 19:45:00.388551: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 26 Chunks of size 2048 totalling 52.0KiB
2018-08-28 19:45:00.393521: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 26 Chunks of size 4096 totalling 104.0KiB
2018-08-28 19:45:00.396338: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 26 Chunks of size 8192 totalling 208.0KiB
2018-08-28 19:45:00.399257: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 9984 totalling 9.8KiB
2018-08-28 19:45:00.402560: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 26 Chunks of size 16128 totalling 409.5KiB
2018-08-28 19:45:00.410325: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 26 Chunks of size 2000128 totalling 49.59MiB
2018-08-28 19:45:00.413282: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 26 Chunks of size 8000000 totalling 198.36MiB
2018-08-28 19:45:00.416082: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 18727936 totalling 17.86MiB
2018-08-28 19:45:00.423934: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 26 Chunks of size 46080000 totalling 1.12GiB
2018-08-28 19:45:00.426999: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:678] Sum Total of in-use chunks: 1.38GiB
2018-08-28 19:45:00.429629: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats:
Limit: 1477630361
InUse: 1477630208
MaxInUse: 1477630208
NumAllocs: 11895
MaxAllocSize: 57145344
2018-08-28 19:45:00.440222: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:279] ****
2018-08-28 19:45:00.445770: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1275] OP_REQUIRES failed at matmul_op.cc:474 : Resource exhausted: OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1278, in _do_call
return fn(*args)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_3, Variable/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "MultilayerPerceptron.py", line 194, in
main()
File "MultilayerPerceptron.py", line 180, in main
obj.train_model(prediction, X_train, X_test, Y_train, Y_test)
File "MultilayerPerceptron.py", line 99, in trainmodel
, c = sess.run([optimizer, cost], feed_dict={self.x:X_train, self.y:y_train})
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 877, in run
run_metadata_ptr)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1272, in _do_run
run_metadata)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_3, Variable/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'MatMul', defined at:
File "MultilayerPerceptron.py", line 194, in
main()
File "MultilayerPerceptron.py", line 130, in main
prediction = obj.ModelGraph()
File "MultilayerPerceptron.py", line 51, in ModelGraph
layer_1 = tf.add(tf.matmul(self.x, self.weights['h1']), self.biases['b1'])
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 2018, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 4750, in mat_mul
name=name)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\util\deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 3155, in create_op
op_def=op_def)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1717, in init
self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_3, Variable/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
@HarikrishnaRanpariya #3
During the MultilayerPerceptron.py execution of the code, execution was interrupted. Problem could be buffer related, please try to resolve as soon as possible.
2018-08-28 19:45:00.440222: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:279] **** 2018-08-28 19:45:00.445770: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1275] OP_REQUIRES failed at matmul_op.cc:474 : Resource exhausted: OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1278, in _do_call return fn(*args) File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1263, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_3, Variable/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "MultilayerPerceptron.py", line 194, in
main()
File "MultilayerPerceptron.py", line 180, in main
obj.train_model(prediction, X_train, X_test, Y_train, Y_test)
File "MultilayerPerceptron.py", line 99, in trainmodel
, c = sess.run([optimizer, cost], feed_dict={self.x:X_train, self.y:y_train})
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 877, in run
run_metadata_ptr)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1272, in _do_run
run_metadata)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_3, Variable/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'MatMul', defined at: File "MultilayerPerceptron.py", line 194, in
main()
File "MultilayerPerceptron.py", line 130, in main
prediction = obj.ModelGraph()
File "MultilayerPerceptron.py", line 51, in ModelGraph
layer_1 = tf.add(tf.matmul(self.x, self.weights['h1']), self.biases['b1'])
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 2018, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 4750, in mat_mul
name=name)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\util\deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 3155, in create_op
op_def=op_def)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1717, in init
self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_3, Variable/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. @HarikrishnaRanpariya #3