out of memory

[dadamsncsa]

Here's the trace using 4 K80 GPUs:

(unet) [dadams2@gpu008 unet-sem]$ python main.py 
Using TensorFlow backend.
{'ngpu': 1, 'crop_size': (256, 256), 'lr': 0.0001, 'nepochs': 5, 'crop': True, 'input_size': (256, 256), 'split': 0.1, 'batch_size': 32, 'max_crop': False, 'augment': True}
/home/dadams2/unet-sem/model.py:58: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("co..., inputs=Tensor("in...)`
  model = Model(input = inputs, output = conv10)
2019-07-21 13:05:37.177869: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-07-21 13:05:37.413692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:83:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-07-21 13:05:37.585531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 1 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:84:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-07-21 13:05:37.585922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1
2019-07-21 13:05:39.949113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-21 13:05:39.949181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1 
2019-07-21 13:05:39.949195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N Y 
2019-07-21 13:05:39.949204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   Y N 
2019-07-21 13:05:39.950090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10759 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:83:00.0, compute capability: 3.7)
2019-07-21 13:05:40.128999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10759 MB memory) -> physical GPU (device: 1, name: Tesla K80, pci bus id: 0000:84:00.0, compute capability: 3.7)
2019-07-21 13:05:52.569072: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.00GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-21 13:05:53.564076: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.38GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-21 13:06:03.565416: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 512.00MiB.  Current allocation summary follows.
2019-07-21 13:06:03.565501: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256):   Total Chunks: 136, Chunks in use: 135. 34.0KiB allocated for chunks. 33.8KiB in use in bin. 7.1KiB client-requested in use in bin.
2019-07-21 13:06:03.565605: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512):   Total Chunks: 26, Chunks in use: 26. 13.0KiB allocated for chunks. 13.0KiB in use in bin. 13.0KiB client-requested in use in bin.
2019-07-21 13:06:03.565648: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024):  Total Chunks: 27, Chunks in use: 27. 27.2KiB allocated for chunks. 27.2KiB in use in bin. 27.0KiB client-requested in use in bin.
2019-07-21 13:06:03.565674: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048):  Total Chunks: 32, Chunks in use: 32. 65.5KiB allocated for chunks. 65.5KiB in use in bin. 65.5KiB client-requested in use in bin.
2019-07-21 13:06:03.565712: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096):  Total Chunks: 17, Chunks in use: 17. 71.0KiB allocated for chunks. 71.0KiB in use in bin. 71.0KiB client-requested in use in bin.
2019-07-21 13:06:03.565736: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 13:06:03.565773: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 13:06:03.565810: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 13:06:03.565847: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 13:06:03.565871: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072):    Total Chunks: 18, Chunks in use: 18. 2.44MiB allocated for chunks. 2.44MiB in use in bin. 2.44MiB client-requested in use in bin.
2019-07-21 13:06:03.565902: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144):    Total Chunks: 12, Chunks in use: 12. 3.38MiB allocated for chunks. 3.38MiB in use in bin. 3.38MiB client-requested in use in bin.
2019-07-21 13:06:03.565933: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288):    Total Chunks: 17, Chunks in use: 17. 9.19MiB allocated for chunks. 9.19MiB in use in bin. 9.19MiB client-requested in use in bin.
2019-07-21 13:06:03.565965: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576):   Total Chunks: 12, Chunks in use: 12. 13.50MiB allocated for chunks. 13.50MiB in use in bin. 13.50MiB client-requested in use in bin.
2019-07-21 13:06:03.565997: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152):   Total Chunks: 18, Chunks in use: 17. 39.00MiB allocated for chunks. 36.75MiB in use in bin. 36.75MiB client-requested in use in bin.
2019-07-21 13:06:03.566345: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304):   Total Chunks: 13, Chunks in use: 12. 58.50MiB allocated for chunks. 54.00MiB in use in bin. 54.00MiB client-requested in use in bin.
2019-07-21 13:06:03.566382: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608):   Total Chunks: 20, Chunks in use: 19. 172.00MiB allocated for chunks. 163.00MiB in use in bin. 163.00MiB client-requested in use in bin.
2019-07-21 13:06:03.566409: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216):  Total Chunks: 13, Chunks in use: 13. 232.00MiB allocated for chunks. 232.00MiB in use in bin. 232.00MiB client-requested in use in bin.
2019-07-21 13:06:03.566444: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432):  Total Chunks: 13, Chunks in use: 11. 462.00MiB allocated for chunks. 394.00MiB in use in bin. 376.00MiB client-requested in use in bin.
2019-07-21 13:06:03.566477: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864):  Total Chunks: 10, Chunks in use: 9. 673.12MiB allocated for chunks. 609.12MiB in use in bin. 576.00MiB client-requested in use in bin.
2019-07-21 13:06:03.566502: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728):     Total Chunks: 10, Chunks in use: 8. 1.38GiB allocated for chunks. 1.06GiB in use in bin. 1.00GiB client-requested in use in bin.
2019-07-21 13:06:03.566538: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456):     Total Chunks: 17, Chunks in use: 16. 7.50GiB allocated for chunks. 7.25GiB in use in bin. 7.25GiB client-requested in use in bin.
2019-07-21 13:06:03.566572: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 512.00MiB was 256.00MiB, Chunk State: 
2019-07-21 13:06:03.566608: I tensorflow/core/common_runtime/bfc_allocator.cc:652]   Size: 256.00MiB | Requested Size: 256.00MiB | in_use: 0, prev:   Size: 128.00MiB | Requested Size: 128.00MiB | in_use: 1, next:   Size: 256.00MiB | Requested Size: 256.00MiB | in_use: 1
2019-07-21 13:06:03.566638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80000 of size 1280
2019-07-21 13:06:03.566671: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80500 of size 256
2019-07-21 13:06:03.566692: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80600 of size 256
2019-07-21 13:06:03.566709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80700 of size 256
2019-07-21 13:06:03.566725: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80800 of size 256
2019-07-21 13:06:03.566747: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80900 of size 256
2019-07-21 13:06:03.566763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80a00 of size 256
2019-07-21 13:06:03.566779: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80b00 of size 256
2019-07-21 13:06:03.566801: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80c00 of size 256
2019-07-21 13:06:03.566819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80d00 of size 256
2019-07-21 13:06:03.566841: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80e00 of size 256
2019-07-21 13:06:03.566858: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d80f00 of size 256
2019-07-21 13:06:03.566875: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d81000 of size 256
2019-07-21 13:06:03.566890: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d81100 of size 256
2019-07-21 13:06:03.566911: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d81200 of size 256
2019-07-21 13:06:03.566927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d81300 of size 256
2019-07-21 13:06:03.566943: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d81400 of size 256
2019-07-21 13:06:03.566961: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d81500 of size 256
2019-07-21 13:06:03.566979: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d81600 of size 2304
2019-07-21 13:06:03.566997: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307d81f00 of size 294912
2019-07-21 13:06:03.567022: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2307dc9f00 of size 4718592
2019-07-21 13:06:03.567042: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2308249f00 of size 18874368
2019-07-21 13:06:03.567063: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2309449f00 of size 37748736
2019-07-21 13:06:03.567086: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230b849f00 of size 4096
2019-07-21 13:06:03.567103: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230b84af00 of size 8388608
2019-07-21 13:06:03.567121: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230c04af00 of size 18874368
2019-07-21 13:06:03.567139: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230d24af00 of size 9437184
2019-07-21 13:06:03.567160: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230db4af00 of size 2048
2019-07-21 13:06:03.567178: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230db4b700 of size 2097152
2019-07-21 13:06:03.567199: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230dd4b700 of size 4718592
2019-07-21 13:06:03.567215: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e1cb700 of size 2359296
2019-07-21 13:06:03.567233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e40b700 of size 1179648
2019-07-21 13:06:03.567249: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e52b700 of size 524288
2019-07-21 13:06:03.567266: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e5ab700 of size 1179648
2019-07-21 13:06:03.567282: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e6cb700 of size 589824
2019-07-21 13:06:03.567308: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e75b700 of size 512
2019-07-21 13:06:03.567331: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e75b900 of size 131072
2019-07-21 13:06:03.567348: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e77b900 of size 294912
2019-07-21 13:06:03.567365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e7c3900 of size 1024
2019-07-21 13:06:03.567382: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e7c3d00 of size 147456
2019-07-21 13:06:03.567399: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e7e7d00 of size 256
2019-07-21 13:06:03.567416: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e7e7e00 of size 4608
2019-07-21 13:06:03.567433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e7e9000 of size 256
2019-07-21 13:06:03.567455: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e7e9100 of size 256
2019-07-21 13:06:03.567471: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e7e9200 of size 256
2019-07-21 13:06:03.567488: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e7e9300 of size 2304
2019-07-21 13:06:03.567506: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e7e9c00 of size 294912
2019-07-21 13:06:03.567524: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e831c00 of size 294912
2019-07-21 13:06:03.567544: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230e879c00 of size 4718592
2019-07-21 13:06:03.567562: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230ecf9c00 of size 18874368
2019-07-21 13:06:03.567579: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fef9c00 of size 18874368
2019-07-21 13:06:03.567599: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23110f9c00 of size 37748736
2019-07-21 13:06:03.567621: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23134f9c00 of size 37748736
2019-07-21 13:06:03.567637: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23158f9c00 of size 4096
2019-07-21 13:06:03.567653: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23158fac00 of size 4096
2019-07-21 13:06:03.567673: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23158fbc00 of size 4096
2019-07-21 13:06:03.567689: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23158fcc00 of size 4096
2019-07-21 13:06:03.567705: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23158fdc00 of size 8388608
2019-07-21 13:06:03.567722: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23160fdc00 of size 8388608
2019-07-21 13:06:03.567738: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23168fdc00 of size 18874368
2019-07-21 13:06:03.567758: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317afdc00 of size 18874368
2019-07-21 13:06:03.567775: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318cfdc00 of size 9437184
2019-07-21 13:06:03.567791: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23195fdc00 of size 9437184
2019-07-21 13:06:03.567807: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319efdc00 of size 9437184
2019-07-21 13:06:03.567825: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231a7fdc00 of size 9437184
2019-07-21 13:06:03.567844: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b0fdc00 of size 2048
2019-07-21 13:06:03.567864: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b0fe400 of size 2048
2019-07-21 13:06:03.567880: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b0fec00 of size 2048
2019-07-21 13:06:03.567901: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b0ff400 of size 2048
2019-07-21 13:06:03.567916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b0ffc00 of size 2048
2019-07-21 13:06:03.567933: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b100400 of size 2048
2019-07-21 13:06:03.567950: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b100c00 of size 2048
2019-07-21 13:06:03.567970: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b101400 of size 2048
2019-07-21 13:06:03.567987: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b101c00 of size 2048
2019-07-21 13:06:03.568007: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b102400 of size 2048
2019-07-21 13:06:03.568034: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b102c00 of size 2097152
2019-07-21 13:06:03.568052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b302c00 of size 2097152
2019-07-21 13:06:03.568072: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b502c00 of size 4718592
2019-07-21 13:06:03.568094: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b982c00 of size 4718592
2019-07-21 13:06:03.568111: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231be02c00 of size 2359296
2019-07-21 13:06:03.568126: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c042c00 of size 2359296
2019-07-21 13:06:03.568143: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c282c00 of size 2359296
2019-07-21 13:06:03.568159: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c4c2c00 of size 1179648
2019-07-21 13:06:03.568180: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c5e2c00 of size 1179648
2019-07-21 13:06:03.568197: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c702c00 of size 524288
2019-07-21 13:06:03.568213: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c782c00 of size 524288
2019-07-21 13:06:03.568229: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c802c00 of size 1179648
2019-07-21 13:06:03.568247: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c922c00 of size 1179648
2019-07-21 13:06:03.568264: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231ca42c00 of size 589824
2019-07-21 13:06:03.568285: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cad2c00 of size 589824
2019-07-21 13:06:03.568302: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cb62c00 of size 589824
2019-07-21 13:06:03.568318: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cbf2c00 of size 589824
2019-07-21 13:06:03.568338: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc82c00 of size 512
2019-07-21 13:06:03.568355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc82e00 of size 512
2019-07-21 13:06:03.568371: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc83000 of size 512
2019-07-21 13:06:03.568387: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc83200 of size 512
2019-07-21 13:06:03.568404: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc83400 of size 512
2019-07-21 13:06:03.568423: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc83600 of size 512
2019-07-21 13:06:03.568443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc83800 of size 512
2019-07-21 13:06:03.568459: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc83a00 of size 512
2019-07-21 13:06:03.568475: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc83c00 of size 512
2019-07-21 13:06:03.568490: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc83e00 of size 512
2019-07-21 13:06:03.568508: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cc84000 of size 131072
2019-07-21 13:06:03.568529: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cca4000 of size 131072
2019-07-21 13:06:03.568544: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231ccc4000 of size 294912
2019-07-21 13:06:03.568565: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd0c000 of size 294912
2019-07-21 13:06:03.568582: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd54000 of size 1024
2019-07-21 13:06:03.568600: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd54400 of size 1024
2019-07-21 13:06:03.568617: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd54800 of size 1024
2019-07-21 13:06:03.568632: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd54c00 of size 1024
2019-07-21 13:06:03.568655: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd55000 of size 1024
2019-07-21 13:06:03.568672: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd55400 of size 1024
2019-07-21 13:06:03.568689: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd55800 of size 1024
2019-07-21 13:06:03.568711: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd55c00 of size 1024
2019-07-21 13:06:03.568727: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd56000 of size 1024
2019-07-21 13:06:03.568745: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd56400 of size 147456
2019-07-21 13:06:03.568764: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd7a400 of size 147456
2019-07-21 13:06:03.568780: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cd9e400 of size 147456
2019-07-21 13:06:03.568797: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdc2400 of size 147456
2019-07-21 13:06:03.568819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde6400 of size 256
2019-07-21 13:06:03.568835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde6500 of size 256
2019-07-21 13:06:03.568855: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde6600 of size 256
2019-07-21 13:06:03.568877: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde6700 of size 256
2019-07-21 13:06:03.568898: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde6800 of size 256
2019-07-21 13:06:03.568918: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde6900 of size 256
2019-07-21 13:06:03.568935: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde6a00 of size 256
2019-07-21 13:06:03.568955: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde6b00 of size 256
2019-07-21 13:06:03.568973: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde6c00 of size 256
2019-07-21 13:06:03.568991: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde6d00 of size 4608
2019-07-21 13:06:03.569011: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde7f00 of size 4608
2019-07-21 13:06:03.569040: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9100 of size 256
2019-07-21 13:06:03.569057: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9200 of size 256
2019-07-21 13:06:03.569073: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9300 of size 256
2019-07-21 13:06:03.569089: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9400 of size 256
2019-07-21 13:06:03.569109: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9500 of size 256
2019-07-21 13:06:03.569126: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9600 of size 256
2019-07-21 13:06:03.569142: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9700 of size 256
2019-07-21 13:06:03.569157: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9800 of size 256
2019-07-21 13:06:03.569174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9900 of size 256
2019-07-21 13:06:03.569191: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9a00 of size 256
2019-07-21 13:06:03.569212: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9b00 of size 256
2019-07-21 13:06:03.569229: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9c00 of size 256
2019-07-21 13:06:03.569247: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9d00 of size 256
2019-07-21 13:06:03.569267: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9e00 of size 256
2019-07-21 13:06:03.569284: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cde9f00 of size 256
2019-07-21 13:06:03.569302: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdea000 of size 256
2019-07-21 13:06:03.569320: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdea100 of size 256
2019-07-21 13:06:03.569337: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdea200 of size 256
2019-07-21 13:06:03.569354: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdea300 of size 256
2019-07-21 13:06:03.569371: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdea400 of size 256
2019-07-21 13:06:03.569389: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdea500 of size 256
2019-07-21 13:06:03.569406: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdea600 of size 256
2019-07-21 13:06:03.569422: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdea700 of size 256
2019-07-21 13:06:03.569438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdea800 of size 256
2019-07-21 13:06:03.569455: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdea900 of size 256
2019-07-21 13:06:03.569477: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdeaa00 of size 256
2019-07-21 13:06:03.569498: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdeab00 of size 256
2019-07-21 13:06:03.569514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdeac00 of size 256
2019-07-21 13:06:03.569536: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdead00 of size 256
2019-07-21 13:06:03.569552: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdeae00 of size 256
2019-07-21 13:06:03.569569: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdeaf00 of size 256
2019-07-21 13:06:03.569589: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdeb000 of size 256
2019-07-21 13:06:03.569605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdeb100 of size 256
2019-07-21 13:06:03.569627: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdeb200 of size 256
2019-07-21 13:06:03.569643: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdeb300 of size 2304
2019-07-21 13:06:03.569661: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdebc00 of size 4096
2019-07-21 13:06:03.569678: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cdecc00 of size 37748736
2019-07-21 13:06:03.569695: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231f1ecc00 of size 2048
2019-07-21 13:06:03.569712: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231f1ed400 of size 8388608
2019-07-21 13:06:03.569728: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231f9ed400 of size 2048
2019-07-21 13:06:03.569745: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231f9edc00 of size 18874368
2019-07-21 13:06:03.569762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320bedc00 of size 2048
2019-07-21 13:06:03.569784: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320bee400 of size 1024
2019-07-21 13:06:03.569801: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320bee800 of size 2097152
2019-07-21 13:06:03.569819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320dee800 of size 1024
2019-07-21 13:06:03.569835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320deec00 of size 4718592
2019-07-21 13:06:03.569851: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232126ec00 of size 1024
2019-07-21 13:06:03.569867: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232126f000 of size 512
2019-07-21 13:06:03.569884: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232126f200 of size 524288
2019-07-21 13:06:03.569900: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23212ef200 of size 512
2019-07-21 13:06:03.569916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23212ef400 of size 1179648
2019-07-21 13:06:03.569933: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232140f400 of size 512
2019-07-21 13:06:03.569952: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232140f600 of size 256
2019-07-21 13:06:03.569968: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232140f700 of size 256
2019-07-21 13:06:03.569989: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232140f800 of size 131072
2019-07-21 13:06:03.570010: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232142f800 of size 256
2019-07-21 13:06:03.570033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232142f900 of size 294912
2019-07-21 13:06:03.570052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321477900 of size 256
2019-07-21 13:06:03.570073: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321477a00 of size 147456
2019-07-21 13:06:03.570094: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232149ba00 of size 147456
2019-07-21 13:06:03.570110: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23214bfa00 of size 256
2019-07-21 13:06:03.570127: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23214bfb00 of size 4608
2019-07-21 13:06:03.570143: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23214c0d00 of size 256
2019-07-21 13:06:03.570163: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23214c0e00 of size 256
2019-07-21 13:06:03.570184: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23214c0f00 of size 512
2019-07-21 13:06:03.570201: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23214c1100 of size 294912
2019-07-21 13:06:03.570219: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321509100 of size 512
2019-07-21 13:06:03.570239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321509300 of size 589824
2019-07-21 13:06:03.570259: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321599300 of size 589824
2019-07-21 13:06:03.570276: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321629300 of size 1024
2019-07-21 13:06:03.570292: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321629700 of size 1179648
2019-07-21 13:06:03.570308: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321749700 of size 1024
2019-07-21 13:06:03.570328: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321749b00 of size 2359296
2019-07-21 13:06:03.570346: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321989b00 of size 2359296
2019-07-21 13:06:03.570362: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321bc9b00 of size 2048
2019-07-21 13:06:03.570383: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321bca300 of size 4718592
2019-07-21 13:06:03.570403: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232204a300 of size 2048
2019-07-21 13:06:03.570420: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232204ab00 of size 9437184
2019-07-21 13:06:03.570442: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232294ab00 of size 9437184
2019-07-21 13:06:03.570463: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232324ab00 of size 4096
2019-07-21 13:06:03.570480: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232324bb00 of size 18874368
2019-07-21 13:06:03.570498: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232444bb00 of size 2304
2019-07-21 13:06:03.570515: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232444c400 of size 256
2019-07-21 13:06:03.570539: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232444c500 of size 2359296
2019-07-21 13:06:03.570556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468c500 of size 256
2019-07-21 13:06:03.570573: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468c600 of size 256
2019-07-21 13:06:03.570593: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468c700 of size 256
2019-07-21 13:06:03.570609: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468c800 of size 256
2019-07-21 13:06:03.570625: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468c900 of size 256
2019-07-21 13:06:03.570643: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468ca00 of size 256
2019-07-21 13:06:03.570660: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468cb00 of size 256
2019-07-21 13:06:03.570677: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468cc00 of size 256
2019-07-21 13:06:03.570696: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468cd00 of size 256
2019-07-21 13:06:03.570713: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468ce00 of size 256
2019-07-21 13:06:03.570729: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468cf00 of size 1024
2019-07-21 13:06:03.570747: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468d300 of size 256
2019-07-21 13:06:03.570765: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468d400 of size 256
2019-07-21 13:06:03.570787: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468d500 of size 256
2019-07-21 13:06:03.570804: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468d600 of size 256
2019-07-21 13:06:03.570820: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468d700 of size 256
2019-07-21 13:06:03.570836: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468d800 of size 256
2019-07-21 13:06:03.570858: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468d900 of size 256
2019-07-21 13:06:03.570874: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468da00 of size 256
2019-07-21 13:06:03.570890: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468db00 of size 256
2019-07-21 13:06:03.570906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468dc00 of size 256
2019-07-21 13:06:03.570923: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232468dd00 of size 4718592
2019-07-21 13:06:03.570939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0dd00 of size 256
2019-07-21 13:06:03.570955: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0de00 of size 256
2019-07-21 13:06:03.570971: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0df00 of size 256
2019-07-21 13:06:03.570987: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0e000 of size 256
2019-07-21 13:06:03.571003: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0e100 of size 256
2019-07-21 13:06:03.571025: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0e200 of size 256
2019-07-21 13:06:03.571043: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0e300 of size 256
2019-07-21 13:06:03.571059: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0e400 of size 256
2019-07-21 13:06:03.571075: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0e500 of size 256
2019-07-21 13:06:03.571095: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0e600 of size 256
2019-07-21 13:06:03.571112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0e700 of size 256
2019-07-21 13:06:03.571130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0e800 of size 256
2019-07-21 13:06:03.571150: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0e900 of size 256
2019-07-21 13:06:03.571166: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0ea00 of size 256
2019-07-21 13:06:03.571182: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0eb00 of size 256
2019-07-21 13:06:03.571202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0ec00 of size 256
2019-07-21 13:06:03.571222: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0ed00 of size 256
2019-07-21 13:06:03.571239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0ee00 of size 256
2019-07-21 13:06:03.571259: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0ef00 of size 256
2019-07-21 13:06:03.571280: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0f000 of size 256
2019-07-21 13:06:03.571296: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b0f100 of size 8388608
2019-07-21 13:06:03.571313: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232530f100 of size 8388608
2019-07-21 13:06:03.571329: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2325b0f100 of size 256
2019-07-21 13:06:03.571345: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2325b0f200 of size 256
2019-07-21 13:06:03.571365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2325b0f300 of size 256
2019-07-21 13:06:03.571388: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2325b0f400 of size 134217728
2019-07-21 13:06:03.571407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232db0f400 of size 67108864
2019-07-21 13:06:03.571426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331b0f400 of size 33554432
2019-07-21 13:06:03.571443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2333b0f400 of size 101842944
2019-07-21 13:06:03.571464: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2339c2f400 of size 200149248
2019-07-21 13:06:03.571485: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2345b0fd00 of size 536870912
2019-07-21 13:06:03.571502: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2365b0fd00 of size 256
2019-07-21 13:06:03.571518: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x2365b0fe00 of size 256
2019-07-21 13:06:03.571534: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2365b0ff00 of size 256
2019-07-21 13:06:03.571551: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2365b10000 of size 2359296
2019-07-21 13:06:03.571568: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2365d50000 of size 1024
2019-07-21 13:06:03.571590: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2365d50400 of size 524288
2019-07-21 13:06:03.571607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2365dd0400 of size 2359296
2019-07-21 13:06:03.571624: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366010400 of size 1024
2019-07-21 13:06:03.571640: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366010800 of size 524288
2019-07-21 13:06:03.571657: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366090800 of size 512
2019-07-21 13:06:03.571678: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366090a00 of size 256
2019-07-21 13:06:03.571698: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366090b00 of size 512
2019-07-21 13:06:03.571715: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366090d00 of size 1179648
2019-07-21 13:06:03.571733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23661b0d00 of size 1179648
2019-07-21 13:06:03.571750: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23662d0d00 of size 512
2019-07-21 13:06:03.571767: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23662d0f00 of size 512
2019-07-21 13:06:03.571788: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23662d1100 of size 256
2019-07-21 13:06:03.571805: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23662d1200 of size 589824
2019-07-21 13:06:03.571828: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366361200 of size 589824
2019-07-21 13:06:03.571844: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23663f1200 of size 512
2019-07-21 13:06:03.571860: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23663f1400 of size 512
2019-07-21 13:06:03.571876: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23663f1600 of size 131072
2019-07-21 13:06:03.571893: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366411600 of size 131072
2019-07-21 13:06:03.571909: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366431600 of size 256
2019-07-21 13:06:03.571925: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366431700 of size 256
2019-07-21 13:06:03.571942: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366431800 of size 294912
2019-07-21 13:06:03.571963: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366479800 of size 294912
2019-07-21 13:06:03.571981: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23664c1800 of size 256
2019-07-21 13:06:03.572001: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23664c1900 of size 256
2019-07-21 13:06:03.572029: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23664c1a00 of size 147456
2019-07-21 13:06:03.572047: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23664e5a00 of size 147456
2019-07-21 13:06:03.572065: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366509a00 of size 256
2019-07-21 13:06:03.572082: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366509b00 of size 256
2019-07-21 13:06:03.572098: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366509c00 of size 4608
2019-07-21 13:06:03.572114: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236650ae00 of size 4608
2019-07-21 13:06:03.572130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236650c000 of size 256
2019-07-21 13:06:03.572147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236650c100 of size 256
2019-07-21 13:06:03.572169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236650c200 of size 294912
2019-07-21 13:06:03.572185: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366554200 of size 294912
2019-07-21 13:06:03.572201: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236659c200 of size 256
2019-07-21 13:06:03.572222: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236659c300 of size 256
2019-07-21 13:06:03.572243: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236659c400 of size 256
2019-07-21 13:06:03.572259: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236659c500 of size 256
2019-07-21 13:06:03.572277: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236659c600 of size 512
2019-07-21 13:06:03.572300: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236659c800 of size 512
2019-07-21 13:06:03.572322: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236659ca00 of size 2304
2019-07-21 13:06:03.572344: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236659d300 of size 2304
2019-07-21 13:06:03.572361: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236659dc00 of size 589824
2019-07-21 13:06:03.572378: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236662dc00 of size 589824
2019-07-21 13:06:03.572396: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23666bdc00 of size 512
2019-07-21 13:06:03.572412: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23666bde00 of size 512
2019-07-21 13:06:03.572428: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23666be000 of size 1179648
2019-07-21 13:06:03.572444: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23667de000 of size 1179648
2019-07-21 13:06:03.572461: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23668fe000 of size 1024
2019-07-21 13:06:03.572479: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23668fe400 of size 1024
2019-07-21 13:06:03.572498: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23668fe800 of size 2359296
2019-07-21 13:06:03.572514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366b3e800 of size 2359296
2019-07-21 13:06:03.572532: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366d7e800 of size 1024
2019-07-21 13:06:03.572549: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366d7ec00 of size 1024
2019-07-21 13:06:03.572566: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366d7f000 of size 4718592
2019-07-21 13:06:03.572588: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23671ff000 of size 4718592
2019-07-21 13:06:03.572609: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236767f000 of size 2048
2019-07-21 13:06:03.572626: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236767f800 of size 2048
2019-07-21 13:06:03.572647: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2367680000 of size 9437184
2019-07-21 13:06:03.572668: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2367f80000 of size 9437184
2019-07-21 13:06:03.572685: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2368880000 of size 2048
2019-07-21 13:06:03.572702: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2368880800 of size 2048
2019-07-21 13:06:03.572719: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2368881000 of size 256
2019-07-21 13:06:03.572741: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2368881100 of size 256
2019-07-21 13:06:03.572763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2368881200 of size 18874368
2019-07-21 13:06:03.572779: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2369a81200 of size 18874368
2019-07-21 13:06:03.572795: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236ac81200 of size 4096
2019-07-21 13:06:03.572811: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236ac82200 of size 4096
2019-07-21 13:06:03.572828: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236ac83200 of size 37748736
2019-07-21 13:06:03.572844: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236d083200 of size 37748736
2019-07-21 13:06:03.572861: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236f483200 of size 4096
2019-07-21 13:06:03.572877: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236f484200 of size 4096
2019-07-21 13:06:03.572898: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236f485200 of size 8388608
2019-07-21 13:06:03.572919: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236fc85200 of size 8388608
2019-07-21 13:06:03.572936: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370485200 of size 2048
2019-07-21 13:06:03.572952: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370485a00 of size 2048
2019-07-21 13:06:03.572968: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370486200 of size 18874368
2019-07-21 13:06:03.572985: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2371686200 of size 18874368
2019-07-21 13:06:03.573001: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372886200 of size 2048
2019-07-21 13:06:03.573022: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372886a00 of size 2048
2019-07-21 13:06:03.573041: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372887200 of size 9437184
2019-07-21 13:06:03.573062: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373187200 of size 9437184
2019-07-21 13:06:03.573083: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373a87200 of size 2048
2019-07-21 13:06:03.573100: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373a87a00 of size 2048
2019-07-21 13:06:03.573116: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373a88200 of size 147456
2019-07-21 13:06:03.573132: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373aac200 of size 147456
2019-07-21 13:06:03.573149: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373ad0200 of size 2097152
2019-07-21 13:06:03.573170: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373cd0200 of size 2097152
2019-07-21 13:06:03.573187: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373ed0200 of size 1024
2019-07-21 13:06:03.573203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373ed0600 of size 1024
2019-07-21 13:06:03.573223: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373ed0a00 of size 4718592
2019-07-21 13:06:03.573246: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2374350a00 of size 4718592
2019-07-21 13:06:03.573262: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23747d0a00 of size 1024
2019-07-21 13:06:03.573279: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23747d0e00 of size 1024
2019-07-21 13:06:03.573296: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23747d1200 of size 256
2019-07-21 13:06:03.573313: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23747d1300 of size 256
2019-07-21 13:06:03.573335: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23747d1400 of size 147456
2019-07-21 13:06:03.573352: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23747f5400 of size 536870912
2019-07-21 13:06:03.573376: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23947f5400 of size 268435456
2019-07-21 13:06:03.573394: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a47f5400 of size 268435456
2019-07-21 13:06:03.573410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23b47f5400 of size 2359296
2019-07-21 13:06:03.573432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23b4a35400 of size 134217728
2019-07-21 13:06:03.573449: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23bca35400 of size 4718592
2019-07-21 13:06:03.573465: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23bceb5400 of size 67108864
2019-07-21 13:06:03.573483: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23c0eb5400 of size 9437184
2019-07-21 13:06:03.573500: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23c17b5400 of size 67108864
2019-07-21 13:06:03.573517: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23c57b5400 of size 67108864
2019-07-21 13:06:03.573535: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23c97b5400 of size 67108864
2019-07-21 13:06:03.573553: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23cd7b5400 of size 16777216
2019-07-21 13:06:03.573570: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ce7b5400 of size 52428800
2019-07-21 13:06:03.573587: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d19b5400 of size 33554432
2019-07-21 13:06:03.573604: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23d39b5400 of size 37748736
2019-07-21 13:06:03.573620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d5db5400 of size 33554432
2019-07-21 13:06:03.573642: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23d7db5400 of size 33554432
2019-07-21 13:06:03.573663: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d9db5400 of size 33554432
2019-07-21 13:06:03.573680: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23dbdb5400 of size 67108864
2019-07-21 13:06:03.573696: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23dfdb5400 of size 67108864
2019-07-21 13:06:03.573712: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e3db5400 of size 134217728
2019-07-21 13:06:03.573729: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ebdb5400 of size 134217728
2019-07-21 13:06:03.573745: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f3db5400 of size 67108864
2019-07-21 13:06:03.573762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f7db5400 of size 67108864
2019-07-21 13:06:03.573778: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23fbdb5400 of size 134217728
2019-07-21 13:06:03.573800: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2403db5400 of size 134217728
2019-07-21 13:06:03.573817: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x240bdb5400 of size 268435456
2019-07-21 13:06:03.573840: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241bdb5400 of size 268435456
2019-07-21 13:06:03.573856: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242bdb5400 of size 134217728
2019-07-21 13:06:03.573872: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2433db5400 of size 134217728
2019-07-21 13:06:03.573889: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x243bdb5400 of size 268435456
2019-07-21 13:06:03.573909: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x244bdb5400 of size 268435456
2019-07-21 13:06:03.573926: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245bdb5400 of size 536870912
2019-07-21 13:06:03.573942: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247bdb5400 of size 536870912
2019-07-21 13:06:03.573964: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249bdb5400 of size 268435456
2019-07-21 13:06:03.573985: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24abdb5400 of size 268435456
2019-07-21 13:06:03.574002: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24bbdb5400 of size 536870912
2019-07-21 13:06:03.574028: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24dbdb5400 of size 536870912
2019-07-21 13:06:03.574053: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24fbdb5400 of size 1073741824
2019-07-21 13:06:03.574070: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253bdb5400 of size 1073741824
2019-07-21 13:06:03.574087: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x257bdb5400 of size 536870912
2019-07-21 13:06:03.574409: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x259bdb5400 of size 208746496
2019-07-21 13:06:03.574444: I tensorflow/core/common_runtime/bfc_allocator.cc:671]      Summary of in-use Chunks by size: 
2019-07-21 13:06:03.574466: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 135 Chunks of size 256 totalling 33.8KiB
2019-07-21 13:06:03.574486: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 26 Chunks of size 512 totalling 13.0KiB
2019-07-21 13:06:03.574506: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 26 Chunks of size 1024 totalling 26.0KiB
2019-07-21 13:06:03.574529: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.2KiB
2019-07-21 13:06:03.574552: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 26 Chunks of size 2048 totalling 52.0KiB
2019-07-21 13:06:03.574576: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 2304 totalling 13.5KiB
2019-07-21 13:06:03.574594: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 4096 totalling 44.0KiB
2019-07-21 13:06:03.574612: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 4608 totalling 27.0KiB
2019-07-21 13:06:03.574631: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 131072 totalling 768.0KiB
2019-07-21 13:06:03.574655: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 147456 totalling 1.69MiB
2019-07-21 13:06:03.574673: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 294912 totalling 3.38MiB
2019-07-21 13:06:03.574696: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 524288 totalling 3.00MiB
2019-07-21 13:06:03.574713: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 589824 totalling 6.19MiB
2019-07-21 13:06:03.574736: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 1179648 totalling 13.50MiB
2019-07-21 13:06:03.574754: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 2097152 totalling 12.00MiB
2019-07-21 13:06:03.574773: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 2359296 totalling 24.75MiB
2019-07-21 13:06:03.574792: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 4718592 totalling 54.00MiB
2019-07-21 13:06:03.574811: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 8 Chunks of size 8388608 totalling 64.00MiB
2019-07-21 13:06:03.574836: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 9437184 totalling 99.00MiB
2019-07-21 13:06:03.574854: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16777216 totalling 16.00MiB
2019-07-21 13:06:03.574874: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 18874368 totalling 216.00MiB
2019-07-21 13:06:03.574897: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 4 Chunks of size 33554432 totalling 128.00MiB
2019-07-21 13:06:03.574916: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 37748736 totalling 216.00MiB
2019-07-21 13:06:03.574939: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 52428800 totalling 50.00MiB
2019-07-21 13:06:03.574962: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 8 Chunks of size 67108864 totalling 512.00MiB
2019-07-21 13:06:03.574981: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 101842944 totalling 97.12MiB
2019-07-21 13:06:03.575006: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 134217728 totalling 896.00MiB
2019-07-21 13:06:03.575032: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 200149248 totalling 190.88MiB
2019-07-21 13:06:03.575051: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 268435456 totalling 1.75GiB
2019-07-21 13:06:03.575070: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 536870912 totalling 3.50GiB
2019-07-21 13:06:03.575089: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1073741824 totalling 2.00GiB
2019-07-21 13:06:03.575112: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 9.79GiB
2019-07-21 13:06:03.575134: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats: 
Limit:                 11281927373
InUse:                 10515600128
MaxInUse:              10649965312
NumAllocs:                     560
MaxAllocSize:           2709569536

2019-07-21 13:06:03.575219: W tensorflow/core/common_runtime/bfc_allocator.cc:279] **********************************************__***************************************************_
2019-07-21 13:06:03.575268: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at conv_ops.cc:693 : Resource exhausted: OOM when allocating tensor with shape[32,64,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "main.py", line 32, in <module>
    run.train()
  File "/home/dadams2/unet-sem/model.py", line 113, in train
    loss = self.model.train_on_batch(aug_imgs,aug_masks)
  File "/home/dadams2/.conda/envs/unet/lib/python3.5/site-packages/keras/engine/training.py", line 1215, in train_on_batch
    outputs = self.train_function(ins)
  File "/home/dadams2/.conda/envs/unet/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2666, in __call__
    return self._call(inputs)
  File "/home/dadams2/.conda/envs/unet/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2636, in _call
    fetched = self._callable_fn(*array_vals)
  File "/home/dadams2/.conda/envs/unet/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1382, in __call__
    run_metadata_ptr)
  File "/home/dadams2/.conda/envs/unet/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,64,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[Node: conv2d_22/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@training/Adam/gradients/conv2d_22/convolution_grad/Conv2DBackpropInput"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_21/Relu, conv2d_22/kernel/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: metrics/acc/Mean_1/_429 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3038_metrics/acc/Mean_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(unet) [dadams2@gpu008 unet-sem]$ exit
srun: error: gpu008: task 0: Exited with exit code 1
[dadams2@login005 unet-sem]$ interact -p GPU --gres=gpu:p100:2

A command prompt will appear when your session begins
"Ctrl+d" or "exit" will end your session

srun: job 6067144 queued and waiting for resources
^Csrun: Job allocation 6067144 has been revoked
srun: Force Terminated job 6067144
[dadams2@login005 unet-sem]$ git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#   modified:   utils.py
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#   __pycache__/
#   data/
no changes added to commit (use "git add" and/or "git commit -a")
[dadams2@login005 unet-sem]$ git add utils.py 
[dadams2@login005 unet-sem]$ git commit -m "fixes #1"
[master c6c1740] fixes #1
 Committer: Darren Adams <dadams2@br005.pvt.bridges.psc.edu>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:

    git config --global user.name "Your Name"
    git config --global user.email you@example.com

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 1 file changed, 2 insertions(+), 2 deletions(-)
[dadams2@login005 unet-sem]$ git config --global user.name "Darren Adams"
[dadams2@login005 unet-sem]$ git config --global user.email dadams@illinois.edu
[dadams2@login005 unet-sem]$ git commit --amend --reset-author
[master 5cf3d60] fixes #1
 1 file changed, 2 insertions(+), 2 deletions(-)
[dadams2@login005 unet-sem]$ git push
warning: push.default is unset; its implicit value is changing in
Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the current behavior after the default changes, use:

  git config --global push.default matching

To squelch this message and adopt the new behavior now, use:

  git config --global push.default simple

See 'git help config' and search for 'push.default' for further information.
(the 'simple' mode was introduced in Git 1.7.11. Use the similar mode
'current' instead of 'simple' if you sometimes use older versions of Git)

Counting objects: 5, done.
Delta compression using up to 28 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 294 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To git@github.com:nanoMFG/unet-sem.git
   abe3dee..5cf3d60  master -> master
[dadams2@login005 unet-sem]$ interact -p GPU --gres=gpu:k80:4

A command prompt will appear when your session begins
"Ctrl+d" or "exit" will end your session

[dadams2@gpu012 unet-sem]$ module load anaconda3
[dadams2@gpu012 unet-sem]$ source activate unet
(unet) [dadams2@gpu012 unet-sem]$ python main.py 
Using TensorFlow backend.
{'input_size': (256, 256), 'batch_size': 32, 'split': 0.1, 'max_crop': False, 'crop': True, 'lr': 0.0001, 'ngpu': 1, 'crop_size': (256, 256), 'augment': True, 'nepochs': 5}
/home/dadams2/unet-sem/model.py:58: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("co..., inputs=Tensor("in...)`
  model = Model(input = inputs, output = conv10)
2019-07-21 13:11:29.826699: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-07-21 13:11:30.139306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:83:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-07-21 13:11:30.336060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 1 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:84:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-07-21 13:11:30.529036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 2 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:8a:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-07-21 13:11:30.726517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 3 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:8b:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-07-21 13:11:30.728333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1, 2, 3
2019-07-21 13:11:32.877048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-21 13:11:32.877111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1 2 3 
2019-07-21 13:11:32.877125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N Y Y Y 
2019-07-21 13:11:32.877135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   Y N Y Y 
2019-07-21 13:11:32.877144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 2:   Y Y N Y 
2019-07-21 13:11:32.877152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 3:   Y Y Y N 
2019-07-21 13:11:32.878632: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10759 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:83:00.0, compute capability: 3.7)
2019-07-21 13:11:33.037580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10759 MB memory) -> physical GPU (device: 1, name: Tesla K80, pci bus id: 0000:84:00.0, compute capability: 3.7)
2019-07-21 13:11:33.233135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10759 MB memory) -> physical GPU (device: 2, name: Tesla K80, pci bus id: 0000:8a:00.0, compute capability: 3.7)
2019-07-21 13:11:33.428766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10759 MB memory) -> physical GPU (device: 3, name: Tesla K80, pci bus id: 0000:8b:00.0, compute capability: 3.7)
2019-07-21 13:11:45.683327: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.00GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-21 13:11:46.713263: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.38GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-21 13:11:56.714689: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 512.00MiB.  Current allocation summary follows.
2019-07-21 13:11:56.714748: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256):   Total Chunks: 135, Chunks in use: 135. 33.8KiB allocated for chunks. 33.8KiB in use in bin. 7.1KiB client-requested in use in bin.
2019-07-21 13:11:56.714773: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512):   Total Chunks: 26, Chunks in use: 26. 13.0KiB allocated for chunks. 13.0KiB in use in bin. 13.0KiB client-requested in use in bin.
2019-07-21 13:11:56.714791: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024):  Total Chunks: 27, Chunks in use: 27. 27.2KiB allocated for chunks. 27.2KiB in use in bin. 27.0KiB client-requested in use in bin.
2019-07-21 13:11:56.714808: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048):  Total Chunks: 32, Chunks in use: 32. 65.5KiB allocated for chunks. 65.5KiB in use in bin. 65.5KiB client-requested in use in bin.
2019-07-21 13:11:56.714831: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096):  Total Chunks: 17, Chunks in use: 17. 71.0KiB allocated for chunks. 71.0KiB in use in bin. 71.0KiB client-requested in use in bin.
2019-07-21 13:11:56.714849: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 13:11:56.714865: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 13:11:56.714881: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 13:11:56.714895: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 13:11:56.715015: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072):    Total Chunks: 18, Chunks in use: 18. 2.44MiB allocated for chunks. 2.44MiB in use in bin. 2.44MiB client-requested in use in bin.
2019-07-21 13:11:56.715047: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144):    Total Chunks: 12, Chunks in use: 12. 3.38MiB allocated for chunks. 3.38MiB in use in bin. 3.38MiB client-requested in use in bin.
2019-07-21 13:11:56.715068: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288):    Total Chunks: 17, Chunks in use: 17. 9.19MiB allocated for chunks. 9.19MiB in use in bin. 9.19MiB client-requested in use in bin.
2019-07-21 13:11:56.715104: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576):   Total Chunks: 12, Chunks in use: 12. 13.50MiB allocated for chunks. 13.50MiB in use in bin. 13.50MiB client-requested in use in bin.
2019-07-21 13:11:56.715124: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152):   Total Chunks: 18, Chunks in use: 17. 39.00MiB allocated for chunks. 36.75MiB in use in bin. 36.75MiB client-requested in use in bin.
2019-07-21 13:11:56.715161: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304):   Total Chunks: 13, Chunks in use: 12. 58.50MiB allocated for chunks. 54.00MiB in use in bin. 54.00MiB client-requested in use in bin.
2019-07-21 13:11:56.715196: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608):   Total Chunks: 20, Chunks in use: 19. 172.00MiB allocated for chunks. 163.00MiB in use in bin. 163.00MiB client-requested in use in bin.
2019-07-21 13:11:56.715227: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216):  Total Chunks: 13, Chunks in use: 13. 232.00MiB allocated for chunks. 232.00MiB in use in bin. 232.00MiB client-requested in use in bin.
2019-07-21 13:11:56.715245: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432):  Total Chunks: 13, Chunks in use: 11. 462.00MiB allocated for chunks. 394.00MiB in use in bin. 376.00MiB client-requested in use in bin.
2019-07-21 13:11:56.715265: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864):  Total Chunks: 10, Chunks in use: 9. 673.12MiB allocated for chunks. 609.12MiB in use in bin. 576.00MiB client-requested in use in bin.
2019-07-21 13:11:56.715293: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728):     Total Chunks: 10, Chunks in use: 8. 1.38GiB allocated for chunks. 1.06GiB in use in bin. 1.00GiB client-requested in use in bin.
2019-07-21 13:11:56.715653: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456):     Total Chunks: 17, Chunks in use: 16. 7.50GiB allocated for chunks. 7.25GiB in use in bin. 7.25GiB client-requested in use in bin.
2019-07-21 13:11:56.715692: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 512.00MiB was 256.00MiB, Chunk State: 
2019-07-21 13:11:56.715725: I tensorflow/core/common_runtime/bfc_allocator.cc:652]   Size: 256.00MiB | Requested Size: 256.00MiB | in_use: 0, prev:   Size: 128.00MiB | Requested Size: 128.00MiB | in_use: 1, next:   Size: 256.00MiB | Requested Size: 256.00MiB | in_use: 1
2019-07-21 13:11:56.715760: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00000 of size 1280
2019-07-21 13:11:56.715775: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00500 of size 256
2019-07-21 13:11:56.715786: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00600 of size 256
2019-07-21 13:11:56.715815: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00700 of size 256
2019-07-21 13:11:56.715833: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00800 of size 256
2019-07-21 13:11:56.715850: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00900 of size 256
2019-07-21 13:11:56.715863: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00a00 of size 256
2019-07-21 13:11:56.715876: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00b00 of size 256
2019-07-21 13:11:56.715897: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00c00 of size 256
2019-07-21 13:11:56.715911: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00d00 of size 256
2019-07-21 13:11:56.715933: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00e00 of size 256
2019-07-21 13:11:56.715955: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb00f00 of size 256
2019-07-21 13:11:56.715974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb01000 of size 256
2019-07-21 13:11:56.715991: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb01100 of size 256
2019-07-21 13:11:56.716004: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb01200 of size 256
2019-07-21 13:11:56.716017: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb01300 of size 256
2019-07-21 13:11:56.716036: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb01400 of size 256
2019-07-21 13:11:56.716049: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb01500 of size 256
2019-07-21 13:11:56.716070: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb01600 of size 2304
2019-07-21 13:11:56.716091: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb01f00 of size 294912
2019-07-21 13:11:56.716105: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230fb49f00 of size 4718592
2019-07-21 13:11:56.716118: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x230ffc9f00 of size 18874368
2019-07-21 13:11:56.716131: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23111c9f00 of size 37748736
2019-07-21 13:11:56.716156: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23135c9f00 of size 4096
2019-07-21 13:11:56.716170: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23135caf00 of size 8388608
2019-07-21 13:11:56.716182: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2313dcaf00 of size 18874368
2019-07-21 13:11:56.716203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314fcaf00 of size 9437184
2019-07-21 13:11:56.716218: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23158caf00 of size 2048
2019-07-21 13:11:56.716239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23158cb700 of size 2097152
2019-07-21 13:11:56.716253: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315acb700 of size 4718592
2019-07-21 13:11:56.716275: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315f4b700 of size 2359296
2019-07-21 13:11:56.716289: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231618b700 of size 1179648
2019-07-21 13:11:56.716305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23162ab700 of size 524288
2019-07-21 13:11:56.716331: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231632b700 of size 1179648
2019-07-21 13:11:56.716357: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231644b700 of size 589824
2019-07-21 13:11:56.716380: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23164db700 of size 512
2019-07-21 13:11:56.716395: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23164db900 of size 131072
2019-07-21 13:11:56.716412: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23164fb900 of size 294912
2019-07-21 13:11:56.716433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316543900 of size 1024
2019-07-21 13:11:56.716456: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316543d00 of size 147456
2019-07-21 13:11:56.716470: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316567d00 of size 256
2019-07-21 13:11:56.716490: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316567e00 of size 4608
2019-07-21 13:11:56.716504: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316569000 of size 256
2019-07-21 13:11:56.716521: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316569100 of size 256
2019-07-21 13:11:56.716535: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316569200 of size 256
2019-07-21 13:11:56.716552: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316569300 of size 2304
2019-07-21 13:11:56.716565: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316569c00 of size 294912
2019-07-21 13:11:56.716586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23165b1c00 of size 294912
2019-07-21 13:11:56.716601: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23165f9c00 of size 4718592
2019-07-21 13:11:56.716623: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316a79c00 of size 18874368
2019-07-21 13:11:56.716636: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317c79c00 of size 18874368
2019-07-21 13:11:56.716658: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318e79c00 of size 37748736
2019-07-21 13:11:56.716674: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b279c00 of size 37748736
2019-07-21 13:11:56.716689: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d679c00 of size 4096
2019-07-21 13:11:56.716712: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d67ac00 of size 4096
2019-07-21 13:11:56.716732: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d67bc00 of size 4096
2019-07-21 13:11:56.716749: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d67cc00 of size 4096
2019-07-21 13:11:56.716762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d67dc00 of size 8388608
2019-07-21 13:11:56.716776: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231de7dc00 of size 8388608
2019-07-21 13:11:56.716793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231e67dc00 of size 18874368
2019-07-21 13:11:56.716816: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231f87dc00 of size 18874368
2019-07-21 13:11:56.716834: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320a7dc00 of size 9437184
2019-07-21 13:11:56.716849: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232137dc00 of size 9437184
2019-07-21 13:11:56.716864: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321c7dc00 of size 9437184
2019-07-21 13:11:56.716879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232257dc00 of size 9437184
2019-07-21 13:11:56.716897: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e7dc00 of size 2048
2019-07-21 13:11:56.716913: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e7e400 of size 2048
2019-07-21 13:11:56.716929: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e7ec00 of size 2048
2019-07-21 13:11:56.716947: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e7f400 of size 2048
2019-07-21 13:11:56.716965: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e7fc00 of size 2048
2019-07-21 13:11:56.716979: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e80400 of size 2048
2019-07-21 13:11:56.717001: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e80c00 of size 2048
2019-07-21 13:11:56.717018: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e81400 of size 2048
2019-07-21 13:11:56.717035: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e81c00 of size 2048
2019-07-21 13:11:56.717052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e82400 of size 2048
2019-07-21 13:11:56.717065: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322e82c00 of size 2097152
2019-07-21 13:11:56.717081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323082c00 of size 2097152
2019-07-21 13:11:56.717096: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323282c00 of size 4718592
2019-07-21 13:11:56.717110: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323702c00 of size 4718592
2019-07-21 13:11:56.717128: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323b82c00 of size 2359296
2019-07-21 13:11:56.717147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323dc2c00 of size 2359296
2019-07-21 13:11:56.717165: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324002c00 of size 2359296
2019-07-21 13:11:56.717178: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324242c00 of size 1179648
2019-07-21 13:11:56.717200: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324362c00 of size 1179648
2019-07-21 13:11:56.717219: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324482c00 of size 524288
2019-07-21 13:11:56.717236: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324502c00 of size 524288
2019-07-21 13:11:56.717252: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324582c00 of size 1179648
2019-07-21 13:11:56.717265: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23246a2c00 of size 1179648
2019-07-21 13:11:56.717283: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23247c2c00 of size 589824
2019-07-21 13:11:56.717297: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324852c00 of size 589824
2019-07-21 13:11:56.717314: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23248e2c00 of size 589824
2019-07-21 13:11:56.717334: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324972c00 of size 589824
2019-07-21 13:11:56.717353: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a02c00 of size 512
2019-07-21 13:11:56.717370: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a02e00 of size 512
2019-07-21 13:11:56.717383: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a03000 of size 512
2019-07-21 13:11:56.717402: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a03200 of size 512
2019-07-21 13:11:56.717419: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a03400 of size 512
2019-07-21 13:11:56.717437: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a03600 of size 512
2019-07-21 13:11:56.717450: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a03800 of size 512
2019-07-21 13:11:56.717468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a03a00 of size 512
2019-07-21 13:11:56.717487: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a03c00 of size 512
2019-07-21 13:11:56.717500: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a03e00 of size 512
2019-07-21 13:11:56.717518: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a04000 of size 131072
2019-07-21 13:11:56.717534: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a24000 of size 131072
2019-07-21 13:11:56.717551: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a44000 of size 294912
2019-07-21 13:11:56.717571: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324a8c000 of size 294912
2019-07-21 13:11:56.717584: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324ad4000 of size 1024
2019-07-21 13:11:56.717601: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324ad4400 of size 1024
2019-07-21 13:11:56.717619: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324ad4800 of size 1024
2019-07-21 13:11:56.717632: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324ad4c00 of size 1024
2019-07-21 13:11:56.717649: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324ad5000 of size 1024
2019-07-21 13:11:56.717664: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324ad5400 of size 1024
2019-07-21 13:11:56.717681: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324ad5800 of size 1024
2019-07-21 13:11:56.717696: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324ad5c00 of size 1024
2019-07-21 13:11:56.717713: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324ad6000 of size 1024
2019-07-21 13:11:56.717731: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324ad6400 of size 147456
2019-07-21 13:11:56.717746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324afa400 of size 147456
2019-07-21 13:11:56.717763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b1e400 of size 147456
2019-07-21 13:11:56.717779: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b42400 of size 147456
2019-07-21 13:11:56.717798: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b66400 of size 256
2019-07-21 13:11:56.717810: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b66500 of size 256
2019-07-21 13:11:56.717827: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b66600 of size 256
2019-07-21 13:11:56.717842: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b66700 of size 256
2019-07-21 13:11:56.717860: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b66800 of size 256
2019-07-21 13:11:56.717874: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b66900 of size 256
2019-07-21 13:11:56.717887: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b66a00 of size 256
2019-07-21 13:11:56.717900: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b66b00 of size 256
2019-07-21 13:11:56.717922: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b66c00 of size 256
2019-07-21 13:11:56.717936: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b66d00 of size 4608
2019-07-21 13:11:56.717957: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b67f00 of size 4608
2019-07-21 13:11:56.717976: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69100 of size 256
2019-07-21 13:11:56.717994: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69200 of size 256
2019-07-21 13:11:56.718011: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69300 of size 256
2019-07-21 13:11:56.718024: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69400 of size 256
2019-07-21 13:11:56.718037: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69500 of size 256
2019-07-21 13:11:56.718057: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69600 of size 256
2019-07-21 13:11:56.718071: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69700 of size 256
2019-07-21 13:11:56.718088: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69800 of size 256
2019-07-21 13:11:56.718109: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69900 of size 256
2019-07-21 13:11:56.718124: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69a00 of size 256
2019-07-21 13:11:56.718142: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69b00 of size 256
2019-07-21 13:11:56.718162: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69c00 of size 256
2019-07-21 13:11:56.718180: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69d00 of size 256
2019-07-21 13:11:56.718193: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69e00 of size 256
2019-07-21 13:11:56.718209: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b69f00 of size 256
2019-07-21 13:11:56.718223: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6a000 of size 256
2019-07-21 13:11:56.718239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6a100 of size 256
2019-07-21 13:11:56.718252: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6a200 of size 256
2019-07-21 13:11:56.718268: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6a300 of size 256
2019-07-21 13:11:56.718281: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6a400 of size 256
2019-07-21 13:11:56.718298: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6a500 of size 256
2019-07-21 13:11:56.718310: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6a600 of size 256
2019-07-21 13:11:56.718332: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6a700 of size 256
2019-07-21 13:11:56.718347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6a800 of size 256
2019-07-21 13:11:56.718365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6a900 of size 256
2019-07-21 13:11:56.718383: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6aa00 of size 256
2019-07-21 13:11:56.718401: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6ab00 of size 256
2019-07-21 13:11:56.718420: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6ac00 of size 256
2019-07-21 13:11:56.718432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6ad00 of size 256
2019-07-21 13:11:56.718455: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6ae00 of size 256
2019-07-21 13:11:56.718473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6af00 of size 256
2019-07-21 13:11:56.718486: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6b000 of size 256
2019-07-21 13:11:56.718509: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6b100 of size 256
2019-07-21 13:11:56.718523: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6b200 of size 256
2019-07-21 13:11:56.718536: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6b300 of size 2304
2019-07-21 13:11:56.718549: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6bc00 of size 4096
2019-07-21 13:11:56.718562: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b6cc00 of size 37748736
2019-07-21 13:11:56.718581: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f6cc00 of size 2048
2019-07-21 13:11:56.718594: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f6d400 of size 8388608
2019-07-21 13:11:56.718607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232776d400 of size 2048
2019-07-21 13:11:56.718620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232776dc00 of size 18874368
2019-07-21 13:11:56.718641: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232896dc00 of size 2048
2019-07-21 13:11:56.718654: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232896e400 of size 1024
2019-07-21 13:11:56.718671: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232896e800 of size 2097152
2019-07-21 13:11:56.718687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328b6e800 of size 1024
2019-07-21 13:11:56.718700: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328b6ec00 of size 4718592
2019-07-21 13:11:56.718714: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328feec00 of size 1024
2019-07-21 13:11:56.718728: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328fef000 of size 512
2019-07-21 13:11:56.718745: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328fef200 of size 524288
2019-07-21 13:11:56.718764: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232906f200 of size 512
2019-07-21 13:11:56.718781: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232906f400 of size 1179648
2019-07-21 13:11:56.718793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232918f400 of size 512
2019-07-21 13:11:56.718810: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232918f600 of size 256
2019-07-21 13:11:56.718823: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232918f700 of size 256
2019-07-21 13:11:56.718837: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232918f800 of size 131072
2019-07-21 13:11:56.718855: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23291af800 of size 256
2019-07-21 13:11:56.718873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23291af900 of size 294912
2019-07-21 13:11:56.718886: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23291f7900 of size 256
2019-07-21 13:11:56.718899: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23291f7a00 of size 147456
2019-07-21 13:11:56.718912: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232921ba00 of size 147456
2019-07-21 13:11:56.718935: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232923fa00 of size 256
2019-07-21 13:11:56.718953: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232923fb00 of size 4608
2019-07-21 13:11:56.718968: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329240d00 of size 256
2019-07-21 13:11:56.718989: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329240e00 of size 256
2019-07-21 13:11:56.719002: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329240f00 of size 512
2019-07-21 13:11:56.719016: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329241100 of size 294912
2019-07-21 13:11:56.719031: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329289100 of size 512
2019-07-21 13:11:56.719052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329289300 of size 589824
2019-07-21 13:11:56.719065: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329319300 of size 589824
2019-07-21 13:11:56.719079: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23293a9300 of size 1024
2019-07-21 13:11:56.719099: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23293a9700 of size 1179648
2019-07-21 13:11:56.719113: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23294c9700 of size 1024
2019-07-21 13:11:56.719127: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23294c9b00 of size 2359296
2019-07-21 13:11:56.719145: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329709b00 of size 2359296
2019-07-21 13:11:56.719166: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329949b00 of size 2048
2019-07-21 13:11:56.719179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232994a300 of size 4718592
2019-07-21 13:11:56.719196: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329dca300 of size 2048
2019-07-21 13:11:56.719217: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329dcab00 of size 9437184
2019-07-21 13:11:56.719231: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232a6cab00 of size 9437184
2019-07-21 13:11:56.719248: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232afcab00 of size 4096
2019-07-21 13:11:56.719265: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232afcbb00 of size 18874368
2019-07-21 13:11:56.719278: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c1cbb00 of size 2304
2019-07-21 13:11:56.719295: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c1cc400 of size 256
2019-07-21 13:11:56.719308: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c1cc500 of size 2359296
2019-07-21 13:11:56.719330: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40c500 of size 256
2019-07-21 13:11:56.719344: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40c600 of size 256
2019-07-21 13:11:56.719361: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40c700 of size 256
2019-07-21 13:11:56.719377: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40c800 of size 256
2019-07-21 13:11:56.719393: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40c900 of size 256
2019-07-21 13:11:56.719410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40ca00 of size 256
2019-07-21 13:11:56.719428: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40cb00 of size 256
2019-07-21 13:11:56.719445: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40cc00 of size 256
2019-07-21 13:11:56.719462: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40cd00 of size 256
2019-07-21 13:11:56.719474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40ce00 of size 256
2019-07-21 13:11:56.719487: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40cf00 of size 1024
2019-07-21 13:11:56.719508: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40d300 of size 256
2019-07-21 13:11:56.719527: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40d400 of size 256
2019-07-21 13:11:56.719540: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40d500 of size 256
2019-07-21 13:11:56.719560: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40d600 of size 256
2019-07-21 13:11:56.719573: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40d700 of size 256
2019-07-21 13:11:56.719586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40d800 of size 256
2019-07-21 13:11:56.719602: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40d900 of size 256
2019-07-21 13:11:56.719616: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40da00 of size 256
2019-07-21 13:11:56.719632: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40db00 of size 256
2019-07-21 13:11:56.719650: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40dc00 of size 256
2019-07-21 13:11:56.719667: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c40dd00 of size 4718592
2019-07-21 13:11:56.719682: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88dd00 of size 256
2019-07-21 13:11:56.719700: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88de00 of size 256
2019-07-21 13:11:56.719722: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88df00 of size 256
2019-07-21 13:11:56.719735: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88e000 of size 256
2019-07-21 13:11:56.719750: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88e100 of size 256
2019-07-21 13:11:56.719767: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88e200 of size 256
2019-07-21 13:11:56.719788: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88e300 of size 256
2019-07-21 13:11:56.719803: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88e400 of size 256
2019-07-21 13:11:56.719820: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88e500 of size 256
2019-07-21 13:11:56.719836: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88e600 of size 256
2019-07-21 13:11:56.719850: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88e700 of size 256
2019-07-21 13:11:56.719864: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88e800 of size 256
2019-07-21 13:11:56.719879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88e900 of size 256
2019-07-21 13:11:56.719900: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88ea00 of size 256
2019-07-21 13:11:56.719920: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88eb00 of size 256
2019-07-21 13:11:56.719934: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88ec00 of size 256
2019-07-21 13:11:56.719952: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88ed00 of size 256
2019-07-21 13:11:56.719970: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88ee00 of size 256
2019-07-21 13:11:56.719988: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88ef00 of size 256
2019-07-21 13:11:56.720003: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88f000 of size 256
2019-07-21 13:11:56.720025: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88f100 of size 256
2019-07-21 13:11:56.720039: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232c88f200 of size 8388608
2019-07-21 13:11:56.720052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232d08f200 of size 8388608
2019-07-21 13:11:56.720068: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232d88f200 of size 256
2019-07-21 13:11:56.720081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232d88f300 of size 256
2019-07-21 13:11:56.720095: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232d88f400 of size 256
2019-07-21 13:11:56.720114: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232d88f500 of size 134217728
2019-07-21 13:11:56.720133: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233588f500 of size 67108864
2019-07-21 13:11:56.720152: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233988f500 of size 33554432
2019-07-21 13:11:56.720168: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233b88f500 of size 101842944
2019-07-21 13:11:56.720190: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23419af500 of size 200149504
2019-07-21 13:11:56.720207: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234d88ff00 of size 2359296
2019-07-21 13:11:56.720221: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234dacff00 of size 536870912
2019-07-21 13:11:56.720239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236dacff00 of size 1024
2019-07-21 13:11:56.720257: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236dad0300 of size 524288
2019-07-21 13:11:56.720271: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236db50300 of size 256
2019-07-21 13:11:56.720286: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236db50400 of size 2359296
2019-07-21 13:11:56.720307: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236dd90400 of size 1024
2019-07-21 13:11:56.720326: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236dd90800 of size 524288
2019-07-21 13:11:56.720341: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236de10800 of size 512
2019-07-21 13:11:56.720354: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236de10a00 of size 256
2019-07-21 13:11:56.720373: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236de10b00 of size 512
2019-07-21 13:11:56.720387: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236de10d00 of size 1179648
2019-07-21 13:11:56.720404: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236df30d00 of size 1179648
2019-07-21 13:11:56.720417: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e050d00 of size 512
2019-07-21 13:11:56.720433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e050f00 of size 512
2019-07-21 13:11:56.720446: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e051100 of size 256
2019-07-21 13:11:56.720459: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e051200 of size 589824
2019-07-21 13:11:56.720478: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e0e1200 of size 589824
2019-07-21 13:11:56.720495: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e171200 of size 512
2019-07-21 13:11:56.720509: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e171400 of size 512
2019-07-21 13:11:56.720523: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e171600 of size 131072
2019-07-21 13:11:56.720542: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e191600 of size 131072
2019-07-21 13:11:56.720556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e1b1600 of size 256
2019-07-21 13:11:56.720569: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e1b1700 of size 256
2019-07-21 13:11:56.720582: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e1b1800 of size 294912
2019-07-21 13:11:56.720595: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e1f9800 of size 294912
2019-07-21 13:11:56.720616: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e241800 of size 256
2019-07-21 13:11:56.720634: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e241900 of size 256
2019-07-21 13:11:56.720649: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e241a00 of size 147456
2019-07-21 13:11:56.720665: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e265a00 of size 147456
2019-07-21 13:11:56.720679: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e289a00 of size 256
2019-07-21 13:11:56.720692: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e289b00 of size 256
2019-07-21 13:11:56.720713: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e289c00 of size 4608
2019-07-21 13:11:56.720731: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e28ae00 of size 4608
2019-07-21 13:11:56.720744: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e28c000 of size 256
2019-07-21 13:11:56.720761: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e28c100 of size 256
2019-07-21 13:11:56.720778: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e28c200 of size 294912
2019-07-21 13:11:56.720797: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e2d4200 of size 294912
2019-07-21 13:11:56.720811: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e31c200 of size 256
2019-07-21 13:11:56.720828: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e31c300 of size 256
2019-07-21 13:11:56.720842: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e31c400 of size 256
2019-07-21 13:11:56.720855: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e31c500 of size 256
2019-07-21 13:11:56.720869: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e31c600 of size 512
2019-07-21 13:11:56.720885: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e31c800 of size 512
2019-07-21 13:11:56.720900: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e31ca00 of size 2304
2019-07-21 13:11:56.720914: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e31d300 of size 2304
2019-07-21 13:11:56.720932: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e31dc00 of size 589824
2019-07-21 13:11:56.720948: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e3adc00 of size 589824
2019-07-21 13:11:56.720962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e43dc00 of size 512
2019-07-21 13:11:56.720976: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e43de00 of size 512
2019-07-21 13:11:56.720994: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e43e000 of size 1179648
2019-07-21 13:11:56.721016: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e55e000 of size 1179648
2019-07-21 13:11:56.721030: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e67e000 of size 1024
2019-07-21 13:11:56.721044: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e67e400 of size 1024
2019-07-21 13:11:56.721065: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e67e800 of size 2359296
2019-07-21 13:11:56.721080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236e8be800 of size 2359296
2019-07-21 13:11:56.721092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236eafe800 of size 1024
2019-07-21 13:11:56.721105: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236eafec00 of size 1024
2019-07-21 13:11:56.721125: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236eaff000 of size 4718592
2019-07-21 13:11:56.721144: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236ef7f000 of size 4718592
2019-07-21 13:11:56.721157: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236f3ff000 of size 2048
2019-07-21 13:11:56.721172: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236f3ff800 of size 2048
2019-07-21 13:11:56.721188: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236f400000 of size 9437184
2019-07-21 13:11:56.721202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236fd00000 of size 9437184
2019-07-21 13:11:56.721217: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370600000 of size 2048
2019-07-21 13:11:56.721230: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370600800 of size 2048
2019-07-21 13:11:56.721243: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370601000 of size 256
2019-07-21 13:11:56.721266: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370601100 of size 256
2019-07-21 13:11:56.721280: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370601200 of size 18874368
2019-07-21 13:11:56.721293: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2371801200 of size 18874368
2019-07-21 13:11:56.721310: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372a01200 of size 4096
2019-07-21 13:11:56.721332: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372a02200 of size 4096
2019-07-21 13:11:56.721347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372a03200 of size 37748736
2019-07-21 13:11:56.721364: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2374e03200 of size 37748736
2019-07-21 13:11:56.721384: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2377203200 of size 4096
2019-07-21 13:11:56.721401: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2377204200 of size 4096
2019-07-21 13:11:56.721414: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2377205200 of size 8388608
2019-07-21 13:11:56.721430: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2377a05200 of size 8388608
2019-07-21 13:11:56.721445: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378205200 of size 2048
2019-07-21 13:11:56.721461: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378205a00 of size 2048
2019-07-21 13:11:56.721474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378206200 of size 18874368
2019-07-21 13:11:56.721495: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2379406200 of size 18874368
2019-07-21 13:11:56.721514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237a606200 of size 2048
2019-07-21 13:11:56.721528: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237a606a00 of size 2048
2019-07-21 13:11:56.721550: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237a607200 of size 9437184
2019-07-21 13:11:56.721568: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237af07200 of size 9437184
2019-07-21 13:11:56.721582: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237b807200 of size 2048
2019-07-21 13:11:56.721603: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237b807a00 of size 2048
2019-07-21 13:11:56.721621: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237b808200 of size 147456
2019-07-21 13:11:56.721641: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237b82c200 of size 147456
2019-07-21 13:11:56.721659: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237b850200 of size 2097152
2019-07-21 13:11:56.721672: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237ba50200 of size 2097152
2019-07-21 13:11:56.721693: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237bc50200 of size 1024
2019-07-21 13:11:56.721710: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237bc50600 of size 1024
2019-07-21 13:11:56.721723: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237bc50a00 of size 4718592
2019-07-21 13:11:56.721741: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c0d0a00 of size 4718592
2019-07-21 13:11:56.721759: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c550a00 of size 1024
2019-07-21 13:11:56.721772: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c550e00 of size 1024
2019-07-21 13:11:56.721789: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c551200 of size 256
2019-07-21 13:11:56.721810: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c551300 of size 256
2019-07-21 13:11:56.721824: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c551400 of size 147456
2019-07-21 13:11:56.721841: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c575400 of size 536870912
2019-07-21 13:11:56.721863: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239c575400 of size 268435456
2019-07-21 13:11:56.721877: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ac575400 of size 268435456
2019-07-21 13:11:56.721890: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23bc575400 of size 2359296
2019-07-21 13:11:56.721911: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23bc7b5400 of size 134217728
2019-07-21 13:11:56.721924: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23c47b5400 of size 4718592
2019-07-21 13:11:56.721938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23c4c35400 of size 67108864
2019-07-21 13:11:56.721955: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23c8c35400 of size 9437184
2019-07-21 13:11:56.721971: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23c9535400 of size 67108864
2019-07-21 13:11:56.721989: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23cd535400 of size 67108864
2019-07-21 13:11:56.722002: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d1535400 of size 67108864
2019-07-21 13:11:56.722020: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d5535400 of size 16777216
2019-07-21 13:11:56.722038: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d6535400 of size 52428800
2019-07-21 13:11:56.722051: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d9735400 of size 33554432
2019-07-21 13:11:56.722068: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23db735400 of size 37748736
2019-07-21 13:11:56.722081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ddb35400 of size 33554432
2019-07-21 13:11:56.722095: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23dfb35400 of size 33554432
2019-07-21 13:11:56.722109: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e1b35400 of size 33554432
2019-07-21 13:11:56.722129: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23e3b35400 of size 67108864
2019-07-21 13:11:56.722143: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e7b35400 of size 67108864
2019-07-21 13:11:56.722158: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ebb35400 of size 134217728
2019-07-21 13:11:56.722181: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f3b35400 of size 134217728
2019-07-21 13:11:56.722200: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23fbb35400 of size 67108864
2019-07-21 13:11:56.722213: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ffb35400 of size 67108864
2019-07-21 13:11:56.722230: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x2403b35400 of size 134217728
2019-07-21 13:11:56.722246: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x240bb35400 of size 134217728
2019-07-21 13:11:56.722260: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2413b35400 of size 268435456
2019-07-21 13:11:56.722277: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2423b35400 of size 268435456
2019-07-21 13:11:56.722295: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2433b35400 of size 134217728
2019-07-21 13:11:56.722312: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x243bb35400 of size 134217728
2019-07-21 13:11:56.722332: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x2443b35400 of size 268435456
2019-07-21 13:11:56.722352: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2453b35400 of size 268435456
2019-07-21 13:11:56.722369: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2463b35400 of size 536870912
2019-07-21 13:11:56.722382: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2483b35400 of size 536870912
2019-07-21 13:11:56.722397: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a3b35400 of size 268435456
2019-07-21 13:11:56.722419: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b3b35400 of size 268435456
2019-07-21 13:11:56.722436: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c3b35400 of size 536870912
2019-07-21 13:11:56.722450: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e3b35400 of size 536870912
2019-07-21 13:11:56.722465: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2503b35400 of size 1073741824
2019-07-21 13:11:56.722485: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2543b35400 of size 1073741824
2019-07-21 13:11:56.722503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2583b35400 of size 536870912
2019-07-21 13:11:56.722516: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x25a3b35400 of size 208746496
2019-07-21 13:11:56.722536: I tensorflow/core/common_runtime/bfc_allocator.cc:671]      Summary of in-use Chunks by size: 
2019-07-21 13:11:56.722554: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 135 Chunks of size 256 totalling 33.8KiB
2019-07-21 13:11:56.722569: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 26 Chunks of size 512 totalling 13.0KiB
2019-07-21 13:11:56.722588: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 26 Chunks of size 1024 totalling 26.0KiB
2019-07-21 13:11:56.722603: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.2KiB
2019-07-21 13:11:56.722617: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 26 Chunks of size 2048 totalling 52.0KiB
2019-07-21 13:11:56.722631: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 2304 totalling 13.5KiB
2019-07-21 13:11:56.722655: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 4096 totalling 44.0KiB
2019-07-21 13:11:56.722675: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 4608 totalling 27.0KiB
2019-07-21 13:11:56.722691: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 131072 totalling 768.0KiB
2019-07-21 13:11:56.722715: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 147456 totalling 1.69MiB
2019-07-21 13:11:56.722734: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 294912 totalling 3.38MiB
2019-07-21 13:11:56.722752: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 524288 totalling 3.00MiB
2019-07-21 13:11:56.722767: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 589824 totalling 6.19MiB
2019-07-21 13:11:56.722781: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 1179648 totalling 13.50MiB
2019-07-21 13:11:56.722796: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 2097152 totalling 12.00MiB
2019-07-21 13:11:56.722818: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 2359296 totalling 24.75MiB
2019-07-21 13:11:56.722837: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 4718592 totalling 54.00MiB
2019-07-21 13:11:56.722853: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 8 Chunks of size 8388608 totalling 64.00MiB
2019-07-21 13:11:56.722876: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 9437184 totalling 99.00MiB
2019-07-21 13:11:56.722892: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16777216 totalling 16.00MiB
2019-07-21 13:11:56.722907: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 18874368 totalling 216.00MiB
2019-07-21 13:11:56.722925: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 4 Chunks of size 33554432 totalling 128.00MiB
2019-07-21 13:11:56.722941: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 37748736 totalling 216.00MiB
2019-07-21 13:11:56.722955: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 52428800 totalling 50.00MiB
2019-07-21 13:11:56.722970: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 8 Chunks of size 67108864 totalling 512.00MiB
2019-07-21 13:11:56.722989: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 101842944 totalling 97.12MiB
2019-07-21 13:11:56.723005: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 134217728 totalling 896.00MiB
2019-07-21 13:11:56.723020: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 200149504 totalling 190.88MiB
2019-07-21 13:11:56.723035: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 268435456 totalling 1.75GiB
2019-07-21 13:11:56.723059: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 536870912 totalling 3.50GiB
2019-07-21 13:11:56.723074: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1073741824 totalling 2.00GiB
2019-07-21 13:11:56.723090: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 9.79GiB
2019-07-21 13:11:56.723110: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats: 
Limit:                 11281927373
InUse:                 10515600384
MaxInUse:              10649965568
NumAllocs:                     560
MaxAllocSize:           2709569536

2019-07-21 13:11:56.723181: W tensorflow/core/common_runtime/bfc_allocator.cc:279] **********************************************__***************************************************_
2019-07-21 13:11:56.723221: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at conv_ops.cc:693 : Resource exhausted: OOM when allocating tensor with shape[32,64,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "main.py", line 32, in <module>
    run.train()
  File "/home/dadams2/unet-sem/model.py", line 113, in train
    loss = self.model.train_on_batch(aug_imgs,aug_masks)
  File "/home/dadams2/.conda/envs/unet/lib/python3.5/site-packages/keras/engine/training.py", line 1215, in train_on_batch
    outputs = self.train_function(ins)
  File "/home/dadams2/.conda/envs/unet/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2666, in __call__
    return self._call(inputs)
  File "/home/dadams2/.conda/envs/unet/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2636, in _call
    fetched = self._callable_fn(*array_vals)
  File "/home/dadams2/.conda/envs/unet/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1382, in __call__
    run_metadata_ptr)
  File "/home/dadams2/.conda/envs/unet/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,64,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[Node: conv2d_22/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@training/Adam/gradients/conv2d_22/convolution_grad/Conv2DBackpropInput"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv2d_21/Relu, conv2d_22/kernel/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: metrics/acc/Mean_1/_429 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3038_metrics/acc/Mean_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

[dadamsncsa]

running on multiple gpus seems to fix it