experiencor / keras-yolo2

Easy training on custom dataset. Various backends (MobileNet and SqueezeNet) supported. A YOLO demo to detect raccoon run entirely in brower is accessible at https://git.io/vF7vI (not on Windows).
MIT License
1.73k stars 787 forks source link

Train error. #383

Open mrtortong opened 5 years ago

mrtortong commented 5 years ago

(venv) tong@tong:~/keras-yolo2$ python train.py -c config.json Using TensorFlow backend. Seen labels: {'person with helmet': 1259, 'helmet': 1363, 'person without helmet': 652, 'person': 56} Given labels: ['helmet', 'person with helmet', 'person without helmet'] Overlap labels: {'person with helmet', 'helmet', 'person without helmet'} 2018-11-15 16:59:51.312226: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2018-11-15 16:59:51.401022: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2018-11-15 16:59:51.401391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62 pciBusID: 0000:01:00.0 totalMemory: 3.95GiB freeMemory: 3.42GiB 2018-11-15 16:59:51.401407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2018-11-15 16:59:51.622722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-11-15 16:59:51.622767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2018-11-15 16:59:51.622773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2018-11-15 16:59:51.622922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3133 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1) (13, 13)


Layer (type) Output Shape Param # Connected to

input_1 (InputLayer) (None, 416, 416, 3) 0


model_1 (Model) (None, 13, 13, 1024) 50547936 input_1[0][0]


DetectionLayer (Conv2D) (None, 13, 13, 40) 41000 model_1[1][0]


reshape_1 (Reshape) (None, 13, 13, 5, 8) 0 DetectionLayer[0][0]


input_2 (InputLayer) (None, 1, 1, 1, 10, 4 0


lambda_2 (Lambda) (None, 13, 13, 5, 8) 0 reshape_1[0][0]
input_2[0][0]

Total params: 50,588,936 Trainable params: 50,568,264 Non-trainable params: 20,672


WARNING:tensorflow:From /home/tong/keras-yolo2/frontend.py:232: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20. Instructions for updating: Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:

    sess = tf.Session()
    with sess.as_default():
        tensor = tf.range(10)
        print_op = tf.print(tensor)
        with tf.control_dependencies([print_op]):
          out = tf.add(tensor, tensor)
        sess.run(out)

Additionally, to use tf.print in python 2.7, users must make sure to import the following:

from __future__ import print_function

WARNING:tensorflow:From /home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead Epoch 1/53 2018-11-15 17:00:15.008560: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 72.0KiB. Current allocation summary follows. 2018-11-15 17:00:15.008676: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 264, Chunks in use: 264. 66.0KiB allocated for chunks. 66.0KiB in use in bin. 15.8KiB client-requested in use in bin. 2018-11-15 17:00:15.008711: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 46, Chunks in use: 46. 23.2KiB allocated for chunks. 23.2KiB in use in bin. 23.1KiB client-requested in use in bin. 2018-11-15 17:00:15.008737: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 62, Chunks in use: 62. 63.2KiB allocated for chunks. 63.2KiB in use in bin. 62.5KiB client-requested in use in bin. 2018-11-15 17:00:15.008761: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 80, Chunks in use: 79. 173.0KiB allocated for chunks. 169.5KiB in use in bin. 166.2KiB client-requested in use in bin. 2018-11-15 17:00:15.008784: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 87, Chunks in use: 87. 348.0KiB allocated for chunks. 348.0KiB in use in bin. 348.0KiB client-requested in use in bin. 2018-11-15 17:00:15.008810: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-11-15 17:00:15.008832: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-11-15 17:00:15.008860: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 13, Chunks in use: 12. 564.5KiB allocated for chunks. 510.0KiB in use in bin. 508.9KiB client-requested in use in bin. 2018-11-15 17:00:15.008884: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 14, Chunks in use: 14. 1.25MiB allocated for chunks. 1.25MiB in use in bin. 1.20MiB client-requested in use in bin. 2018-11-15 17:00:15.008909: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 18, Chunks in use: 18. 2.44MiB allocated for chunks. 2.44MiB in use in bin. 2.44MiB client-requested in use in bin. 2018-11-15 17:00:15.008939: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 11, Chunks in use: 11. 3.09MiB allocated for chunks. 3.09MiB in use in bin. 3.09MiB client-requested in use in bin. 2018-11-15 17:00:15.008988: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 11, Chunks in use: 11. 5.50MiB allocated for chunks. 5.50MiB in use in bin. 5.50MiB client-requested in use in bin. 2018-11-15 17:00:15.009038: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 11, Chunks in use: 11. 12.38MiB allocated for chunks. 12.38MiB in use in bin. 12.38MiB client-requested in use in bin. 2018-11-15 17:00:15.009076: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 11, Chunks in use: 11. 22.00MiB allocated for chunks. 22.00MiB in use in bin. 22.00MiB client-requested in use in bin. 2018-11-15 17:00:15.009116: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 16, Chunks in use: 16. 72.00MiB allocated for chunks. 72.00MiB in use in bin. 72.00MiB client-requested in use in bin. 2018-11-15 17:00:15.009149: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-11-15 17:00:15.009202: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 17, Chunks in use: 17. 327.65MiB allocated for chunks. 327.65MiB in use in bin. 319.69MiB client-requested in use in bin. 2018-11-15 17:00:15.009249: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 17, Chunks in use: 17. 666.00MiB allocated for chunks. 666.00MiB in use in bin. 666.00MiB client-requested in use in bin. 2018-11-15 17:00:15.009288: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 1, Chunks in use: 1. 84.50MiB allocated for chunks. 84.50MiB in use in bin. 84.50MiB client-requested in use in bin. 2018-11-15 17:00:15.009333: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 1, Chunks in use: 1. 245.55MiB allocated for chunks. 245.55MiB in use in bin. 169.00MiB client-requested in use in bin. 2018-11-15 17:00:15.009373: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 5, Chunks in use: 5. 1.65GiB allocated for chunks. 1.65GiB in use in bin. 1.65GiB client-requested in use in bin. 2018-11-15 17:00:15.009409: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 72.0KiB was 64.0KiB, Chunk State: 2018-11-15 17:00:15.009451: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000000 of size 256 2018-11-15 17:00:15.009482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000100 of size 256 2018-11-15 17:00:15.009510: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000200 of size 256 2018-11-15 17:00:15.009532: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000300 of size 256 2018-11-15 17:00:15.009562: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000400 of size 256 2018-11-15 17:00:15.009593: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000500 of size 256 2018-11-15 17:00:15.009620: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000600 of size 256 2018-11-15 17:00:15.009648: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000700 of size 256 2018-11-15 17:00:15.009675: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000800 of size 256 2018-11-15 17:00:15.009703: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000900 of size 256 2018-11-15 17:00:15.009730: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000a00 of size 256 2018-11-15 17:00:15.009757: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000b00 of size 256 2018-11-15 17:00:15.009784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000c00 of size 256 2018-11-15 17:00:15.009812: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000d00 of size 256 2018-11-15 17:00:15.009839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000e00 of size 256 2018-11-15 17:00:15.009866: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0000f00 of size 256 2018-11-15 17:00:15.009894: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001000 of size 256 2018-11-15 17:00:15.009921: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001100 of size 256 2018-11-15 17:00:15.009950: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001200 of size 256 2018-11-15 17:00:15.009978: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001300 of size 256 2018-11-15 17:00:15.010004: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001400 of size 256 2018-11-15 17:00:15.010033: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001500 of size 256 2018-11-15 17:00:15.010059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001600 of size 256 2018-11-15 17:00:15.010086: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001700 of size 256 2018-11-15 17:00:15.010112: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001800 of size 256 2018-11-15 17:00:15.010138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001900 of size 256 2018-11-15 17:00:15.010162: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001a00 of size 256 2018-11-15 17:00:15.010187: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001b00 of size 256 2018-11-15 17:00:15.010214: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0001c00 of size 4096 2018-11-15 17:00:15.010239: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0002c00 of size 4096 2018-11-15 17:00:15.010264: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0003c00 of size 256 2018-11-15 17:00:15.010291: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0003d00 of size 256 2018-11-15 17:00:15.010318: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0003e00 of size 512 2018-11-15 17:00:15.010343: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0004000 of size 512 2018-11-15 17:00:15.010370: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0004200 of size 1024 2018-11-15 17:00:15.010398: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0004600 of size 1024 2018-11-15 17:00:15.010426: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0004a00 of size 2048 2018-11-15 17:00:15.010454: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0005200 of size 2048 2018-11-15 17:00:15.010481: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0005a00 of size 1280 2018-11-15 17:00:15.010507: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0005f00 of size 4096 2018-11-15 17:00:15.010534: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0006f00 of size 4096 2018-11-15 17:00:15.010561: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0007f00 of size 4096 2018-11-15 17:00:15.010588: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0008f00 of size 4096 2018-11-15 17:00:15.010615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0009f00 of size 4096 2018-11-15 17:00:15.010642: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a000af00 of size 4096 2018-11-15 17:00:15.010669: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a000bf00 of size 4096 2018-11-15 17:00:15.010696: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a000cf00 of size 4096 2018-11-15 17:00:15.010723: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a000df00 of size 4096 2018-11-15 17:00:15.010750: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a000ef00 of size 4096 2018-11-15 17:00:15.010778: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a000ff00 of size 4096 2018-11-15 17:00:15.010804: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0010f00 of size 4096 2018-11-15 17:00:15.010831: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0011f00 of size 4096 2018-11-15 17:00:15.010858: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0012f00 of size 4096 2018-11-15 17:00:15.010887: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0013f00 of size 4096 2018-11-15 17:00:15.010914: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0014f00 of size 4096 2018-11-15 17:00:15.010941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0015f00 of size 4096 2018-11-15 17:00:15.010968: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0016f00 of size 4096 2018-11-15 17:00:15.010996: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0017f00 of size 4096 2018-11-15 17:00:15.011023: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0018f00 of size 4096 2018-11-15 17:00:15.011050: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0019f00 of size 4096 2018-11-15 17:00:15.011077: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001af00 of size 4096 2018-11-15 17:00:15.011105: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001bf00 of size 4096 2018-11-15 17:00:15.011132: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001cf00 of size 4096 2018-11-15 17:00:15.011160: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001df00 of size 256 2018-11-15 17:00:15.011187: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001e000 of size 256 2018-11-15 17:00:15.011215: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001e100 of size 256 2018-11-15 17:00:15.011241: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001e200 of size 256 2018-11-15 17:00:15.011269: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001e300 of size 256 2018-11-15 17:00:15.011296: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001e400 of size 256 2018-11-15 17:00:15.011323: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001e500 of size 256 2018-11-15 17:00:15.011351: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001e600 of size 256 2018-11-15 17:00:15.011378: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001e700 of size 256 2018-11-15 17:00:15.011404: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001e800 of size 256 2018-11-15 17:00:15.011432: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001e900 of size 256 2018-11-15 17:00:15.011459: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001ea00 of size 256 2018-11-15 17:00:15.011487: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001eb00 of size 512 2018-11-15 17:00:15.011513: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001ed00 of size 512 2018-11-15 17:00:15.011541: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001ef00 of size 512 2018-11-15 17:00:15.011570: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001f100 of size 512 2018-11-15 17:00:15.011596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001f300 of size 512 2018-11-15 17:00:15.011624: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001f500 of size 512 2018-11-15 17:00:15.011676: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001f700 of size 512 2018-11-15 17:00:15.011704: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001f900 of size 512 2018-11-15 17:00:15.011731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001fb00 of size 512 2018-11-15 17:00:15.011759: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001fd00 of size 512 2018-11-15 17:00:15.011810: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a001ff00 of size 512 2018-11-15 17:00:15.011838: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0020100 of size 512 2018-11-15 17:00:15.011865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0020300 of size 1024 2018-11-15 17:00:15.011892: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0020700 of size 1024 2018-11-15 17:00:15.011920: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0020b00 of size 1024 2018-11-15 17:00:15.011947: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0020f00 of size 1024 2018-11-15 17:00:15.011973: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0021300 of size 1024 2018-11-15 17:00:15.012000: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0021700 of size 1024 2018-11-15 17:00:15.012028: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0021b00 of size 1024 2018-11-15 17:00:15.012055: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0021f00 of size 1024 2018-11-15 17:00:15.012082: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0022300 of size 1024 2018-11-15 17:00:15.012109: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0022700 of size 1024 2018-11-15 17:00:15.012137: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0022b00 of size 1024 2018-11-15 17:00:15.012164: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0022f00 of size 1024 2018-11-15 17:00:15.012191: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0023300 of size 2048 2018-11-15 17:00:15.012219: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0023b00 of size 2048 2018-11-15 17:00:15.012248: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0024300 of size 2048 2018-11-15 17:00:15.012276: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0024b00 of size 2048 2018-11-15 17:00:15.012303: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0025300 of size 2048 2018-11-15 17:00:15.012330: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0025b00 of size 2048 2018-11-15 17:00:15.012357: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0026300 of size 2048 2018-11-15 17:00:15.012384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0026b00 of size 2048 2018-11-15 17:00:15.012411: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0027300 of size 2048 2018-11-15 17:00:15.012438: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0027b00 of size 2048 2018-11-15 17:00:15.012465: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0028300 of size 2048 2018-11-15 17:00:15.012491: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0028b00 of size 2048 2018-11-15 17:00:15.012519: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0029300 of size 2048 2018-11-15 17:00:15.012546: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a0029b00 of size 2048 2018-11-15 17:00:15.012573: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a002a300 of size 2048 2018-11-15 17:00:15.012600: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a002ab00 of size 2048 2018-11-15 17:00:15.012627: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a002b300 of size 2048 2018-11-15 17:00:15.012654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a002bb00 of size 2048 2018-11-15 17:00:15.012682: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a002c300 of size 2048 2018-11-15 17:00:15.012721: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a002cb00 of size 3584 2018-11-15 17:00:15.012749: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a002d900 of size 524288 2018-11-15 17:00:15.012776: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a00ad900 of size 524288 2018-11-15 17:00:15.012805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a012d900 of size 2097152 2018-11-15 17:00:15.012832: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a032d900 of size 2097152 2018-11-15 17:00:15.012859: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a052d900 of size 18874368 2018-11-15 17:00:15.012888: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a172d900 of size 18874368 2018-11-15 17:00:15.012917: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a292d900 of size 18874368 2018-11-15 17:00:15.012945: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a3b2d900 of size 73728 2018-11-15 17:00:15.012974: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a3b3f900 of size 37748736 2018-11-15 17:00:15.013001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a5f3f900 of size 37748736 2018-11-15 17:00:15.013030: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a833f900 of size 131072 2018-11-15 17:00:15.013059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95a835f900 of size 47185920 2018-11-15 17:00:15.013087: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ab05f900 of size 32768 2018-11-15 17:00:15.013115: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ab067900 of size 294912 2018-11-15 17:00:15.013142: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ab0af900 of size 294912 2018-11-15 17:00:15.013170: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ab0f7900 of size 131072 2018-11-15 17:00:15.013198: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ab117900 of size 1179648 2018-11-15 17:00:15.013226: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ab237900 of size 1179648 2018-11-15 17:00:15.013254: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ab357900 of size 4718592 2018-11-15 17:00:15.013281: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ab7d7900 of size 4718592 2018-11-15 17:00:15.013309: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95abc57900 of size 4718592 2018-11-15 17:00:15.013336: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d7900 of size 256 2018-11-15 17:00:15.013364: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d7a00 of size 256 2018-11-15 17:00:15.013391: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d7b00 of size 256 2018-11-15 17:00:15.013418: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d7c00 of size 256 2018-11-15 17:00:15.013445: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d7d00 of size 1024 2018-11-15 17:00:15.013473: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d8100 of size 1024 2018-11-15 17:00:15.013500: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d8500 of size 1024 2018-11-15 17:00:15.013527: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d8900 of size 1024 2018-11-15 17:00:15.013555: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d8d00 of size 2048 2018-11-15 17:00:15.013584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d9500 of size 256 2018-11-15 17:00:15.013612: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d9600 of size 256 2018-11-15 17:00:15.013639: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d9700 of size 256 2018-11-15 17:00:15.013668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac0d9800 of size 163840 2018-11-15 17:00:15.013695: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac101800 of size 256 2018-11-15 17:00:15.013722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac101900 of size 256 2018-11-15 17:00:15.013750: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac101a00 of size 256 2018-11-15 17:00:15.013777: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac101b00 of size 256 2018-11-15 17:00:15.013804: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac101c00 of size 256 2018-11-15 17:00:15.013831: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac101d00 of size 2097152 2018-11-15 17:00:15.013858: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ac301d00 of size 18874368 2018-11-15 17:00:15.013885: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ad501d00 of size 131072 2018-11-15 17:00:15.013913: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ad521d00 of size 37748736 2018-11-15 17:00:15.013940: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95af921d00 of size 47185920 2018-11-15 17:00:15.013968: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2621d00 of size 4096 2018-11-15 17:00:15.013995: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2622d00 of size 163840 2018-11-15 17:00:15.014023: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b264ad00 of size 256 2018-11-15 17:00:15.014050: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b264ae00 of size 3584 2018-11-15 17:00:15.014077: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b264bc00 of size 256 2018-11-15 17:00:15.014104: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b264bd00 of size 73728 2018-11-15 17:00:15.014132: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b265dd00 of size 256 2018-11-15 17:00:15.014160: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b265de00 of size 294912 2018-11-15 17:00:15.014187: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b26a5e00 of size 131072 2018-11-15 17:00:15.014214: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b26c5e00 of size 512 2018-11-15 17:00:15.020472: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b26c6000 of size 1179648 2018-11-15 17:00:15.020498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b27e6000 of size 32768 2018-11-15 17:00:15.020510: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b27ee000 of size 524288 2018-11-15 17:00:15.020520: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b286e000 of size 1024 2018-11-15 17:00:15.020530: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b286e400 of size 4718592 2018-11-15 17:00:15.020539: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2cee400 of size 2048 2018-11-15 17:00:15.020549: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2ceec00 of size 256 2018-11-15 17:00:15.020559: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2ceed00 of size 256 2018-11-15 17:00:15.020568: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2ceee00 of size 256 2018-11-15 17:00:15.020577: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2ceef00 of size 256 2018-11-15 17:00:15.020587: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2cef000 of size 256 2018-11-15 17:00:15.020596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2cef100 of size 256 2018-11-15 17:00:15.020606: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2cef200 of size 256 2018-11-15 17:00:15.020615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2cef300 of size 2097152 2018-11-15 17:00:15.020625: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b2eef300 of size 2097152 2018-11-15 17:00:15.020635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b30ef300 of size 2097152 2018-11-15 17:00:15.020644: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b32ef300 of size 2097152 2018-11-15 17:00:15.020653: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b34ef300 of size 18874368 2018-11-15 17:00:15.020663: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b46ef300 of size 18874368 2018-11-15 17:00:15.020672: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b58ef300 of size 18874368 2018-11-15 17:00:15.020682: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b6aef300 of size 18874368 2018-11-15 17:00:15.020691: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b7cef300 of size 18874368 2018-11-15 17:00:15.020700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95b8eef300 of size 18874368 2018-11-15 17:00:15.020709: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ba0ef300 of size 131072 2018-11-15 17:00:15.020719: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ba10f300 of size 131072 2018-11-15 17:00:15.020728: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ba12f300 of size 37748736 2018-11-15 17:00:15.020739: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95bc52f300 of size 37748736 2018-11-15 17:00:15.020748: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95be92f300 of size 37748736 2018-11-15 17:00:15.020757: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c0d2f300 of size 37748736 2018-11-15 17:00:15.020767: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c312f300 of size 47185920 2018-11-15 17:00:15.020776: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c5e2f300 of size 47185920 2018-11-15 17:00:15.020785: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b2f300 of size 4096 2018-11-15 17:00:15.020795: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b30300 of size 4096 2018-11-15 17:00:15.020804: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b31300 of size 4096 2018-11-15 17:00:15.020815: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b32300 of size 4096 2018-11-15 17:00:15.020824: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b33300 of size 4096 2018-11-15 17:00:15.021142: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b34300 of size 4096 2018-11-15 17:00:15.021163: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b35300 of size 4096 2018-11-15 17:00:15.021174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b36300 of size 4096 2018-11-15 17:00:15.021183: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b37300 of size 4096 2018-11-15 17:00:15.021194: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b38300 of size 4096 2018-11-15 17:00:15.021204: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b39300 of size 4096 2018-11-15 17:00:15.021213: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b3a300 of size 4096 2018-11-15 17:00:15.021222: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b3b300 of size 4096 2018-11-15 17:00:15.021231: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b3c300 of size 4096 2018-11-15 17:00:15.021240: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b3d300 of size 4096 2018-11-15 17:00:15.021249: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b3e300 of size 4096 2018-11-15 17:00:15.021259: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b3f300 of size 4096 2018-11-15 17:00:15.021269: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b40300 of size 4096 2018-11-15 17:00:15.021278: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b41300 of size 4096 2018-11-15 17:00:15.021287: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b42300 of size 4096 2018-11-15 17:00:15.021296: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b43300 of size 4096 2018-11-15 17:00:15.021305: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b44300 of size 4096 2018-11-15 17:00:15.021315: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b45300 of size 4096 2018-11-15 17:00:15.021324: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b46300 of size 4096 2018-11-15 17:00:15.021333: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b47300 of size 163840 2018-11-15 17:00:15.021343: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b6f300 of size 163840 2018-11-15 17:00:15.021353: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b97300 of size 256 2018-11-15 17:00:15.021362: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b97400 of size 256 2018-11-15 17:00:15.021372: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b97500 of size 3584 2018-11-15 17:00:15.021381: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b98300 of size 3584 2018-11-15 17:00:15.021391: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b99100 of size 256 2018-11-15 17:00:15.021400: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b99200 of size 256 2018-11-15 17:00:15.021410: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b99300 of size 256 2018-11-15 17:00:15.021419: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b99400 of size 256 2018-11-15 17:00:15.021429: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8b99500 of size 73728 2018-11-15 17:00:15.021438: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bab500 of size 73728 2018-11-15 17:00:15.021448: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbd500 of size 256 2018-11-15 17:00:15.021831: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbd600 of size 256 2018-11-15 17:00:15.021851: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbd700 of size 256 2018-11-15 17:00:15.021863: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbd800 of size 256 2018-11-15 17:00:15.021873: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbd900 of size 256 2018-11-15 17:00:15.021883: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbda00 of size 256 2018-11-15 17:00:15.021892: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbdb00 of size 256 2018-11-15 17:00:15.021902: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbdc00 of size 256 2018-11-15 17:00:15.021911: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbdd00 of size 256 2018-11-15 17:00:15.021921: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbde00 of size 256 2018-11-15 17:00:15.021931: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbdf00 of size 256 2018-11-15 17:00:15.021940: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbe000 of size 256 2018-11-15 17:00:15.021949: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8bbe100 of size 294912 2018-11-15 17:00:15.021959: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8c06100 of size 294912 2018-11-15 17:00:15.021968: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8c4e100 of size 294912 2018-11-15 17:00:15.021976: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8c96100 of size 294912 2018-11-15 17:00:15.021986: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8cde100 of size 131072 2018-11-15 17:00:15.021995: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8cfe100 of size 131072 2018-11-15 17:00:15.022004: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1e100 of size 512 2018-11-15 17:00:15.022013: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1e300 of size 512 2018-11-15 17:00:15.022023: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1e500 of size 512 2018-11-15 17:00:15.022032: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1e700 of size 512 2018-11-15 17:00:15.022042: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1e900 of size 512 2018-11-15 17:00:15.022051: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1eb00 of size 512 2018-11-15 17:00:15.022060: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1ed00 of size 512 2018-11-15 17:00:15.022070: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1ef00 of size 512 2018-11-15 17:00:15.022081: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1f100 of size 512 2018-11-15 17:00:15.022091: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1f300 of size 512 2018-11-15 17:00:15.022100: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1f500 of size 512 2018-11-15 17:00:15.022110: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1f700 of size 512 2018-11-15 17:00:15.022119: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8d1f900 of size 1179648 2018-11-15 17:00:15.022129: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8e3f900 of size 1179648 2018-11-15 17:00:15.022138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c8f5f900 of size 1179648 2018-11-15 17:00:15.022147: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c907f900 of size 1179648 2018-11-15 17:00:15.022156: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c919f900 of size 32768 2018-11-15 17:00:15.022165: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c91a7900 of size 32768 2018-11-15 17:00:15.022174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c91af900 of size 524288 2018-11-15 17:00:15.022560: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c922f900 of size 524288 2018-11-15 17:00:15.022572: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c92af900 of size 524288 2018-11-15 17:00:15.022582: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c932f900 of size 524288 2018-11-15 17:00:15.022591: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93af900 of size 1024 2018-11-15 17:00:15.022601: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93afd00 of size 1024 2018-11-15 17:00:15.022610: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b0100 of size 1024 2018-11-15 17:00:15.022620: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b0500 of size 1024 2018-11-15 17:00:15.022629: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b0900 of size 1024 2018-11-15 17:00:15.022639: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b0d00 of size 1024 2018-11-15 17:00:15.022648: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b1100 of size 1024 2018-11-15 17:00:15.022657: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b1500 of size 1024 2018-11-15 17:00:15.022666: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b1900 of size 1024 2018-11-15 17:00:15.022675: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b1d00 of size 1024 2018-11-15 17:00:15.022685: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b2100 of size 1024 2018-11-15 17:00:15.022694: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b2500 of size 1024 2018-11-15 17:00:15.022703: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b2900 of size 1024 2018-11-15 17:00:15.022713: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b2d00 of size 1024 2018-11-15 17:00:15.022722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b3100 of size 1024 2018-11-15 17:00:15.022731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b3500 of size 1024 2018-11-15 17:00:15.022741: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c93b3900 of size 4718592 2018-11-15 17:00:15.022750: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c9833900 of size 4718592 2018-11-15 17:00:15.022760: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95c9cb3900 of size 4718592 2018-11-15 17:00:15.022769: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ca133900 of size 4718592 2018-11-15 17:00:15.022778: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ca5b3900 of size 4718592 2018-11-15 17:00:15.022788: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caa33900 of size 4718592 2018-11-15 17:00:15.022797: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb3900 of size 2048 2018-11-15 17:00:15.022806: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb4100 of size 2048 2018-11-15 17:00:15.022815: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb4900 of size 2048 2018-11-15 17:00:15.022824: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb5100 of size 2048 2018-11-15 17:00:15.022833: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb5900 of size 2048 2018-11-15 17:00:15.022843: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb6100 of size 2048 2018-11-15 17:00:15.022852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb6900 of size 2048 2018-11-15 17:00:15.023273: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb7100 of size 2048 2018-11-15 17:00:15.023285: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb7900 of size 2048 2018-11-15 17:00:15.023295: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb8100 of size 2048 2018-11-15 17:00:15.023304: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb8900 of size 2048 2018-11-15 17:00:15.023314: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb9100 of size 2048 2018-11-15 17:00:15.023323: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeb9900 of size 2048 2018-11-15 17:00:15.023332: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeba100 of size 2048 2018-11-15 17:00:15.023341: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeba900 of size 2048 2018-11-15 17:00:15.023351: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebb100 of size 2048 2018-11-15 17:00:15.023360: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebb900 of size 2048 2018-11-15 17:00:15.023370: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebc100 of size 2048 2018-11-15 17:00:15.023379: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebc900 of size 2048 2018-11-15 17:00:15.023388: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebd100 of size 2048 2018-11-15 17:00:15.023398: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebd900 of size 256 2018-11-15 17:00:15.023407: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebda00 of size 256 2018-11-15 17:00:15.023416: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebdb00 of size 256 2018-11-15 17:00:15.023426: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebdc00 of size 256 2018-11-15 17:00:15.023435: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebdd00 of size 256 2018-11-15 17:00:15.023444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebde00 of size 256 2018-11-15 17:00:15.023453: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebdf00 of size 256 2018-11-15 17:00:15.023463: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebe000 of size 256 2018-11-15 17:00:15.023473: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebe100 of size 256 2018-11-15 17:00:15.023482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebe200 of size 256 2018-11-15 17:00:15.023491: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebe300 of size 256 2018-11-15 17:00:15.023501: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebe400 of size 256 2018-11-15 17:00:15.023510: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebe500 of size 256 2018-11-15 17:00:15.023520: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebe600 of size 256 2018-11-15 17:00:15.023528: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebe700 of size 256 2018-11-15 17:00:15.023538: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebe800 of size 256 2018-11-15 17:00:15.023547: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebe900 of size 256 2018-11-15 17:00:15.023556: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebea00 of size 256 2018-11-15 17:00:15.023565: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebeb00 of size 256 2018-11-15 17:00:15.024014: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebec00 of size 256 2018-11-15 17:00:15.024027: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebed00 of size 256 2018-11-15 17:00:15.024037: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebee00 of size 256 2018-11-15 17:00:15.024046: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebef00 of size 256 2018-11-15 17:00:15.024055: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebf000 of size 256 2018-11-15 17:00:15.024065: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebf100 of size 256 2018-11-15 17:00:15.024073: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebf200 of size 256 2018-11-15 17:00:15.024082: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebf300 of size 256 2018-11-15 17:00:15.024091: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebf400 of size 256 2018-11-15 17:00:15.024100: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebf500 of size 256 2018-11-15 17:00:15.024110: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebf600 of size 256 2018-11-15 17:00:15.024118: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebf700 of size 256 2018-11-15 17:00:15.024128: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebf800 of size 256 2018-11-15 17:00:15.024137: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebf900 of size 256 2018-11-15 17:00:15.024146: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebfa00 of size 256 2018-11-15 17:00:15.024156: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebfb00 of size 256 2018-11-15 17:00:15.024165: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebfc00 of size 256 2018-11-15 17:00:15.024174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebfd00 of size 256 2018-11-15 17:00:15.024183: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebfe00 of size 256 2018-11-15 17:00:15.024193: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caebff00 of size 256 2018-11-15 17:00:15.024202: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0000 of size 256 2018-11-15 17:00:15.024211: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0100 of size 256 2018-11-15 17:00:15.024220: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0200 of size 256 2018-11-15 17:00:15.024229: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0300 of size 256 2018-11-15 17:00:15.024238: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0400 of size 256 2018-11-15 17:00:15.024247: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0500 of size 256 2018-11-15 17:00:15.024257: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0600 of size 256 2018-11-15 17:00:15.024266: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0700 of size 256 2018-11-15 17:00:15.024275: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0800 of size 256 2018-11-15 17:00:15.024284: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0900 of size 256 2018-11-15 17:00:15.024293: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0a00 of size 256 2018-11-15 17:00:15.024301: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0b00 of size 256 2018-11-15 17:00:15.024311: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0c00 of size 256 2018-11-15 17:00:15.024320: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0d00 of size 256 2018-11-15 17:00:15.024329: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0e00 of size 256 2018-11-15 17:00:15.024337: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec0f00 of size 256 2018-11-15 17:00:15.024711: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec1000 of size 256 2018-11-15 17:00:15.024724: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec1100 of size 256 2018-11-15 17:00:15.024735: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caec1200 of size 108288 2018-11-15 17:00:15.024745: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caedb900 of size 256 2018-11-15 17:00:15.024754: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caedba00 of size 108288 2018-11-15 17:00:15.024763: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef6100 of size 256 2018-11-15 17:00:15.024771: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef6200 of size 256 2018-11-15 17:00:15.024780: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef6300 of size 256 2018-11-15 17:00:15.024789: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef6400 of size 256 2018-11-15 17:00:15.024798: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef6500 of size 256 2018-11-15 17:00:15.024807: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef6600 of size 2048 2018-11-15 17:00:15.024816: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef6e00 of size 256 2018-11-15 17:00:15.024826: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef6f00 of size 256 2018-11-15 17:00:15.024835: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef7000 of size 512 2018-11-15 17:00:15.024844: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef7200 of size 256 2018-11-15 17:00:15.024854: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef7300 of size 512 2018-11-15 17:00:15.024863: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef7500 of size 4096 2018-11-15 17:00:15.024872: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef8500 of size 4096 2018-11-15 17:00:15.024880: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef9500 of size 1024 2018-11-15 17:00:15.024889: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef9900 of size 256 2018-11-15 17:00:15.024899: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef9a00 of size 256 2018-11-15 17:00:15.024906: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caef9b00 of size 4096 2018-11-15 17:00:15.024913: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caefab00 of size 4096 2018-11-15 17:00:15.024922: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caefbb00 of size 4096 2018-11-15 17:00:15.024931: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caefcb00 of size 4096 2018-11-15 17:00:15.024940: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caefdb00 of size 512 2018-11-15 17:00:15.024949: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caefdd00 of size 4096 2018-11-15 17:00:15.024959: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caefed00 of size 4096 2018-11-15 17:00:15.024968: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caeffd00 of size 2048 2018-11-15 17:00:15.024978: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf00500 of size 2048 2018-11-15 17:00:15.024987: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf00d00 of size 4096 2018-11-15 17:00:15.024996: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf01d00 of size 1024 2018-11-15 17:00:15.025464: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf02100 of size 4096 2018-11-15 17:00:15.025476: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf03100 of size 2048 2018-11-15 17:00:15.025486: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf03900 of size 2048 2018-11-15 17:00:15.025495: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf04100 of size 4096 2018-11-15 17:00:15.025505: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf05100 of size 4096 2018-11-15 17:00:15.025514: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf06100 of size 2048 2018-11-15 17:00:15.025524: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf06900 of size 2048 2018-11-15 17:00:15.025533: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf07100 of size 2048 2018-11-15 17:00:15.025543: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf07900 of size 1024 2018-11-15 17:00:15.025551: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf07d00 of size 1024 2018-11-15 17:00:15.025560: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf08100 of size 2048 2018-11-15 17:00:15.025569: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf08900 of size 1024 2018-11-15 17:00:15.025579: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf08d00 of size 1024 2018-11-15 17:00:15.025587: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf09100 of size 2048 2018-11-15 17:00:15.025596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf09900 of size 1024 2018-11-15 17:00:15.025606: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf09d00 of size 512 2018-11-15 17:00:15.025615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf09f00 of size 1024 2018-11-15 17:00:15.025624: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0a300 of size 512 2018-11-15 17:00:15.025634: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0a500 of size 256 2018-11-15 17:00:15.025643: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0a600 of size 512 2018-11-15 17:00:15.025653: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0a800 of size 256 2018-11-15 17:00:15.025663: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0a900 of size 256 2018-11-15 17:00:15.025673: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0aa00 of size 256 2018-11-15 17:00:15.025682: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ab00 of size 256 2018-11-15 17:00:15.025691: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ac00 of size 256 2018-11-15 17:00:15.025701: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ad00 of size 256 2018-11-15 17:00:15.025710: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ae00 of size 256 2018-11-15 17:00:15.025719: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0af00 of size 256 2018-11-15 17:00:15.025729: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0b000 of size 256 2018-11-15 17:00:15.025738: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0b100 of size 256 2018-11-15 17:00:15.025747: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0b200 of size 256 2018-11-15 17:00:15.025755: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0b300 of size 256 2018-11-15 17:00:15.025765: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0b400 of size 256 2018-11-15 17:00:15.025774: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0b500 of size 256 2018-11-15 17:00:15.025783: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0b600 of size 256 2018-11-15 17:00:15.025792: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0b700 of size 256 2018-11-15 17:00:15.026191: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0b800 of size 256 2018-11-15 17:00:15.026204: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0b900 of size 256 2018-11-15 17:00:15.026213: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ba00 of size 256 2018-11-15 17:00:15.026223: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0bb00 of size 256 2018-11-15 17:00:15.026232: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0bc00 of size 256 2018-11-15 17:00:15.026241: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0bd00 of size 256 2018-11-15 17:00:15.026250: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0be00 of size 256 2018-11-15 17:00:15.026259: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0bf00 of size 256 2018-11-15 17:00:15.026268: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0c000 of size 256 2018-11-15 17:00:15.026277: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0c100 of size 256 2018-11-15 17:00:15.026287: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0c200 of size 256 2018-11-15 17:00:15.026295: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0c300 of size 256 2018-11-15 17:00:15.026305: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0c400 of size 256 2018-11-15 17:00:15.026314: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0c500 of size 256 2018-11-15 17:00:15.026323: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0c600 of size 256 2018-11-15 17:00:15.026332: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0c700 of size 256 2018-11-15 17:00:15.026341: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0c800 of size 256 2018-11-15 17:00:15.026350: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0c900 of size 256 2018-11-15 17:00:15.026359: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ca00 of size 256 2018-11-15 17:00:15.026368: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0cb00 of size 256 2018-11-15 17:00:15.026378: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0cc00 of size 256 2018-11-15 17:00:15.026387: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0cd00 of size 256 2018-11-15 17:00:15.026396: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ce00 of size 256 2018-11-15 17:00:15.026405: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0cf00 of size 256 2018-11-15 17:00:15.026414: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0d000 of size 256 2018-11-15 17:00:15.026423: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0d100 of size 256 2018-11-15 17:00:15.026432: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0d200 of size 256 2018-11-15 17:00:15.026442: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0d300 of size 256 2018-11-15 17:00:15.026451: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0d400 of size 256 2018-11-15 17:00:15.026460: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0d500 of size 256 2018-11-15 17:00:15.026469: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0d600 of size 256 2018-11-15 17:00:15.026477: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0d700 of size 256 2018-11-15 17:00:15.026901: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0d800 of size 256 2018-11-15 17:00:15.026913: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0d900 of size 256 2018-11-15 17:00:15.026923: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0da00 of size 256 2018-11-15 17:00:15.026932: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0db00 of size 256 2018-11-15 17:00:15.026942: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0dc00 of size 256 2018-11-15 17:00:15.026951: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0dd00 of size 256 2018-11-15 17:00:15.026961: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0de00 of size 256 2018-11-15 17:00:15.026970: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0df00 of size 256 2018-11-15 17:00:15.026980: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0e000 of size 256 2018-11-15 17:00:15.026989: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0e100 of size 256 2018-11-15 17:00:15.026999: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0e200 of size 256 2018-11-15 17:00:15.027008: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0e300 of size 256 2018-11-15 17:00:15.027017: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0e400 of size 256 2018-11-15 17:00:15.027026: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0e500 of size 256 2018-11-15 17:00:15.027034: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0e600 of size 256 2018-11-15 17:00:15.027043: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0e700 of size 256 2018-11-15 17:00:15.027052: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0e800 of size 256 2018-11-15 17:00:15.027061: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0e900 of size 256 2018-11-15 17:00:15.027071: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ea00 of size 256 2018-11-15 17:00:15.027080: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0eb00 of size 256 2018-11-15 17:00:15.027090: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ec00 of size 256 2018-11-15 17:00:15.027099: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ed00 of size 256 2018-11-15 17:00:15.027108: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ee00 of size 256 2018-11-15 17:00:15.027117: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0ef00 of size 256 2018-11-15 17:00:15.027127: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0f000 of size 256 2018-11-15 17:00:15.027136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0f100 of size 256 2018-11-15 17:00:15.027144: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0f200 of size 256 2018-11-15 17:00:15.027153: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0f300 of size 256 2018-11-15 17:00:15.027162: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0f400 of size 256 2018-11-15 17:00:15.027172: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0f500 of size 256 2018-11-15 17:00:15.027180: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0f600 of size 256 2018-11-15 17:00:15.027189: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0f700 of size 256 2018-11-15 17:00:15.027198: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0f800 of size 256 2018-11-15 17:00:15.027207: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0f900 of size 256 2018-11-15 17:00:15.027216: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0fa00 of size 256 2018-11-15 17:00:15.027225: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0fb00 of size 256 2018-11-15 17:00:15.027637: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0fc00 of size 256 2018-11-15 17:00:15.027650: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf0fd00 of size 1536 2018-11-15 17:00:15.027659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f95caf10300 of size 3584 2018-11-15 17:00:15.027668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf11100 of size 256 2018-11-15 17:00:15.027677: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf11200 of size 256 2018-11-15 17:00:15.027685: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf11300 of size 256 2018-11-15 17:00:15.027694: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf11400 of size 256 2018-11-15 17:00:15.027703: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf11500 of size 256 2018-11-15 17:00:15.027714: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf11600 of size 54272 2018-11-15 17:00:15.027723: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf1ea00 of size 54272 2018-11-15 17:00:15.027733: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf2be00 of size 108032 2018-11-15 17:00:15.027742: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf46400 of size 108288 2018-11-15 17:00:15.027752: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf60b00 of size 54272 2018-11-15 17:00:15.027760: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf6df00 of size 108288 2018-11-15 17:00:15.027770: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf88600 of size 54272 2018-11-15 17:00:15.027778: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95caf95a00 of size 54272 2018-11-15 17:00:15.027788: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95cafa2e00 of size 108288 2018-11-15 17:00:15.027796: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95cafbd500 of size 54272 2018-11-15 17:00:15.027805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95cafca900 of size 524288 2018-11-15 17:00:15.027814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95cb04a900 of size 4718592 2018-11-15 17:00:15.027824: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95cb4ca900 of size 27220480 2018-11-15 17:00:15.027833: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccec0300 of size 256 2018-11-15 17:00:15.027842: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccec0400 of size 108288 2018-11-15 17:00:15.027851: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccedab00 of size 108288 2018-11-15 17:00:15.027860: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccef5200 of size 1024 2018-11-15 17:00:15.027870: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccef5600 of size 73728 2018-11-15 17:00:15.027878: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf07600 of size 4096 2018-11-15 17:00:15.027887: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf08600 of size 4096 2018-11-15 17:00:15.027896: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf09600 of size 2048 2018-11-15 17:00:15.027905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf09e00 of size 3328 2018-11-15 17:00:15.027915: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf0ab00 of size 4096 2018-11-15 17:00:15.027924: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf0bb00 of size 256 2018-11-15 17:00:15.028342: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf0bc00 of size 256 2018-11-15 17:00:15.028354: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf0bd00 of size 4096 2018-11-15 17:00:15.028363: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf0cd00 of size 2048 2018-11-15 17:00:15.028373: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf0d500 of size 2816 2018-11-15 17:00:15.028382: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf0e000 of size 2048 2018-11-15 17:00:15.028391: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf0e800 of size 4096 2018-11-15 17:00:15.028401: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf0f800 of size 4096 2018-11-15 17:00:15.028410: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf10800 of size 4096 2018-11-15 17:00:15.028419: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf11800 of size 256 2018-11-15 17:00:15.028428: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf11900 of size 4096 2018-11-15 17:00:15.028447: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf12900 of size 256 2018-11-15 17:00:15.028457: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf12a00 of size 256 2018-11-15 17:00:15.028466: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf12b00 of size 4096 2018-11-15 17:00:15.028476: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf13b00 of size 4096 2018-11-15 17:00:15.028485: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf14b00 of size 4096 2018-11-15 17:00:15.028494: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf15b00 of size 4096 2018-11-15 17:00:15.028503: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf16b00 of size 3584 2018-11-15 17:00:15.028512: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf17900 of size 256 2018-11-15 17:00:15.028521: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf17a00 of size 512 2018-11-15 17:00:15.028530: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf17c00 of size 512 2018-11-15 17:00:15.028539: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf17e00 of size 256 2018-11-15 17:00:15.028548: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf17f00 of size 256 2018-11-15 17:00:15.028557: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf18000 of size 512 2018-11-15 17:00:15.028566: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf18200 of size 512 2018-11-15 17:00:15.028575: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf18400 of size 256 2018-11-15 17:00:15.028583: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf18500 of size 1024 2018-11-15 17:00:15.028592: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf18900 of size 1024 2018-11-15 17:00:15.028601: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf18d00 of size 512 2018-11-15 17:00:15.028610: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf18f00 of size 512 2018-11-15 17:00:15.028619: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf19100 of size 1024 2018-11-15 17:00:15.028628: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf19500 of size 1024 2018-11-15 17:00:15.028636: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf19900 of size 2048 2018-11-15 17:00:15.028645: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf1a100 of size 256 2018-11-15 17:00:15.028654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf1a200 of size 2048 2018-11-15 17:00:15.028663: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf1aa00 of size 1024 2018-11-15 17:00:15.028672: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf1ae00 of size 1024 2018-11-15 17:00:15.029051: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf1b200 of size 2560 2018-11-15 17:00:15.029063: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf1bc00 of size 2048 2018-11-15 17:00:15.029072: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f95ccf1c400 of size 55808 2018-11-15 17:00:15.029083: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ccf29e00 of size 33226752 2018-11-15 17:00:15.029093: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ceed9e00 of size 1280 2018-11-15 17:00:15.029103: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95ceeda300 of size 354418688 2018-11-15 17:00:15.029111: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95e40da300 of size 354418688 2018-11-15 17:00:15.029120: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95f92da300 of size 2097152 2018-11-15 17:00:15.029129: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95f94da300 of size 18874368 2018-11-15 17:00:15.029138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95fa6da300 of size 2097152 2018-11-15 17:00:15.029147: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95fa8da300 of size 18874368 2018-11-15 17:00:15.029155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95fbada300 of size 37748736 2018-11-15 17:00:15.029164: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95fdeda300 of size 131072 2018-11-15 17:00:15.029173: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f95fdefa300 of size 37748736 2018-11-15 17:00:15.029182: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96002fa300 of size 47185920 2018-11-15 17:00:15.029191: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9602ffa300 of size 294912 2018-11-15 17:00:15.029200: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9603042300 of size 163840 2018-11-15 17:00:15.029209: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960306a300 of size 32768 2018-11-15 17:00:15.029218: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9603072300 of size 294912 2018-11-15 17:00:15.029227: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96030ba300 of size 1179648 2018-11-15 17:00:15.029237: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96031da300 of size 131072 2018-11-15 17:00:15.029245: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96031fa300 of size 1179648 2018-11-15 17:00:15.029254: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960331a300 of size 4718592 2018-11-15 17:00:15.029263: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960379a300 of size 524288 2018-11-15 17:00:15.029272: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960381a300 of size 4718592 2018-11-15 17:00:15.029281: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9603c9a300 of size 2048 2018-11-15 17:00:15.029290: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9603c9ab00 of size 256 2018-11-15 17:00:15.029299: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9603c9ac00 of size 2048 2018-11-15 17:00:15.029306: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9603c9b400 of size 2048 2018-11-15 17:00:15.029312: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9603c9bc00 of size 18874368 2018-11-15 17:00:15.029318: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9604e9bc00 of size 4096 2018-11-15 17:00:15.029326: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9604e9cc00 of size 4096 2018-11-15 17:00:15.035980: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9604e9dc00 of size 2097152 2018-11-15 17:00:15.036006: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960509dc00 of size 2048 2018-11-15 17:00:15.036018: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960509e400 of size 2048 2018-11-15 17:00:15.036029: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960509ec00 of size 18874368 2018-11-15 17:00:15.036039: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960629ec00 of size 256 2018-11-15 17:00:15.036048: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960629ed00 of size 4096 2018-11-15 17:00:15.036056: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960629fd00 of size 4096 2018-11-15 17:00:15.036064: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96062a0d00 of size 2097152 2018-11-15 17:00:15.036071: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96064a0d00 of size 2048 2018-11-15 17:00:15.036076: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96064a1500 of size 2048 2018-11-15 17:00:15.036083: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96064a1d00 of size 18874368 2018-11-15 17:00:15.036092: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96076a1d00 of size 4096 2018-11-15 17:00:15.036101: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96076a2d00 of size 4096 2018-11-15 17:00:15.036110: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f96076a3d00 of size 37748736 2018-11-15 17:00:15.036119: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9609aa3d00 of size 4096 2018-11-15 17:00:15.036128: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9609aa4d00 of size 256 2018-11-15 17:00:15.036137: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9609aa4e00 of size 4096 2018-11-15 17:00:15.036146: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9609aa5e00 of size 131072 2018-11-15 17:00:15.036155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9609ac5e00 of size 256 2018-11-15 17:00:15.036164: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9609ac5f00 of size 256 2018-11-15 17:00:15.036174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9609ac6000 of size 37748736 2018-11-15 17:00:15.036184: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960bec6000 of size 4096 2018-11-15 17:00:15.036193: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960bec7000 of size 4096 2018-11-15 17:00:15.036202: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960bec8000 of size 47185920 2018-11-15 17:00:15.036211: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ebc8000 of size 4096 2018-11-15 17:00:15.036220: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ebc9000 of size 4096 2018-11-15 17:00:15.036229: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ebca000 of size 3584 2018-11-15 17:00:15.036238: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ebcae00 of size 294912 2018-11-15 17:00:15.036247: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec12e00 of size 163840 2018-11-15 17:00:15.036256: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec3ae00 of size 256 2018-11-15 17:00:15.036265: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec3af00 of size 512 2018-11-15 17:00:15.036274: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec3b100 of size 512 2018-11-15 17:00:15.036281: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec3b300 of size 32768 2018-11-15 17:00:15.036290: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec43300 of size 256 2018-11-15 17:00:15.036299: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec43400 of size 256 2018-11-15 17:00:15.036308: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec43500 of size 294912 2018-11-15 17:00:15.036316: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec8b500 of size 512 2018-11-15 17:00:15.036325: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec8b700 of size 512 2018-11-15 17:00:15.036334: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960ec8b900 of size 1179648 2018-11-15 17:00:15.036343: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960edab900 of size 256 2018-11-15 17:00:15.036353: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960edaba00 of size 1024 2018-11-15 17:00:15.036362: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960edabe00 of size 1024 2018-11-15 17:00:15.036371: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960edac200 of size 131072 2018-11-15 17:00:15.036379: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960edcc200 of size 512 2018-11-15 17:00:15.036388: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960edcc400 of size 512 2018-11-15 17:00:15.036397: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960edcc600 of size 1179648 2018-11-15 17:00:15.036407: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960eeec600 of size 1024 2018-11-15 17:00:15.036416: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960eeeca00 of size 1024 2018-11-15 17:00:15.036425: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960eeece00 of size 4718592 2018-11-15 17:00:15.036434: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f36ce00 of size 2048 2018-11-15 17:00:15.036443: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f36d600 of size 256 2018-11-15 17:00:15.036451: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f36d700 of size 2048 2018-11-15 17:00:15.036460: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f36df00 of size 524288 2018-11-15 17:00:15.036469: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f3edf00 of size 1024 2018-11-15 17:00:15.036477: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f3ee300 of size 1024 2018-11-15 17:00:15.036486: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f3ee700 of size 73728 2018-11-15 17:00:15.036495: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f400700 of size 4718592 2018-11-15 17:00:15.036507: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f880700 of size 2048 2018-11-15 17:00:15.036517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f880f00 of size 2048 2018-11-15 17:00:15.036526: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f881700 of size 524288 2018-11-15 17:00:15.036535: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f901700 of size 256 2018-11-15 17:00:15.036544: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f901800 of size 1024 2018-11-15 17:00:15.036553: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f901c00 of size 1024 2018-11-15 17:00:15.036562: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960f902000 of size 4718592 2018-11-15 17:00:15.036571: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960fd82000 of size 256 2018-11-15 17:00:15.036580: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960fd82100 of size 768 2018-11-15 17:00:15.036588: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960fd82400 of size 1280 2018-11-15 17:00:15.036598: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f960fd82900 of size 354418688 2018-11-15 17:00:15.036680: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9624f82900 of size 354418688 2018-11-15 17:00:15.036691: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f963a182900 of size 354418688 2018-11-15 17:00:15.036701: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f964f382900 of size 88604672 2018-11-15 17:00:15.036711: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f9654802900 of size 257480448 2018-11-15 17:00:15.036720: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size: 2018-11-15 17:00:15.036736: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 264 Chunks of size 256 totalling 66.0KiB 2018-11-15 17:00:15.036749: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 45 Chunks of size 512 totalling 22.5KiB 2018-11-15 17:00:15.036760: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 768 totalling 768B 2018-11-15 17:00:15.036770: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 58 Chunks of size 1024 totalling 58.0KiB 2018-11-15 17:00:15.036781: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 1280 totalling 3.8KiB 2018-11-15 17:00:15.036792: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1536 totalling 1.5KiB 2018-11-15 17:00:15.036803: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 70 Chunks of size 2048 totalling 140.0KiB 2018-11-15 17:00:15.036814: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2560 totalling 2.5KiB 2018-11-15 17:00:15.036824: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2816 totalling 2.8KiB 2018-11-15 17:00:15.036834: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3328 totalling 3.2KiB 2018-11-15 17:00:15.036844: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 3584 totalling 21.0KiB 2018-11-15 17:00:15.036855: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 87 Chunks of size 4096 totalling 348.0KiB 2018-11-15 17:00:15.036866: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 32768 totalling 192.0KiB 2018-11-15 17:00:15.036877: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 54272 totalling 318.0KiB 2018-11-15 17:00:15.036887: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 73728 totalling 432.0KiB 2018-11-15 17:00:15.036898: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 108032 totalling 105.5KiB 2018-11-15 17:00:15.036909: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 108288 totalling 740.2KiB 2018-11-15 17:00:15.036920: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 12 Chunks of size 131072 totalling 1.50MiB 2018-11-15 17:00:15.036930: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 163840 totalling 960.0KiB 2018-11-15 17:00:15.036940: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 11 Chunks of size 294912 totalling 3.09MiB 2018-11-15 17:00:15.036951: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 11 Chunks of size 524288 totalling 5.50MiB 2018-11-15 17:00:15.036962: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 11 Chunks of size 1179648 totalling 12.38MiB 2018-11-15 17:00:15.036972: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 11 Chunks of size 2097152 totalling 22.00MiB 2018-11-15 17:00:15.036983: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 16 Chunks of size 4718592 totalling 72.00MiB 2018-11-15 17:00:15.036993: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 15 Chunks of size 18874368 totalling 270.00MiB 2018-11-15 17:00:15.037004: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 27220480 totalling 25.96MiB 2018-11-15 17:00:15.037016: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 33226752 totalling 31.69MiB 2018-11-15 17:00:15.037026: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 11 Chunks of size 37748736 totalling 396.00MiB 2018-11-15 17:00:15.037037: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 47185920 totalling 270.00MiB 2018-11-15 17:00:15.037623: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 88604672 totalling 84.50MiB 2018-11-15 17:00:15.037638: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 257480448 totalling 245.55MiB 2018-11-15 17:00:15.037646: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 354418688 totalling 1.65GiB 2018-11-15 17:00:15.037652: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 3.06GiB 2018-11-15 17:00:15.037661: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit: 3285778432 InUse: 3285719040 MaxInUse: 3285719808 NumAllocs: 842 MaxAllocSize: 872216064

2018-11-15 17:00:15.037698: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **xx 2018-11-15 17:00:15.037737: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at conv_ops.cc:735 : Resource exhausted: OOM when allocating tensor with shape[64,32,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[64,32,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node model_1/conv_2/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@train...propFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model_1/max_pooling2d_1/MaxPool, conv_2/kernel/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train.py", line 101, in main(args) File "train.py", line 97, in main debug = config['train']['debug']) File "/home/tong/keras-yolo2/frontend.py", line 336, in train max_queue_size = 8)
File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 87, in wrapper return func(*args, kwargs) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/engine/training.py", line 2042, in fit_generator class_weight=class_weight) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/engine/training.py", line 1762, in train_on_batch outputs = self.train_function(ins) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2273, in call self.session_kwargs) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[64,32,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node model_1/conv_2/convolution (defined at /home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:3164) = Conv2D[T=DT_FLOAT, _class=["loc:@train...propFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model_1/max_pooling2d_1/MaxPool, conv_2/kernel/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'model_1/conv_2/convolution', defined at: File "train.py", line 101, in main(args) File "train.py", line 70, in main anchors = config['model']['anchors']) File "/home/tong/keras-yolo2/frontend.py", line 60, in init features = self.feature_extractor.extract(input_image) File "/home/tong/keras-yolo2/backend.py", line 34, in extract return self.feature_extractor(input_image) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/engine/topology.py", line 602, in call output = self.call(inputs, kwargs) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/engine/topology.py", line 2058, in call outputtensors, , _ = self.run_internal_graph(inputs, masks) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/engine/topology.py", line 2209, in run_internal_graph output_tensors = _to_list(layer.call(computed_tensor, kwargs)) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/layers/convolutional.py", line 164, in call dilation_rate=self.dilation_rate) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 3164, in conv2d data_format='NHWC') File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 780, in convolution return op(input, filter) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 868, in call return self.conv_op(inp, filter) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 520, in call return self.call(inp, filter) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 204, in call name=self.name) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 957, in conv2d data_format=data_format, dilations=dilations, name=name) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/home/tong/keras-yolo2/venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[64,32,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node model_1/conv_2/convolution (defined at /home/tong/keras-yolo2/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:3164) = Conv2D[T=DT_FLOAT, _class=["loc:@train...propFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model_1/max_pooling2d_1/MaxPool, conv_2/kernel/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

alice17 commented 5 years ago

You allocated too much memory, try to reduce the batch size.

OctaM commented 5 years ago

I tried with batch_size=2 and it works, but is there any way to make it functional with a bigger batch size?

EDIT: And why a batch size as 16, for training, needs so much memory? It just loads 16 images at once and feed them all through the network, right?

danFromTelAviv commented 5 years ago

Depending on the network - it can many layers with a large latent space. networks can very quickly reach gigabytes of ram. If you want the updates to happen only after 16 images have been propagated in the network you can edit the optimizer you are using to accumulate the gradients over multiple batches and update every n batches. look at this for reference A larger momentum term + lower learning rate can have a very similar effect.