search

microsoft / MMdnn

MMdnn is a set of tools to help users inter-operate among different deep learning frameworks. E.g. model conversion and visualization. Convert models between Caffe, Keras, MXNet, Tensorflow, CNTK, PyTorch Onnx and CoreML.
MIT License
5.79k stars 964 forks source link

Keras to IR: ResourceExhaustedError: OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc #575

Open missingdaysqxy opened 5 years ago

missingdaysqxy commented 5 years ago

I want convert this model into pytorch, I downloaded the converted weight file Keras_FCN8s_face_seg_YuvalNirkin.h5. Then I add 

jsonstr=model.to_json()
with open("Keras_FCN8s_Model.json","w+") as fp:
    fp.write(jsonstr)

at the end of FCN8s_keras_demo.ipynb, it works well and I got a keras json model. Then I try to convert keras into IR, got an error as title. I don't know how to do. Here's my environment:

Platform: Red Hat 4.8.5-11 with 4 Nvidia Tesla P40 GPUs Python version: Python 3.6.5: Anaconda, Inc. MMdnn: installed from master (0bf2f4d) (pip install -U git+https://github.com/Microsoft/MMdnn.git@master), version 0.2.3 Source framework with version : Keras 2.2.4 GPU from pip, with backend tensorflow-gpu 1.12.0 Destination framework with version : IR Pre-trained model path : local disk

Running scripts: mmtoir -f keras -d fcn8s_ir -w Keras_FCN8s_face_seg_YuvalNirkin.h5 -n Keras_FCN8s_Model.json

Output Messages: 

[liuqixuan_i@ml-gpu-ser544 face-segmentation-keras]$ mmtoir -f keras -d fcn8s_ir -w Keras_FCN8s_face_seg_YuvalNirkin.h5 -n Keras_FCN8s_Model.json
/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
2019-01-29 16:21:48.091209: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-01-29 16:21:49.432212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:02:00.0
totalMemory: 22.38GiB freeMemory: 839.12MiB
2019-01-29 16:21:49.625955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:03:00.0
totalMemory: 22.38GiB freeMemory: 21.98GiB
2019-01-29 16:21:49.822598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 2 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:83:00.0
totalMemory: 22.38GiB freeMemory: 21.98GiB
2019-01-29 16:21:50.031561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 3 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:84:00.0
totalMemory: 22.38GiB freeMemory: 21.98GiB
2019-01-29 16:21:50.031966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1, 2, 3
2019-01-29 16:21:51.567269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-01-29 16:21:51.567336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1 2 3
2019-01-29 16:21:51.567352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y N N
2019-01-29 16:21:51.567364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N N N
2019-01-29 16:21:51.567376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 2:   N N N Y
2019-01-29 16:21:51.567386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 3:   N N Y N
2019-01-29 16:21:51.569215: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 550 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-01-29 16:21:51.570124: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 21319 MB memory) -> physical GPU (device: 1, name: Tesla P40, pci bus id: 0000:03:00.0, compute capability: 6.1)
2019-01-29 16:21:51.570908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 21319 MB memory) -> physical GPU (device: 2, name: Tesla P40, pci bus id: 0000:83:00.0, compute capability: 6.1)
2019-01-29 16:21:51.571207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 21319 MB memory) -> physical GPU (device: 3, name: Tesla P40, pci bus id: 0000:84:00.0, compute capability: 6.1)
2019-01-29 16:22:01.794290: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.0KiB.  Current allocation summary follows.
2019-01-29 16:22:01.794355: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:01.794370: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:01.794381: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794393: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794404: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:01.794414: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794425: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794435: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794444: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794455: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794467: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794478: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794488: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:01.794499: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:01.794518: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:01.794529: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:01.794539: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794549: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794558: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794568: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794580: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:01.794590: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 16.0KiB was 16.0KiB, Chunk State:
2019-01-29 16:22:01.794607: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:01.794614: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:01.794622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:01.794628: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:01.794635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:01.794641: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:01.794647: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:01.794654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:01.794661: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:01.794668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:01.794674: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:01.794680: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:01.794687: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:01.794693: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:01.794700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:01.794706: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:01.794732: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:01.794740: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:01.794746: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:01.794753: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:01.794759: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:01.794765: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:01.794771: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:01.794778: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:01.794784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:01.794790: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:01.794796: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:01.794802: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:01.794809: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:01.794815: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:01.794821: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:01.794827: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:01.794833: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:01.794839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:01.794845: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:01.794852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:01.794858: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:01.794864: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:01.794871: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:01.794877: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:01.794884: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:01.794890: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:01.794897: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:01.794903: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:01.794910: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:01.794916: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:01.794922: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:01.794929: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:01.794935: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:01.794941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:01.794948: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:01.794954: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:01.794961: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:01.794967: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:01.795052: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:01.795060: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:01.795067: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:01.795073: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:01.795080: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:01.795086: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:01.795092: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:01.795099: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:01.795105: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:01.795113: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:01.795121: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:01.795131: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:01.795139: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:01.795147: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:01.795154: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:01.795162: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:01.795169: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:01.795177: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:01.795184: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:01.795192: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:01.795200: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:01.795207: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:01.795214: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:01.795222: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:01.795229: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:01.795237: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:01.795245: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:01.795256: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:01.795273: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:01.795316: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:22:11.795511: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.0KiB.  Current allocation summary follows.
2019-01-29 16:22:11.795574: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:11.795589: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:11.795599: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795610: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795621: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:11.795631: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795642: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795652: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795661: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795672: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795684: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795695: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795705: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:11.795715: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:11.795725: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:11.795736: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:11.795746: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795755: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795764: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795773: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795784: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:11.795794: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 16.0KiB was 16.0KiB, Chunk State:
2019-01-29 16:22:11.795804: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:11.795811: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:11.795818: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:11.795824: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:11.795831: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:11.795837: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:11.795844: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:11.795850: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:11.795857: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:11.795864: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:11.795870: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:11.795876: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:11.795883: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:11.795889: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:11.795896: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:11.795902: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:11.795966: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:11.795974: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:11.795981: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:11.795987: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:11.795994: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:11.796000: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:11.796006: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:11.796013: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:11.796019: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:11.796025: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:11.796031: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:11.796038: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:11.796044: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:11.796050: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:11.796056: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:11.796062: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:11.796068: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:11.796075: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:11.796081: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:11.796087: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:11.796093: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:11.796100: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:11.796106: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:11.796112: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:11.796119: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:11.796125: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:11.796131: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:11.796138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:11.796144: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:11.796151: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:11.796157: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:11.796163: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:11.796170: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:11.796176: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:11.796182: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:11.796189: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:11.796195: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:11.796202: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:11.796287: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:11.796296: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:11.796302: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:11.796309: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:11.796315: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:11.796322: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:11.796328: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:11.796334: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:11.796341: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:11.796349: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:11.796356: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:11.796366: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:11.796374: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:11.796381: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:11.796389: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:11.796396: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:11.796404: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:11.796411: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:11.796419: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:11.796427: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:11.796435: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:11.796442: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:11.796450: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:11.796457: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:11.796465: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:11.796473: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:11.796480: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:11.796492: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:11.796515: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:11.796547: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:22:21.796736: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 64.00MiB.  Current allocation summary follows.
2019-01-29 16:22:21.796796: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:21.796810: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:21.796822: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796833: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796844: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:21.796854: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796866: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796875: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796885: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796897: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796908: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796919: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796929: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:21.796939: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:21.796949: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:21.796960: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:21.796970: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796979: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796988: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796997: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.797008: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:21.797018: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 64.00MiB was 64.00MiB, Chunk State:
2019-01-29 16:22:21.797028: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:21.797035: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:21.797042: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:21.797048: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:21.797055: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:21.797061: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:21.797068: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:21.797074: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:21.797081: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:21.797087: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:21.797094: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:21.797100: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:21.797106: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:21.797113: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:21.797120: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:21.797126: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:21.797150: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:21.797157: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:21.797164: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:21.797170: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:21.797176: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:21.797182: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:21.797188: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:21.797195: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:21.797201: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:21.797207: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:21.797213: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:21.797219: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:21.797225: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:21.797232: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:21.797238: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:21.797244: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:21.797250: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:21.797256: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:21.797262: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:21.797269: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:21.797275: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:21.797281: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:21.797287: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:21.797293: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:21.797300: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:21.797306: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:21.797313: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:21.797319: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:21.797325: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:21.797332: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:21.797338: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:21.797344: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:21.797350: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:21.797357: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:21.797363: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:21.797369: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:21.797376: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:21.797382: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:21.797469: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:21.797478: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:21.797484: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:21.797490: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:21.797497: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:21.797509: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:21.797515: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:21.797522: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:21.797529: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:21.797537: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:21.797543: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:21.797553: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:21.797561: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:21.797569: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:21.797576: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:21.797584: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:21.797592: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:21.797599: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:21.797607: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:21.797615: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:21.797622: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:21.797630: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:21.797637: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:21.797644: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:21.797652: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:21.797660: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:21.797668: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:21.797679: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:21.797696: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:21.797728: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[1,1,4096,4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:22:31.797915: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 336.0KiB.  Current allocation summary follows.
2019-01-29 16:22:31.797981: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:31.797995: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:31.798006: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798017: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798028: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:31.798038: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798048: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798058: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798067: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798078: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798089: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798100: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798111: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:31.798122: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:31.798132: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:31.798143: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:31.798152: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798162: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798171: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798180: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798191: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:31.798201: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 336.0KiB was 256.0KiB, Chunk State:
2019-01-29 16:22:31.798212: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:31.798219: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:31.798225: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:31.798232: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:31.798238: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:31.798244: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:31.798251: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:31.798257: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:31.798264: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:31.798271: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:31.798277: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:31.798283: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:31.798289: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:31.798296: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:31.798302: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:31.798308: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:31.798339: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:31.798346: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:31.798353: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:31.798359: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:31.798365: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:31.798372: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:31.798378: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:31.798384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:31.798390: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:31.798397: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:31.798403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:31.798409: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:31.798415: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:31.798421: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:31.798427: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:31.798433: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:31.798439: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:31.798445: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:31.798451: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:31.798458: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:31.798464: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:31.798470: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:31.798476: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:31.798482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:31.798489: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:31.798495: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:31.798513: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:31.798520: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:31.798527: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:31.798533: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:31.798540: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:31.798546: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:31.798552: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:31.798559: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:31.798565: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:31.798572: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:31.798578: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:31.798584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:31.798671: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:31.798680: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:31.798686: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:31.798692: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:31.798699: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:31.798705: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:31.798711: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:31.798718: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:31.798724: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:31.798731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:31.798739: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:31.798748: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:31.798756: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:31.798764: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:31.798771: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:31.798779: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:31.798786: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:31.798793: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:31.798801: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:31.798809: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:31.798817: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:31.798824: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:31.798832: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:31.798839: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:31.798847: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:31.798854: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:31.798862: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:31.798873: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:31.798890: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:31.798922: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[1,1,4096,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:22:41.799103: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 21.0KiB.  Current allocation summary follows.
2019-01-29 16:22:41.799165: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:41.799179: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:41.799190: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799202: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799212: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:41.799222: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799234: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799243: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799253: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799264: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799275: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799286: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799297: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:41.799307: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:41.799317: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:41.799328: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:41.799338: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799347: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799356: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799365: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799377: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:41.799386: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 21.0KiB was 16.0KiB, Chunk State:
2019-01-29 16:22:41.799397: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:41.799404: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:41.799411: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:41.799417: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:41.799424: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:41.799430: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:41.799436: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:41.799442: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:41.799449: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:41.799456: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:41.799462: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:41.799468: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:41.799475: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:41.799481: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:41.799488: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:41.799494: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:41.799529: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:41.799537: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:41.799543: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:41.799550: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:41.799556: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:41.799562: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:41.799568: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:41.799575: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:41.799581: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:41.799587: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:41.799593: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:41.799600: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:41.799606: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:41.799612: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:41.799618: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:41.799624: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:41.799631: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:41.799637: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:41.799643: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:41.799649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:41.799655: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:41.799662: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:41.799668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:41.799674: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:41.799681: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:41.799687: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:41.799694: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:41.799700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:41.799706: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:41.799713: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:41.799719: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:41.799725: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:41.799731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:41.799738: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:41.799744: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:41.799751: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:41.799757: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:41.799763: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:41.799847: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:41.799856: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:41.799862: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:41.799868: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:41.799875: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:41.799881: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:41.799888: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:41.799894: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:41.799902: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:41.799909: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:41.799916: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:41.799925: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:41.799934: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:41.799941: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:41.799949: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:41.799956: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:41.799964: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:41.799971: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:41.799979: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:41.799987: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:41.799995: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:41.800003: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:41.800010: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:41.800018: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:41.800025: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:41.800033: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:41.800041: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:41.800052: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:41.800070: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:41.800103: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[1,1,256,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:22:51.800275: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 84B.  Current allocation summary follows.
2019-01-29 16:22:51.800344: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:51.800358: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:51.800369: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800381: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800392: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:51.800402: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800413: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800423: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800433: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800444: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800456: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800466: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800477: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:51.800487: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:51.800497: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:51.800518: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:51.800529: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800539: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800548: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800558: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800569: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:51.800578: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 256B was 256B, Chunk State:
2019-01-29 16:22:51.800589: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:51.800596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:51.800603: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:51.800609: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:51.800615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:51.800622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:51.800628: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:51.800634: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:51.800641: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:51.800648: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:51.800654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:51.800660: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:51.800666: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:51.800673: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:51.800679: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:51.800686: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:51.800717: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:51.800725: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:51.800731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:51.800738: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:51.800744: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:51.800750: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:51.800756: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:51.800762: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:51.800769: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:51.800775: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:51.800781: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:51.800787: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:51.800793: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:51.800799: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:51.800805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:51.800812: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:51.800818: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:51.800824: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:51.800830: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:51.800836: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:51.800842: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:51.800849: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:51.800855: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:51.800861: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:51.800868: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:51.800874: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:51.800880: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:51.800886: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:51.800893: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:51.800899: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:51.800906: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:51.800912: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:51.800918: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:51.800924: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:51.800931: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:51.800937: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:51.800943: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:51.800950: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:51.801032: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:51.801040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:51.801046: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:51.801052: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:51.801059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:51.801065: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:51.801072: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:51.801078: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:51.801084: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:51.801092: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:51.801099: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:51.801109: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:51.801117: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:51.801124: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:51.801131: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:51.801139: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:51.801146: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:51.801154: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:51.801161: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:51.801169: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:51.801176: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:51.801184: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:51.801191: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:51.801198: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:51.801206: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:51.801214: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:51.801222: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:51.801233: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:51.801250: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:51.801282: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:01.801458: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 84B.  Current allocation summary follows.
2019-01-29 16:23:01.801533: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:01.801548: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:01.801559: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801572: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801582: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:01.801592: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801603: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801613: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801622: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801634: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801645: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801656: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801667: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:01.801677: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:01.801688: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:01.801699: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:01.801709: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801718: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801728: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801737: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801748: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:01.801757: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 256B was 256B, Chunk State:
2019-01-29 16:23:01.801768: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:01.801775: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:01.801782: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:01.801788: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:01.801795: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:01.801801: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:01.801807: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:01.801814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:01.801821: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:01.801827: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:01.801833: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:01.801840: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:01.801846: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:01.801853: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:01.801859: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:01.801866: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:01.801899: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:01.801907: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:01.801913: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:01.801920: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:01.801926: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:01.801932: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:01.801938: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:01.801945: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:01.801951: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:01.801957: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:01.801964: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:01.801970: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:01.801976: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:01.801982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:01.801989: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:01.801995: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:01.802001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:01.802007: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:01.802013: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:01.802020: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:01.802026: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:01.802032: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:01.802038: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:01.802045: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:01.802051: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:01.802058: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:01.802064: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:01.802070: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:01.802077: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:01.802083: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:01.802090: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:01.802096: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:01.802102: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:01.802109: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:01.802115: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:01.802122: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:01.802128: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:01.802135: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:01.802222: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:01.802231: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:01.802237: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:01.802243: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:01.802250: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:01.802257: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:01.802263: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:01.802269: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:01.802277: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:01.802285: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:01.802291: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:01.802301: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:01.802310: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:01.802317: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:01.802324: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:01.802332: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:01.802340: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:01.802347: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:01.802355: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:01.802363: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:01.802371: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:01.802378: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:01.802386: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:01.802393: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:01.802401: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:01.802409: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:01.802417: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:01.802428: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:01.802445: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:01.802477: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:11.802660: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 84B.  Current allocation summary follows.
2019-01-29 16:23:11.802728: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:11.802743: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:11.802754: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802766: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802776: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:11.802787: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802798: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802808: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802817: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802828: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802840: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802851: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802861: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:11.802872: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:11.802882: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:11.802893: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:11.802903: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802913: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802922: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802931: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802943: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:11.802952: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 256B was 256B, Chunk State:
2019-01-29 16:23:11.802963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:11.802969: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:11.802976: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:11.802982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:11.802989: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:11.802995: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:11.803001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:11.803008: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:11.803015: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:11.803021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:11.803028: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:11.803034: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:11.803040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:11.803046: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:11.803053: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:11.803059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:11.803088: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:11.803095: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:11.803102: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:11.803108: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:11.803115: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:11.803121: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:11.803127: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:11.803133: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:11.803140: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:11.803146: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:11.803152: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:11.803158: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:11.803164: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:11.803170: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:11.803177: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:11.803183: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:11.803189: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:11.803195: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:11.803201: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:11.803207: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:11.803214: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:11.803220: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:11.803226: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:11.803233: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:11.803239: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:11.803246: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:11.803252: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:11.803258: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:11.803264: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:11.803271: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:11.803277: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:11.803283: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:11.803290: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:11.803296: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:11.803302: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:11.803309: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:11.803315: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:11.803322: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:11.803403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:11.803411: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:11.803417: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:11.803423: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:11.803430: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:11.803436: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:11.803443: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:11.803449: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:11.803456: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:11.803463: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:11.803471: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:11.803480: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:11.803488: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:11.803496: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:11.803511: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:11.803519: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:11.803527: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:11.803534: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:11.803542: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:11.803550: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:11.803558: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:11.803565: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:11.803573: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:11.803580: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:11.803588: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:11.803596: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:11.803604: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:11.803615: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:11.803632: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:11.803664: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:21.803849: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 42.0KiB.  Current allocation summary follows.
2019-01-29 16:23:21.803906: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:21.803920: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:21.803932: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:21.803943: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:21.803954: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:21.803964: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.803975: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:21.803985: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.803994: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.804005: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:21.804017: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:21.804029: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:21.804039: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:21.804050: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:21.804060: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:21.804071: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:21.804081: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.804090: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.804099: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.804108: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.804120: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:21.804129: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 42.0KiB was 32.0KiB, Chunk State:
2019-01-29 16:23:21.804140: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:21.804147: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:21.804154: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:21.804160: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:21.804167: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:21.804173: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:21.804180: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:21.804186: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:21.804193: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:21.804199: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:21.804205: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:21.804212: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:21.804218: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:21.804225: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:21.804231: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:21.804238: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:21.804299: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:21.804307: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:21.804313: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:21.804319: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:21.804326: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:21.804332: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:21.804338: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:21.804344: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:21.804351: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:21.804357: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:21.804363: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:21.804369: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:21.804376: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:21.804382: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:21.804388: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:21.804394: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:21.804400: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:21.804406: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:21.804413: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:21.804419: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:21.804425: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:21.804431: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:21.804437: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:21.804444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:21.804450: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:21.804457: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:21.804463: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:21.804469: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:21.804475: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:21.804482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:21.804488: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:21.804495: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:21.804507: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:21.804515: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:21.804521: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:21.804528: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:21.804534: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:21.804541: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:21.804621: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:21.804629: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:21.804636: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:21.804642: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:21.804649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:21.804655: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:21.804662: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:21.804668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:21.804676: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:21.804683: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:21.804690: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:21.804699: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:21.804707: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:21.804715: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:21.804722: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:21.804730: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:21.804737: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:21.804745: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:21.804753: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:21.804760: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:21.804768: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:21.804775: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:21.804783: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:21.804790: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:21.804798: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:21.804806: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:21.804814: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:21.804825: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:21.804841: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:21.804873: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[1,1,512,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:31.805056: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 441.0KiB.  Current allocation summary follows.
2019-01-29 16:23:31.805120: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:31.805134: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:31.805145: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805157: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805168: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:31.805179: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805190: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805200: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805210: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805222: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805233: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805244: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805256: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:31.805267: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:31.805277: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:31.805288: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:31.805298: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805307: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805316: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805325: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805337: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:31.805347: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 441.0KiB was 256.0KiB, Chunk State:
2019-01-29 16:23:31.805357: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:31.805364: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:31.805371: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:31.805377: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:31.805384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:31.805390: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:31.805396: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:31.805403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:31.805410: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:31.805416: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:31.805423: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:31.805429: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:31.805435: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:31.805441: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:31.805448: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:31.805454: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:31.805477: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:31.805485: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:31.805491: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:31.805498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:31.805514: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:31.805521: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:31.805527: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:31.805534: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:31.805540: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:31.805546: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:31.805553: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:31.805559: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:31.805565: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:31.805572: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:31.805578: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:31.805584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:31.805590: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:31.805596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:31.805602: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:31.805608: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:31.805615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:31.805621: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:31.805627: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:31.805634: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:31.805640: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:31.805647: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:31.805653: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:31.805659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:31.805665: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:31.805672: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:31.805678: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:31.805685: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:31.805691: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:31.805697: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:31.805703: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:31.805710: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:31.805716: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:31.805722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:31.805806: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:31.805814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:31.805820: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:31.805826: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:31.805833: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:31.805839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:31.805846: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:31.805852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:31.805858: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:31.805865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:31.805872: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:31.805883: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:31.805892: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:31.805899: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:31.805907: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:31.805915: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:31.805922: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:31.805930: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:31.805938: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:31.805946: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:31.805953: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:31.805961: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:31.805968: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:31.805975: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:31.805983: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:31.805991: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:31.805999: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:31.806011: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:31.806028: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:31.806060: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[16,16,21,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:41.806240: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 27.6KiB.  Current allocation summary follows.
2019-01-29 16:23:41.806303: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:41.806317: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:41.806328: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806340: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806351: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:41.806361: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806372: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806382: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806392: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806404: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806415: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806426: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806437: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:41.806447: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:41.806458: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:41.806469: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:41.806478: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806488: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806497: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806517: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806530: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:41.806540: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 27.8KiB was 16.0KiB, Chunk State:
2019-01-29 16:23:41.806551: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:41.806558: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:41.806565: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:41.806571: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:41.806578: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:41.806584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:41.806591: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:41.806597: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:41.806604: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:41.806610: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:41.806617: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:41.806623: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:41.806629: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:41.806636: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:41.806642: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:41.806649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:41.806678: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:41.806686: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:41.806692: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:41.806699: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:41.806705: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:41.806711: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:41.806717: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:41.806724: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:41.806730: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:41.806736: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:41.806742: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:41.806748: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:41.806754: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:41.806761: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:41.806767: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:41.806773: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:41.806779: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:41.806785: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:41.806791: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:41.806797: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:41.806803: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:41.806810: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:41.806816: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:41.806822: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:41.806829: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:41.806835: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:41.806842: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:41.806848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:41.806854: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:41.806860: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:41.806867: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:41.806873: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:41.806879: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:41.806886: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:41.806892: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:41.806898: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:41.806905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:41.806911: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:41.806994: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:41.807003: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:41.807010: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:41.807016: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:41.807022: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:41.807029: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:41.807035: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:41.807041: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:41.807048: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:41.807056: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:41.807063: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:41.807073: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:41.807081: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:41.807088: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:41.807096: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:41.807103: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:41.807111: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:41.807118: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:41.807126: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:41.807134: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:41.807142: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:41.807149: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:41.807157: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:41.807164: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:41.807172: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:41.807180: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:41.807188: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:41.807199: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:41.807216: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:41.807248: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[4,4,21,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:51.807434: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 27.6KiB.  Current allocation summary follows.
2019-01-29 16:23:51.807496: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:51.807518: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:51.807530: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807543: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807554: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:51.807564: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807575: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807585: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807595: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807607: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807618: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807629: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807640: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:51.807650: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:51.807660: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:51.807671: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:51.807681: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807691: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807700: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807709: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807720: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:51.807730: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 27.8KiB was 16.0KiB, Chunk State:
2019-01-29 16:23:51.807741: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:51.807748: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:51.807755: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:51.807761: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:51.807767: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:51.807774: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:51.807780: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:51.807787: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:51.807794: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:51.807801: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:51.807807: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:51.807813: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:51.807819: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:51.807826: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:51.807833: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:51.807839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:51.807862: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:51.807869: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:51.807875: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:51.807882: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:51.807888: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:51.807894: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:51.807900: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:51.807907: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:51.807913: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:51.807919: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:51.807926: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:51.807932: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:51.807938: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:51.807944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:51.807950: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:51.807956: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:51.807963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:51.807969: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:51.807975: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:51.807981: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:51.807987: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:51.807994: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:51.808000: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:51.808006: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:51.808013: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:51.808019: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:51.808026: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:51.808032: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:51.808038: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:51.808045: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:51.808051: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:51.808057: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:51.808064: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:51.808070: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:51.808076: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:51.808083: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:51.808089: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:51.808096: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:51.808181: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:51.808189: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:51.808195: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:51.808201: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:51.808208: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:51.808214: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:51.808221: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:51.808227: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:51.808234: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:51.808241: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:51.808248: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:51.808258: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:51.808266: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:51.808274: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:51.808281: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:51.808289: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:51.808296: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:51.808304: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:51.808311: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:51.808319: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:51.808327: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:51.808334: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:51.808342: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:51.808349: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:51.808357: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:51.808365: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:51.808373: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:51.808384: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:51.808401: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:51.808433: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[4,4,21,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node fc6/bias/Assign}} = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fc6/bias, fc6/Const)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/liuqixuan_i/anaconda3/bin/mmtoir", line 11, in <module>
    sys.exit(_main())
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/_script/convertToIR.py", line 192, in _main
    ret = _convert(args)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/_script/convertToIR.py", line 46, in _convert
    parser = Keras2Parser(model)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/keras/keras2_parser.py", line 126, in __init__
    model = self._load_model(model[0], model[1])
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/keras/keras2_parser.py", line 83, in _load_model
    loaded_model.load_weights(model_weight_path)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 1166, in load_weights
    f, self.layers, reshape=reshape)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 1058, in load_weights_from_hdf5_group
    K.batch_set_value(weight_value_tuples)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2470, in batch_set_value
    get_session().run(assign_ops, feed_dict=feed_dict)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 206, in get_session
    session.run(tf.variables_initializer(uninitialized_vars))
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node fc6/bias/Assign (defined at /home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:402)  = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fc6/bias, fc6/Const)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'fc6/bias/Assign', defined at:
  File "/home/liuqixuan_i/anaconda3/bin/mmtoir", line 11, in <module>
    sys.exit(_main())
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/_script/convertToIR.py", line 192, in _main
    ret = _convert(args)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/_script/convertToIR.py", line 46, in _convert
    parser = Keras2Parser(model)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/keras/keras2_parser.py", line 126, in __init__
    model = self._load_model(model[0], model[1])
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/keras/keras2_parser.py", line 78, in _load_model
    'DepthwiseConv2D': layers.DepthwiseConv2D})
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 492, in model_from_json
    return deserialize(config, custom_objects=custom_objects)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
    list(custom_objects.items())))
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 1032, in from_config
    process_node(layer, node_data)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 991, in process_node
    layer(unpack_singleton(input_tensors), **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 431, in __call__
    self.build(unpack_singleton(input_shapes))
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 147, in build
    constraint=self.bias_constraint)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 252, in add_weight
    constraint=constraint)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 402, in variable
    v = tf.Variable(value, dtype=tf.as_dtype(dtype), name=name)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 183, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 146, in _variable_v1_call
    aggregation=aggregation)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 125, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2444, in default_variable_creator
    expected_shape=expected_shape, import_scope=import_scope)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 187, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1329, in __init__
    constraint=constraint)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1481, in _init_from_args
    validate_shape=validate_shape).op
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 221, in assign
    validate_shape=validate_shape)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
    use_locking=use_locking, name=name)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node fc6/bias/Assign (defined at /home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:402)  = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fc6/bias, fc6/Const)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
rainLiuplus commented 5 years ago

Hi @missingdaysqxy , seems that your graphics memory is not enough. Thanks!