2018-04-12 10:47:08.094678: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-04-12 10:47:08.094730: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-04-12 10:47:08.094744: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-04-12 10:47:08.094758: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-04-12 10:47:08.094772: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2018-04-12 10:47:08.352114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.2155
pciBusID 0000:03:00.0
Total memory: 11.92GiB
Free memory: 222.94MiB
2018-04-12 10:47:08.352169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0
2018-04-12 10:47:08.352181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y
2018-04-12 10:47:08.352205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
2018-04-12 10:47:21.296425: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 27.00MiB. Current allocation summary follows.
2018-04-12 10:47:21.296492: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296501: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296510: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296518: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296527: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296537: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296546: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296555: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296563: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296571: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296579: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296587: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296596: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296604: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296613: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296621: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296631: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 1, Chunks in use: 0 22.94MiB allocated for chunks. 13.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296640: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296648: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296657: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296665: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.296673: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 27.00MiB was 16.00MiB, Chunk State:
2018-04-12 10:47:21.296689: I tensorflow/core/common_runtime/bfc_allocator.cc:666] Size: 22.94MiB | Requested Size: 13.50MiB | in_use: 0, prev: Size: 256B | Requested Size: 4B | in_use: 1
2018-04-12 10:47:21.296698: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0000 of size 1280
2018-04-12 10:47:21.296705: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0500 of size 256
2018-04-12 10:47:21.296711: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0600 of size 256
2018-04-12 10:47:21.296717: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x420a3c0700 of size 24049920
2018-04-12 10:47:21.296722: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size:
2018-04-12 10:47:21.296730: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 256 totalling 512B
2018-04-12 10:47:21.296738: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB
2018-04-12 10:47:21.296745: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.8KiB
2018-04-12 10:47:21.296757: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
Limit: 24051712
InUse: 1792
MaxInUse: 24051712
NumAllocs: 20
MaxAllocSize: 24049920
2018-04-12 10:47:21.296769: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *___
2018-04-12 10:47:21.300202: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 27.00MiB. Current allocation summary follows.
2018-04-12 10:47:21.300224: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300234: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300242: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300250: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300258: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300266: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300274: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300282: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300290: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300298: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300308: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300315: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300323: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300331: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300340: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300348: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300358: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 1, Chunks in use: 0 22.94MiB allocated for chunks. 13.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300368: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300378: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300385: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300394: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.300403: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 27.00MiB was 16.00MiB, Chunk State:
2018-04-12 10:47:21.300414: I tensorflow/core/common_runtime/bfc_allocator.cc:666] Size: 22.94MiB | Requested Size: 13.50MiB | in_use: 0, prev: Size: 256B | Requested Size: 4B | in_use: 1
2018-04-12 10:47:21.300422: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0000 of size 1280
2018-04-12 10:47:21.300429: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0500 of size 256
2018-04-12 10:47:21.300436: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0600 of size 256
2018-04-12 10:47:21.300442: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x420a3c0700 of size 24049920
2018-04-12 10:47:21.300449: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size:
2018-04-12 10:47:21.300457: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 256 totalling 512B
2018-04-12 10:47:21.300465: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB
2018-04-12 10:47:21.300475: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.8KiB
2018-04-12 10:47:21.300483: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
Limit: 24051712
InUse: 1792
MaxInUse: 24051712
NumAllocs: 20
MaxAllocSize: 24049920
2018-04-12 10:47:21.300492: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *___
2018-04-12 10:47:21.302355: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 27.00MiB. Current allocation summary follows.
2018-04-12 10:47:21.302386: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302394: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302402: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302412: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302420: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302428: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302436: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302444: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302453: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302461: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302469: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302476: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302484: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302492: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302500: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302508: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302520: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 1, Chunks in use: 0 22.94MiB allocated for chunks. 13.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302528: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302536: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302545: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302553: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.302564: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 27.00MiB was 16.00MiB, Chunk State:
2018-04-12 10:47:21.302576: I tensorflow/core/common_runtime/bfc_allocator.cc:666] Size: 22.94MiB | Requested Size: 13.50MiB | in_use: 0, prev: Size: 256B | Requested Size: 4B | in_use: 1
2018-04-12 10:47:21.302584: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0000 of size 1280
2018-04-12 10:47:21.302589: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0500 of size 256
2018-04-12 10:47:21.302596: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0600 of size 256
2018-04-12 10:47:21.302600: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x420a3c0700 of size 24049920
2018-04-12 10:47:21.302607: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size:
2018-04-12 10:47:21.302613: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 256 totalling 512B
2018-04-12 10:47:21.302620: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB
2018-04-12 10:47:21.302627: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.8KiB
2018-04-12 10:47:21.302635: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
Limit: 24051712
InUse: 1792
MaxInUse: 24051712
NumAllocs: 20
MaxAllocSize: 24049920
2018-04-12 10:47:21.302644: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *___
2018-04-12 10:47:21.911379: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 64.00MiB. Current allocation summary follows.
2018-04-12 10:47:21.911405: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911414: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911421: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911428: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911436: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911444: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911453: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911463: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911473: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911482: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911492: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911500: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911508: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911517: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911525: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911533: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911545: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 1, Chunks in use: 0 22.94MiB allocated for chunks. 13.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911554: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911564: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911572: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911580: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.911589: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 64.00MiB was 64.00MiB, Chunk State:
2018-04-12 10:47:21.911596: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0000 of size 1280
2018-04-12 10:47:21.911604: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0500 of size 256
2018-04-12 10:47:21.911611: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0600 of size 256
2018-04-12 10:47:21.911619: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x420a3c0700 of size 24049920
2018-04-12 10:47:21.911626: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size:
2018-04-12 10:47:21.911634: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 256 totalling 512B
2018-04-12 10:47:21.911642: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB
2018-04-12 10:47:21.911649: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.8KiB
2018-04-12 10:47:21.911659: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
Limit: 24051712
InUse: 1792
MaxInUse: 24051712
NumAllocs: 20
MaxAllocSize: 24049920
2018-04-12 10:47:21.911669: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *___
2018-04-12 10:47:21.924118: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 128.00MiB. Current allocation summary follows.
2018-04-12 10:47:21.924160: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924177: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924194: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924211: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924227: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924242: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924257: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924272: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924287: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924302: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924317: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924332: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924347: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924362: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924377: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924392: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924412: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 1, Chunks in use: 0 22.94MiB allocated for chunks. 13.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924428: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924443: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924458: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924473: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-12 10:47:21.924491: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 128.00MiB was 128.00MiB, Chunk State:
2018-04-12 10:47:21.924505: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0000 of size 1280
2018-04-12 10:47:21.924517: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0500 of size 256
2018-04-12 10:47:21.924530: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0600 of size 256
2018-04-12 10:47:21.924541: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x420a3c0700 of size 24049920
2018-04-12 10:47:21.924552: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size:
2018-04-12 10:47:21.924564: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 256 totalling 512B
2018-04-12 10:47:21.924580: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB
2018-04-12 10:47:21.924594: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.8KiB
2018-04-12 10:47:21.924610: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
Limit: 24051712
InUse: 1792
MaxInUse: 24051712
NumAllocs: 20
MaxAllocSize: 24049920
2018-04-12 10:47:21.924628: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *___
Traceback (most recent call last):
File "train_c3d_ucf101.py", line 261, in
tf.app.run()
File "/home/lyynew/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train_c3d_ucf101.py", line 258, in main
run_training()
File "train_c3d_ucf101.py", line 204, in run_training
sess.run(init)
File "/home/lyynew/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 778, in run
run_metadata_ptr)
File "/home/lyynew/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "/home/lyynew/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "/home/lyynew/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[Node: cond_6/Switch_1/_157 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_390_cond_6/Switch_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
2018-04-12 10:47:08.094678: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2018-04-12 10:47:08.094730: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2018-04-12 10:47:08.094744: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2018-04-12 10:47:08.094758: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2018-04-12 10:47:08.094772: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. 2018-04-12 10:47:08.352114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate (GHz) 1.2155 pciBusID 0000:03:00.0 Total memory: 11.92GiB Free memory: 222.94MiB 2018-04-12 10:47:08.352169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 2018-04-12 10:47:08.352181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y 2018-04-12 10:47:08.352205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0) 2018-04-12 10:47:21.296425: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 27.00MiB. Current allocation summary follows. 2018-04-12 10:47:21.296492: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296501: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296510: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296518: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296527: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296537: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296546: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296555: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296563: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296571: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296579: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296587: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296596: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296604: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296613: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296621: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296631: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 1, Chunks in use: 0 22.94MiB allocated for chunks. 13.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296640: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296648: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296657: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296665: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.296673: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 27.00MiB was 16.00MiB, Chunk State: 2018-04-12 10:47:21.296689: I tensorflow/core/common_runtime/bfc_allocator.cc:666] Size: 22.94MiB | Requested Size: 13.50MiB | in_use: 0, prev: Size: 256B | Requested Size: 4B | in_use: 1 2018-04-12 10:47:21.296698: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0000 of size 1280 2018-04-12 10:47:21.296705: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0500 of size 256 2018-04-12 10:47:21.296711: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0600 of size 256 2018-04-12 10:47:21.296717: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x420a3c0700 of size 24049920 2018-04-12 10:47:21.296722: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size: 2018-04-12 10:47:21.296730: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 256 totalling 512B 2018-04-12 10:47:21.296738: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB 2018-04-12 10:47:21.296745: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.8KiB 2018-04-12 10:47:21.296757: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats: Limit: 24051712 InUse: 1792 MaxInUse: 24051712 NumAllocs: 20 MaxAllocSize: 24049920
2018-04-12 10:47:21.296769: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *___ 2018-04-12 10:47:21.300202: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 27.00MiB. Current allocation summary follows. 2018-04-12 10:47:21.300224: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300234: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300242: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300250: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300258: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300266: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300274: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300282: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300290: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300298: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300308: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300315: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300323: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300331: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300340: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300348: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300358: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 1, Chunks in use: 0 22.94MiB allocated for chunks. 13.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300368: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300378: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300385: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300394: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.300403: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 27.00MiB was 16.00MiB, Chunk State: 2018-04-12 10:47:21.300414: I tensorflow/core/common_runtime/bfc_allocator.cc:666] Size: 22.94MiB | Requested Size: 13.50MiB | in_use: 0, prev: Size: 256B | Requested Size: 4B | in_use: 1 2018-04-12 10:47:21.300422: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0000 of size 1280 2018-04-12 10:47:21.300429: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0500 of size 256 2018-04-12 10:47:21.300436: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0600 of size 256 2018-04-12 10:47:21.300442: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x420a3c0700 of size 24049920 2018-04-12 10:47:21.300449: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size: 2018-04-12 10:47:21.300457: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 256 totalling 512B 2018-04-12 10:47:21.300465: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB 2018-04-12 10:47:21.300475: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.8KiB 2018-04-12 10:47:21.300483: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats: Limit: 24051712 InUse: 1792 MaxInUse: 24051712 NumAllocs: 20 MaxAllocSize: 24049920
2018-04-12 10:47:21.300492: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *___ 2018-04-12 10:47:21.302355: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 27.00MiB. Current allocation summary follows. 2018-04-12 10:47:21.302386: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302394: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302402: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302412: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302420: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302428: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302436: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302444: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302453: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302461: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302469: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302476: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302484: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302492: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302500: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302508: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302520: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 1, Chunks in use: 0 22.94MiB allocated for chunks. 13.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302528: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302536: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302545: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302553: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.302564: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 27.00MiB was 16.00MiB, Chunk State: 2018-04-12 10:47:21.302576: I tensorflow/core/common_runtime/bfc_allocator.cc:666] Size: 22.94MiB | Requested Size: 13.50MiB | in_use: 0, prev: Size: 256B | Requested Size: 4B | in_use: 1 2018-04-12 10:47:21.302584: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0000 of size 1280 2018-04-12 10:47:21.302589: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0500 of size 256 2018-04-12 10:47:21.302596: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0600 of size 256 2018-04-12 10:47:21.302600: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x420a3c0700 of size 24049920 2018-04-12 10:47:21.302607: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size: 2018-04-12 10:47:21.302613: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 256 totalling 512B 2018-04-12 10:47:21.302620: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB 2018-04-12 10:47:21.302627: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.8KiB 2018-04-12 10:47:21.302635: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats: Limit: 24051712 InUse: 1792 MaxInUse: 24051712 NumAllocs: 20 MaxAllocSize: 24049920
2018-04-12 10:47:21.302644: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *___ 2018-04-12 10:47:21.911379: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 64.00MiB. Current allocation summary follows. 2018-04-12 10:47:21.911405: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911414: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911421: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911428: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911436: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911444: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911453: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911463: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911473: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911482: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911492: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911500: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911508: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911517: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911525: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911533: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911545: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 1, Chunks in use: 0 22.94MiB allocated for chunks. 13.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911554: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911564: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911572: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911580: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.911589: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 64.00MiB was 64.00MiB, Chunk State: 2018-04-12 10:47:21.911596: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0000 of size 1280 2018-04-12 10:47:21.911604: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0500 of size 256 2018-04-12 10:47:21.911611: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0600 of size 256 2018-04-12 10:47:21.911619: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x420a3c0700 of size 24049920 2018-04-12 10:47:21.911626: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size: 2018-04-12 10:47:21.911634: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 256 totalling 512B 2018-04-12 10:47:21.911642: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB 2018-04-12 10:47:21.911649: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.8KiB 2018-04-12 10:47:21.911659: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats: Limit: 24051712 InUse: 1792 MaxInUse: 24051712 NumAllocs: 20 MaxAllocSize: 24049920
2018-04-12 10:47:21.911669: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *___ 2018-04-12 10:47:21.924118: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 128.00MiB. Current allocation summary follows. 2018-04-12 10:47:21.924160: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924177: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924194: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924211: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924227: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924242: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924257: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924272: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924287: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924302: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924317: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924332: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924347: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924362: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924377: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924392: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924412: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 1, Chunks in use: 0 22.94MiB allocated for chunks. 13.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924428: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924443: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924458: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924473: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-04-12 10:47:21.924491: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 128.00MiB was 128.00MiB, Chunk State: 2018-04-12 10:47:21.924505: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0000 of size 1280 2018-04-12 10:47:21.924517: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0500 of size 256 2018-04-12 10:47:21.924530: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x420a3c0600 of size 256 2018-04-12 10:47:21.924541: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x420a3c0700 of size 24049920 2018-04-12 10:47:21.924552: I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size: 2018-04-12 10:47:21.924564: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 256 totalling 512B 2018-04-12 10:47:21.924580: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB 2018-04-12 10:47:21.924594: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.8KiB 2018-04-12 10:47:21.924610: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats: Limit: 24051712 InUse: 1792 MaxInUse: 24051712 NumAllocs: 20 MaxAllocSize: 24049920
2018-04-12 10:47:21.924628: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *___ Traceback (most recent call last): File "train_c3d_ucf101.py", line 261, in
tf.app.run()
File "/home/lyynew/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train_c3d_ucf101.py", line 258, in main
run_training()
File "train_c3d_ucf101.py", line 204, in run_training
sess.run(init)
File "/home/lyynew/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 778, in run
run_metadata_ptr)
File "/home/lyynew/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "/home/lyynew/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "/home/lyynew/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[Node: cond_6/Switch_1/_157 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_390_cond_6/Switch_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]