error occurs when I train on my own dataset

[mayeeeeeeee]

IMAGE_SHAPE = 2700, 2100, 3 MAX_LABEL = 25 batch_size = 2 num_images_per_id = 1

Console output as follows：

2020-03-24 18:12:19.131661: W tensorflow/core/framework/allocator.cc:122] Allocation of 5825757184 exceeds 10% of system memory. 2020-03-24 18:12:21.766310: W tensorflow/core/framework/allocator.cc:122] Allocation of 5825757184 exceeds 10% of system memory. 2020-03-24 18:12:24.392377: W tensorflow/core/framework/allocator.cc:122] Allocation of 5825757184 exceeds 10% of system memory. 2020-03-24 18:12:26.255581: W tensorflow/core/framework/allocator.cc:122] Allocation of 5825757184 exceeds 10% of system memory. 2020-03-24 18:12:28.144805: W tensorflow/core/framework/allocator.cc:122] Allocation of 5825757184 exceeds 10% of system memory. 2020-03-24 18:13:17.173072: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 5.42GiB. Current allocation summary follows. 2020-03-24 18:13:17.173170: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 89, Chunks in use: 89. 22.2KiB allocated for chunks. 22.2KiB in use in bin. 13.5KiB client-requested in use in bin. 2020-03-24 18:13:17.173214: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2020-03-24 18:13:17.173234: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin. 2020-03-24 18:13:17.173254: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 1, Chunks in use: 1. 3.5KiB allocated for chunks. 3.5KiB in use in bin. 3.4KiB client-requested in use in bin. 2020-03-24 18:13:17.173271: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:17.173288: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 5, Chunks in use: 5. 45.0KiB allocated for chunks. 45.0KiB in use in bin. 45.0KiB client-requested in use in bin. 2020-03-24 18:13:17.173304: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:17.173323: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 5, Chunks in use: 5. 164.0KiB allocated for chunks. 164.0KiB in use in bin. 164.0KiB client-requested in use in bin. 2020-03-24 18:13:17.173340: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 4, Chunks in use: 4. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin. 2020-03-24 18:13:17.173357: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 10, Chunks in use: 10. 1.41MiB allocated for chunks. 1.41MiB in use in bin. 1.41MiB client-requested in use in bin. 2020-03-24 18:13:17.173374: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 4, Chunks in use: 4. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin. 2020-03-24 18:13:17.173390: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 10, Chunks in use: 10. 5.62MiB allocated for chunks. 5.62MiB in use in bin. 5.62MiB client-requested in use in bin. 2020-03-24 18:13:17.173406: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:17.173422: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:17.173437: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:17.173453: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:17.173468: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:17.173483: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:17.173499: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:17.173514: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:17.173543: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 3, Chunks in use: 2. 14.58GiB allocated for chunks. 10.85GiB in use in bin. 10.85GiB client-requested in use in bin. 2020-03-24 18:13:17.173560: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 5.42GiB was 256.00MiB, Chunk State: 2020-03-24 18:13:17.173583: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 3.73GiB | Requested Size: 0B | in_use: 0, prev: Size: 5.42GiB | Requested Size: 5.42GiB | in_use: 1 2020-03-24 18:13:17.173601: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824000000 of size 13312 2020-03-24 18:13:17.173643: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003400 of size 256 2020-03-24 18:13:17.173657: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003500 of size 256 2020-03-24 18:13:17.173670: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003600 of size 256 2020-03-24 18:13:17.173684: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003700 of size 256 2020-03-24 18:13:17.173704: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003800 of size 256 2020-03-24 18:13:17.173728: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003900 of size 256 2020-03-24 18:13:17.173741: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003a00 of size 256 2020-03-24 18:13:17.173765: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003b00 of size 3584 2020-03-24 18:13:17.173779: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004900 of size 256 2020-03-24 18:13:17.173799: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004a00 of size 256 2020-03-24 18:13:17.173812: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004b00 of size 256 2020-03-24 18:13:17.173825: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004c00 of size 256 2020-03-24 18:13:17.173838: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004d00 of size 256 2020-03-24 18:13:17.173851: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004e00 of size 256 2020-03-24 18:13:17.173864: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004f00 of size 256 2020-03-24 18:13:17.173876: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005000 of size 256 2020-03-24 18:13:17.173889: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005100 of size 256 2020-03-24 18:13:17.173903: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005200 of size 256 2020-03-24 18:13:17.173921: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005300 of size 256 2020-03-24 18:13:17.173933: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005400 of size 256 2020-03-24 18:13:17.173950: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005500 of size 256 2020-03-24 18:13:17.173967: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005600 of size 256 2020-03-24 18:13:17.173980: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005700 of size 256 2020-03-24 18:13:17.173993: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005800 of size 256 2020-03-24 18:13:17.174015: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005900 of size 36864 2020-03-24 18:13:17.174028: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400e900 of size 256 2020-03-24 18:13:17.174041: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400ea00 of size 256 2020-03-24 18:13:17.174054: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400eb00 of size 256 2020-03-24 18:13:17.174071: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400ec00 of size 256 2020-03-24 18:13:17.174085: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400ed00 of size 256 2020-03-24 18:13:17.174098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400ee00 of size 256 2020-03-24 18:13:17.174110: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400ef00 of size 256 2020-03-24 18:13:17.174124: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400f000 of size 73728 2020-03-24 18:13:17.174137: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021000 of size 256 2020-03-24 18:13:17.174150: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021100 of size 256 2020-03-24 18:13:17.174163: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021200 of size 256 2020-03-24 18:13:17.174176: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021300 of size 256 2020-03-24 18:13:17.174189: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021400 of size 256 2020-03-24 18:13:17.174201: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021500 of size 256 2020-03-24 18:13:17.174215: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021600 of size 8192 2020-03-24 18:13:17.174228: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023600 of size 256 2020-03-24 18:13:17.174242: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023700 of size 256 2020-03-24 18:13:17.174254: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023800 of size 256 2020-03-24 18:13:17.174267: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023900 of size 256 2020-03-24 18:13:17.174280: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023a00 of size 256 2020-03-24 18:13:17.174293: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023b00 of size 256 2020-03-24 18:13:17.174306: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023c00 of size 147456 2020-03-24 18:13:17.174319: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824047c00 of size 256 2020-03-24 18:13:17.174333: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824047d00 of size 256 2020-03-24 18:13:17.174345: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824047e00 of size 256 2020-03-24 18:13:17.174358: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824047f00 of size 256 2020-03-24 18:13:17.174371: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824048000 of size 512 2020-03-24 18:13:17.174384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824048200 of size 512 2020-03-24 18:13:17.174397: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824048400 of size 512 2020-03-24 18:13:17.174411: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824048600 of size 294912 2020-03-24 18:13:17.174424: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090600 of size 256 2020-03-24 18:13:17.174437: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090700 of size 512 2020-03-24 18:13:17.174450: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090900 of size 256 2020-03-24 18:13:17.174463: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090a00 of size 256 2020-03-24 18:13:17.174475: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090b00 of size 256 2020-03-24 18:13:17.174488: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090c00 of size 256 2020-03-24 18:13:17.174501: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090d00 of size 32768 2020-03-24 18:13:17.174517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824098d00 of size 256 2020-03-24 18:13:17.174535: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824098e00 of size 512 2020-03-24 18:13:17.174548: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824099000 of size 512 2020-03-24 18:13:17.174560: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824099200 of size 512 2020-03-24 18:13:17.174573: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824099400 of size 256 2020-03-24 18:13:17.174585: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824099500 of size 512 2020-03-24 18:13:17.174599: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824099700 of size 589824 2020-03-24 18:13:17.174632: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824129700 of size 256 2020-03-24 18:13:17.174646: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824129800 of size 512 2020-03-24 18:13:17.174659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824129a00 of size 512 2020-03-24 18:13:17.174671: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824129c00 of size 512 2020-03-24 18:13:17.174684: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824129e00 of size 512 2020-03-24 18:13:17.174697: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382412a000 of size 512 2020-03-24 18:13:17.174709: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382412a200 of size 512 2020-03-24 18:13:17.174722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382412a400 of size 5825757184 2020-03-24 18:13:17.174736: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50a400 of size 256 2020-03-24 18:13:17.174749: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50a500 of size 256 2020-03-24 18:13:17.174762: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50a600 of size 1280 2020-03-24 18:13:17.174775: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50ab00 of size 256 2020-03-24 18:13:17.174788: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50ac00 of size 256 2020-03-24 18:13:17.174800: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50ad00 of size 256 2020-03-24 18:13:17.174813: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50ae00 of size 256 2020-03-24 18:13:17.174825: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50af00 of size 256 2020-03-24 18:13:17.174838: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50b000 of size 256 2020-03-24 18:13:17.174851: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50b100 of size 256 2020-03-24 18:13:17.174864: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50b200 of size 73728 2020-03-24 18:13:17.174876: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f51d200 of size 73728 2020-03-24 18:13:17.174889: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f52f200 of size 73728 2020-03-24 18:13:17.174926: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f541200 of size 256 2020-03-24 18:13:17.174956: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f541300 of size 256 2020-03-24 18:13:17.174969: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f541400 of size 256 2020-03-24 18:13:17.174982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f541500 of size 147456 2020-03-24 18:13:17.175021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565500 of size 256 2020-03-24 18:13:17.175036: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565600 of size 256 2020-03-24 18:13:17.175063: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565700 of size 256 2020-03-24 18:13:17.175076: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565800 of size 256 2020-03-24 18:13:17.175089: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565900 of size 256 2020-03-24 18:13:17.175102: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565a00 of size 8192 2020-03-24 18:13:17.175114: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f567a00 of size 8192 2020-03-24 18:13:17.175127: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f569a00 of size 8192 2020-03-24 18:13:17.175139: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56ba00 of size 256 2020-03-24 18:13:17.175152: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56bb00 of size 256 2020-03-24 18:13:17.175165: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56bc00 of size 256 2020-03-24 18:13:17.175177: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56bd00 of size 256 2020-03-24 18:13:17.175190: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56be00 of size 256 2020-03-24 18:13:17.175202: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56bf00 of size 147456 2020-03-24 18:13:17.175215: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f58ff00 of size 256 2020-03-24 18:13:17.175237: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f590000 of size 256 2020-03-24 18:13:17.175257: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f590100 of size 256 2020-03-24 18:13:17.175271: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f590200 of size 147456 2020-03-24 18:13:17.175284: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f5b4200 of size 147456 2020-03-24 18:13:17.175296: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f5d8200 of size 147456 2020-03-24 18:13:17.175309: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f5fc200 of size 147456 2020-03-24 18:13:17.175328: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f620200 of size 147456 2020-03-24 18:13:17.175341: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f644200 of size 147456 2020-03-24 18:13:17.175354: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f668200 of size 147456 2020-03-24 18:13:17.175367: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c200 of size 256 2020-03-24 18:13:17.175379: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c300 of size 256 2020-03-24 18:13:17.175392: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c400 of size 256 2020-03-24 18:13:17.175405: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c500 of size 256 2020-03-24 18:13:17.175418: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c600 of size 256 2020-03-24 18:13:17.175430: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c700 of size 512 2020-03-24 18:13:17.175443: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c900 of size 512 2020-03-24 18:13:17.175455: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68cb00 of size 512 2020-03-24 18:13:17.175468: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68cd00 of size 512 2020-03-24 18:13:17.175481: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68cf00 of size 512 2020-03-24 18:13:17.175494: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68d100 of size 294912 2020-03-24 18:13:17.175507: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f6d5100 of size 294912 2020-03-24 18:13:17.175523: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f71d100 of size 294912 2020-03-24 18:13:17.175536: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f765100 of size 512 2020-03-24 18:13:17.175548: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f765300 of size 512 2020-03-24 18:13:17.175561: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f765500 of size 512 2020-03-24 18:13:17.175574: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f765700 of size 589824 2020-03-24 18:13:17.175586: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5700 of size 256 2020-03-24 18:13:17.175599: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5800 of size 256 2020-03-24 18:13:17.175619: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5900 of size 256 2020-03-24 18:13:17.175641: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5a00 of size 256 2020-03-24 18:13:17.175654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5b00 of size 256 2020-03-24 18:13:17.175666: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5c00 of size 32768 2020-03-24 18:13:17.175679: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7fdc00 of size 32768 2020-03-24 18:13:17.175692: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f805c00 of size 32768 2020-03-24 18:13:17.175704: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80dc00 of size 512 2020-03-24 18:13:17.175717: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80de00 of size 512 2020-03-24 18:13:17.175730: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80e000 of size 512 2020-03-24 18:13:17.175742: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80e200 of size 512 2020-03-24 18:13:17.175755: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80e400 of size 512 2020-03-24 18:13:17.175768: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80e600 of size 589824 2020-03-24 18:13:17.175780: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f89e600 of size 512 2020-03-24 18:13:17.175793: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f89e800 of size 512 2020-03-24 18:13:17.175805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f89ea00 of size 512 2020-03-24 18:13:17.175818: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f89ec00 of size 589824 2020-03-24 18:13:17.175831: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f92ec00 of size 589824 2020-03-24 18:13:17.175844: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f9bec00 of size 589824 2020-03-24 18:13:17.175856: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fa4ec00 of size 589824 2020-03-24 18:13:17.175869: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fadec00 of size 589824 2020-03-24 18:13:17.175881: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fb6ec00 of size 589824 2020-03-24 18:13:17.175894: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fbfec00 of size 589824 2020-03-24 18:13:17.175907: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8ec00 of size 512 2020-03-24 18:13:17.175920: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8ee00 of size 512 2020-03-24 18:13:17.175933: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8f000 of size 512 2020-03-24 18:13:17.175945: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8f200 of size 512 2020-03-24 18:13:17.175958: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8f400 of size 512 2020-03-24 18:13:17.175970: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8f600 of size 512 2020-03-24 18:13:17.175986: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8f800 of size 512 2020-03-24 18:13:17.175999: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8fa00 of size 512 2020-03-24 18:13:17.176012: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8fc00 of size 512 2020-03-24 18:13:17.176024: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8fe00 of size 512 2020-03-24 18:13:17.176037: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc90000 of size 5825757184 2020-03-24 18:13:17.176050: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2b3adb070000 of size 4008896000 2020-03-24 18:13:17.176062: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size: 2020-03-24 18:13:17.176079: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 89 Chunks of size 256 totalling 22.2KiB 2020-03-24 18:13:17.176095: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 40 Chunks of size 512 totalling 20.0KiB 2020-03-24 18:13:17.176110: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB 2020-03-24 18:13:17.176125: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3584 totalling 3.5KiB 2020-03-24 18:13:17.176140: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 8192 totalling 32.0KiB 2020-03-24 18:13:17.176154: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 13312 totalling 13.0KiB 2020-03-24 18:13:17.176169: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 32768 totalling 128.0KiB 2020-03-24 18:13:17.176184: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 36864 totalling 36.0KiB 2020-03-24 18:13:17.176199: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 73728 totalling 288.0KiB 2020-03-24 18:13:17.176214: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 147456 totalling 1.41MiB 2020-03-24 18:13:17.176242: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 294912 totalling 1.12MiB 2020-03-24 18:13:17.176260: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 589824 totalling 5.62MiB 2020-03-24 18:13:17.176276: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 5825757184 totalling 10.85GiB 2020-03-24 18:13:17.176291: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 10.86GiB 2020-03-24 18:13:17.176309: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit: 15669519975 InUse: 11660623872 MaxInUse: 11660623872 NumAllocs: 171 MaxAllocSize: 5825757184

2020-03-24 18:13:17.176337: W tensorflow/core/common_runtime/bfcallocator.cc:271] ***____ 2020-03-24 18:13:17.176401: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[11378432,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc 2020-03-24 18:13:27.176677: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 5.42GiB. Current allocation summary follows. 2020-03-24 18:13:27.176777: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 89, Chunks in use: 89. 22.2KiB allocated for chunks. 22.2KiB in use in bin. 13.5KiB client-requested in use in bin. 2020-03-24 18:13:27.176807: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2020-03-24 18:13:27.176829: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin. 2020-03-24 18:13:27.176857: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 1, Chunks in use: 1. 3.5KiB allocated for chunks. 3.5KiB in use in bin. 3.4KiB client-requested in use in bin. 2020-03-24 18:13:27.176877: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:27.176898: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 5, Chunks in use: 5. 45.0KiB allocated for chunks. 45.0KiB in use in bin. 45.0KiB client-requested in use in bin. 2020-03-24 18:13:27.176917: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:27.176938: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 5, Chunks in use: 5. 164.0KiB allocated for chunks. 164.0KiB in use in bin. 164.0KiB client-requested in use in bin. 2020-03-24 18:13:27.176960: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 4, Chunks in use: 4. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin. 2020-03-24 18:13:27.176980: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 10, Chunks in use: 10. 1.41MiB allocated for chunks. 1.41MiB in use in bin. 1.41MiB client-requested in use in bin. 2020-03-24 18:13:27.176999: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 4, Chunks in use: 4. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin. 2020-03-24 18:13:27.177019: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 10, Chunks in use: 10. 5.62MiB allocated for chunks. 5.62MiB in use in bin. 5.62MiB client-requested in use in bin. 2020-03-24 18:13:27.177037: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:27.177056: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:27.177073: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:27.177097: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:27.177137: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:27.177160: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:27.177178: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:27.177195: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-03-24 18:13:27.177218: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 3, Chunks in use: 2. 14.58GiB allocated for chunks. 10.85GiB in use in bin. 10.85GiB client-requested in use in bin. 2020-03-24 18:13:27.177241: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 5.42GiB was 256.00MiB, Chunk State: 2020-03-24 18:13:27.177268: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 3.73GiB | Requested Size: 0B | in_use: 0, prev: Size: 5.42GiB | Requested Size: 5.42GiB | in_use: 1 2020-03-24 18:13:27.177290: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824000000 of size 13312 2020-03-24 18:13:27.177306: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003400 of size 256 2020-03-24 18:13:27.177321: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003500 of size 256 2020-03-24 18:13:27.177335: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003600 of size 256 2020-03-24 18:13:27.177349: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003700 of size 256 2020-03-24 18:13:27.177364: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003800 of size 256 2020-03-24 18:13:27.177378: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003900 of size 256 2020-03-24 18:13:27.177392: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003a00 of size 256 2020-03-24 18:13:27.177420: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824003b00 of size 3584 2020-03-24 18:13:27.177446: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004900 of size 256 2020-03-24 18:13:27.177462: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004a00 of size 256 2020-03-24 18:13:27.177477: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004b00 of size 256 2020-03-24 18:13:27.177491: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004c00 of size 256 2020-03-24 18:13:27.177506: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004d00 of size 256 2020-03-24 18:13:27.177520: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004e00 of size 256 2020-03-24 18:13:27.177535: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824004f00 of size 256 2020-03-24 18:13:27.177549: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005000 of size 256 2020-03-24 18:13:27.177564: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005100 of size 256 2020-03-24 18:13:27.177579: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005200 of size 256 2020-03-24 18:13:27.177593: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005300 of size 256 2020-03-24 18:13:27.177649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005400 of size 256 2020-03-24 18:13:27.177667: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005500 of size 256 2020-03-24 18:13:27.177682: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005600 of size 256 2020-03-24 18:13:27.177696: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005700 of size 256 2020-03-24 18:13:27.177710: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005800 of size 256 2020-03-24 18:13:27.177726: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824005900 of size 36864 2020-03-24 18:13:27.177741: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400e900 of size 256 2020-03-24 18:13:27.177757: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400ea00 of size 256 2020-03-24 18:13:27.177771: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400eb00 of size 256 2020-03-24 18:13:27.177786: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400ec00 of size 256 2020-03-24 18:13:27.177801: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400ed00 of size 256 2020-03-24 18:13:27.177816: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400ee00 of size 256 2020-03-24 18:13:27.177835: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400ef00 of size 256 2020-03-24 18:13:27.177851: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382400f000 of size 73728 2020-03-24 18:13:27.177867: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021000 of size 256 2020-03-24 18:13:27.177882: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021100 of size 256 2020-03-24 18:13:27.177905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021200 of size 256 2020-03-24 18:13:27.177920: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021300 of size 256 2020-03-24 18:13:27.177935: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021400 of size 256 2020-03-24 18:13:27.177950: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021500 of size 256 2020-03-24 18:13:27.177966: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824021600 of size 8192 2020-03-24 18:13:27.177982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023600 of size 256 2020-03-24 18:13:27.177997: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023700 of size 256 2020-03-24 18:13:27.178012: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023800 of size 256 2020-03-24 18:13:27.178027: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023900 of size 256 2020-03-24 18:13:27.178042: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023a00 of size 256 2020-03-24 18:13:27.178057: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023b00 of size 256 2020-03-24 18:13:27.178073: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824023c00 of size 147456 2020-03-24 18:13:27.178088: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824047c00 of size 256 2020-03-24 18:13:27.178103: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824047d00 of size 256 2020-03-24 18:13:27.178118: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824047e00 of size 256 2020-03-24 18:13:27.178133: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824047f00 of size 256 2020-03-24 18:13:27.178148: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824048000 of size 512 2020-03-24 18:13:27.178164: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824048200 of size 512 2020-03-24 18:13:27.178179: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824048400 of size 512 2020-03-24 18:13:27.178194: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824048600 of size 294912 2020-03-24 18:13:27.178210: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090600 of size 256 2020-03-24 18:13:27.178225: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090700 of size 512 2020-03-24 18:13:27.178240: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090900 of size 256 2020-03-24 18:13:27.178256: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090a00 of size 256 2020-03-24 18:13:27.178271: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090b00 of size 256 2020-03-24 18:13:27.178286: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090c00 of size 256 2020-03-24 18:13:27.178301: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824090d00 of size 32768 2020-03-24 18:13:27.178317: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824098d00 of size 256 2020-03-24 18:13:27.178332: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824098e00 of size 512 2020-03-24 18:13:27.178347: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824099000 of size 512 2020-03-24 18:13:27.178365: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824099200 of size 512 2020-03-24 18:13:27.178381: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824099400 of size 256 2020-03-24 18:13:27.178396: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824099500 of size 512 2020-03-24 18:13:27.178411: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824099700 of size 589824 2020-03-24 18:13:27.178427: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824129700 of size 256 2020-03-24 18:13:27.178442: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824129800 of size 512 2020-03-24 18:13:27.178457: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824129a00 of size 512 2020-03-24 18:13:27.178472: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824129c00 of size 512 2020-03-24 18:13:27.178487: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b3824129e00 of size 512 2020-03-24 18:13:27.178502: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382412a000 of size 512 2020-03-24 18:13:27.178517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382412a200 of size 512 2020-03-24 18:13:27.178533: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b382412a400 of size 5825757184 2020-03-24 18:13:27.178548: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50a400 of size 256 2020-03-24 18:13:27.178563: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50a500 of size 256 2020-03-24 18:13:27.178578: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50a600 of size 1280 2020-03-24 18:13:27.178594: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50ab00 of size 256 2020-03-24 18:13:27.178630: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50ac00 of size 256 2020-03-24 18:13:27.178646: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50ad00 of size 256 2020-03-24 18:13:27.178661: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50ae00 of size 256 2020-03-24 18:13:27.178675: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50af00 of size 256 2020-03-24 18:13:27.178689: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50b000 of size 256 2020-03-24 18:13:27.178704: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50b100 of size 256 2020-03-24 18:13:27.178718: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f50b200 of size 73728 2020-03-24 18:13:27.178733: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f51d200 of size 73728 2020-03-24 18:13:27.178747: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f52f200 of size 73728 2020-03-24 18:13:27.178762: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f541200 of size 256 2020-03-24 18:13:27.178776: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f541300 of size 256 2020-03-24 18:13:27.178790: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f541400 of size 256 2020-03-24 18:13:27.178805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f541500 of size 147456 2020-03-24 18:13:27.178819: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565500 of size 256 2020-03-24 18:13:27.178834: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565600 of size 256 2020-03-24 18:13:27.178848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565700 of size 256 2020-03-24 18:13:27.178863: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565800 of size 256 2020-03-24 18:13:27.178878: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565900 of size 256 2020-03-24 18:13:27.178892: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f565a00 of size 8192 2020-03-24 18:13:27.178920: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f567a00 of size 8192 2020-03-24 18:13:27.178943: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f569a00 of size 8192 2020-03-24 18:13:27.178968: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56ba00 of size 256 2020-03-24 18:13:27.178983: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56bb00 of size 256 2020-03-24 18:13:27.178997: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56bc00 of size 256 2020-03-24 18:13:27.179012: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56bd00 of size 256 2020-03-24 18:13:27.179026: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56be00 of size 256 2020-03-24 18:13:27.179041: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f56bf00 of size 147456 2020-03-24 18:13:27.179055: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f58ff00 of size 256 2020-03-24 18:13:27.179086: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f590000 of size 256 2020-03-24 18:13:27.179103: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f590100 of size 256 2020-03-24 18:13:27.179118: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f590200 of size 147456 2020-03-24 18:13:27.179133: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f5b4200 of size 147456 2020-03-24 18:13:27.179148: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f5d8200 of size 147456 2020-03-24 18:13:27.179162: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f5fc200 of size 147456 2020-03-24 18:13:27.179177: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f620200 of size 147456 2020-03-24 18:13:27.179191: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f644200 of size 147456 2020-03-24 18:13:27.179206: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f668200 of size 147456 2020-03-24 18:13:27.179272: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c200 of size 256 2020-03-24 18:13:27.179299: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c300 of size 256 2020-03-24 18:13:27.179315: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c400 of size 256 2020-03-24 18:13:27.179329: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c500 of size 256 2020-03-24 18:13:27.179344: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c600 of size 256 2020-03-24 18:13:27.179358: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c700 of size 512 2020-03-24 18:13:27.179372: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68c900 of size 512 2020-03-24 18:13:27.179387: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68cb00 of size 512 2020-03-24 18:13:27.179401: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68cd00 of size 512 2020-03-24 18:13:27.179415: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68cf00 of size 512 2020-03-24 18:13:27.179430: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f68d100 of size 294912 2020-03-24 18:13:27.179445: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f6d5100 of size 294912 2020-03-24 18:13:27.179459: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f71d100 of size 294912 2020-03-24 18:13:27.179474: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f765100 of size 512 2020-03-24 18:13:27.179488: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f765300 of size 512 2020-03-24 18:13:27.179503: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f765500 of size 512 2020-03-24 18:13:27.179522: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f765700 of size 589824 2020-03-24 18:13:27.179537: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5700 of size 256 2020-03-24 18:13:27.179551: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5800 of size 256 2020-03-24 18:13:27.179568: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5900 of size 256 2020-03-24 18:13:27.179582: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5a00 of size 256 2020-03-24 18:13:27.179596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5b00 of size 256 2020-03-24 18:13:27.179636: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7f5c00 of size 32768 2020-03-24 18:13:27.179652: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f7fdc00 of size 32768 2020-03-24 18:13:27.179666: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f805c00 of size 32768 2020-03-24 18:13:27.179681: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80dc00 of size 512 2020-03-24 18:13:27.179695: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80de00 of size 512 2020-03-24 18:13:27.179709: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80e000 of size 512 2020-03-24 18:13:27.179724: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80e200 of size 512 2020-03-24 18:13:27.179738: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80e400 of size 512 2020-03-24 18:13:27.179753: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f80e600 of size 589824 2020-03-24 18:13:27.179767: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f89e600 of size 512 2020-03-24 18:13:27.179782: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f89e800 of size 512 2020-03-24 18:13:27.179796: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f89ea00 of size 512 2020-03-24 18:13:27.179811: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f89ec00 of size 589824 2020-03-24 18:13:27.179826: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f92ec00 of size 589824 2020-03-24 18:13:27.179840: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397f9bec00 of size 589824 2020-03-24 18:13:27.179855: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fa4ec00 of size 589824 2020-03-24 18:13:27.179869: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fadec00 of size 589824 2020-03-24 18:13:27.179884: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fb6ec00 of size 589824 2020-03-24 18:13:27.179899: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fbfec00 of size 589824 2020-03-24 18:13:27.179922: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8ec00 of size 512 2020-03-24 18:13:27.179937: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8ee00 of size 512 2020-03-24 18:13:27.179951: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8f000 of size 512 2020-03-24 18:13:27.179965: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8f200 of size 512 2020-03-24 18:13:27.179980: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8f400 of size 512 2020-03-24 18:13:27.179994: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8f600 of size 512 2020-03-24 18:13:27.180009: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8f800 of size 512 2020-03-24 18:13:27.180023: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8fa00 of size 512 2020-03-24 18:13:27.180038: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8fc00 of size 512 2020-03-24 18:13:27.180056: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc8fe00 of size 512 2020-03-24 18:13:27.180071: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2b397fc90000 of size 5825757184 2020-03-24 18:13:27.180086: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2b3adb070000 of size 4008896000 2020-03-24 18:13:27.180099: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size: 2020-03-24 18:13:27.180120: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 89 Chunks of size 256 totalling 22.2KiB 2020-03-24 18:13:27.180138: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 40 Chunks of size 512 totalling 20.0KiB 2020-03-24 18:13:27.180155: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB 2020-03-24 18:13:27.180172: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3584 totalling 3.5KiB 2020-03-24 18:13:27.180190: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 8192 totalling 32.0KiB 2020-03-24 18:13:27.180208: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 13312 totalling 13.0KiB 2020-03-24 18:13:27.180226: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 32768 totalling 128.0KiB 2020-03-24 18:13:27.180243: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 36864 totalling 36.0KiB 2020-03-24 18:13:27.180261: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 73728 totalling 288.0KiB 2020-03-24 18:13:27.180278: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 147456 totalling 1.41MiB 2020-03-24 18:13:27.180295: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 294912 totalling 1.12MiB 2020-03-24 18:13:27.180312: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 589824 totalling 5.62MiB 2020-03-24 18:13:27.180330: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 5825757184 totalling 10.85GiB 2020-03-24 18:13:27.180348: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 10.86GiB 2020-03-24 18:13:27.180368: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit: 15669519975 InUse: 11660623872 MaxInUse: 11660623872 NumAllocs: 171 MaxAllocSize: 5825757184

2020-03-24 18:13:27.180402: W tensorflow/core/common_runtime/bfcallocator.cc:271] ***____ 2020-03-24 18:13:27.180457: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[11378432,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Train set size: 193 images, 23 identities feature dimensionality: 128

Run ID: cosine-softmax Log directory: /tmp/cattles_logdir/cosine-softmax

Traceback (most recent call last): File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[11378432,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node fc1/weights/Adam_1/Assign}} = Assign[T=DT_FLOAT, _class=["loc:@fc1/weights/Assign"], _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fc1/weights/Adam_1, fc1/weights/Adam/Initializer/zeros)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train_cattledata.py", line 127, in main() File "train_cattledata.py", line 68, in main train_kwargs) File "/workfs/zhangjian2/cosine_metric_learning/train_app.py", line 199, in train_loop save_interval_secs=save_interval_secs, number_of_steps=number_of_steps) File "/workfs/zhangjian2/cosine_metric_learning/queued_trainer.py", line 408, in run kwargs) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 748, in train master, start_standard_services=False, config=session_config) as sess: File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/contextlib.py", line 81, in enter return next(self.gen) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 1004, in managed_session self.stop(close_summary_writer=close_summary_writer) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 832, in stop ignore_live_threads=ignore_live_threads) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 389, in join six.reraise(*self._exc_info_to_raise) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 993, in managed_session start_standard_services=start_standard_services) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 730, in prepare_or_wait_for_session init_feed_dict=self._init_feed_dict, init_fn=self._init_fn) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/session_manager.py", line 294, in prepare_session sess.run(init_op, feed_dict=init_feed_dict) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[11378432,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node fc1/weights/Adam_1/Assign (defined at /workfs/zhangjian2/cosine_metric_learning/train_app.py:286) = Assign[T=DT_FLOAT, _class=["loc:@fc1/weights/Assign"], _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fc1/weights/Adam_1, fc1/weights/Adam/Initializer/zeros)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'fc1/weights/Adam_1/Assign', defined at: File "train_cattledata.py", line 127, in main() File "train_cattledata.py", line 68, in main train_kwargs) File "/workfs/zhangjian2/cosine_metric_learning/train_app.py", line 188, in train_loop trainable_scopes=trainable_scopes) File "/workfs/zhangjian2/cosine_metric_learning/train_app.py", line 286, in create_trainer variables_to_train=variables_to_train) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 439, in create_train_op check_numerics=check_numerics) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/contrib/training/python/training/training.py", line 458, in create_train_op grad_updates = optimizer.apply_gradients(grads, global_step=global_step) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 593, in apply_gradients self._create_slots(var_list) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/adam.py", line 136, in _create_slots self._zeros_slot(v, "v", self._name) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 1139, in _zeros_slot new_slot_variable = slot_creator.create_zeros_slot(var, op_name) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 183, in create_zeros_slot colocate_with_primary=colocate_with_primary) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 157, in create_slot_with_initializer dtype) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 65, in _create_slot_var validate_shape=validate_shape) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1487, in get_variable aggregation=aggregation) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1237, in get_variable aggregation=aggregation) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 540, in get_variable aggregation=aggregation) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 492, in _true_getter aggregation=aggregation) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 922, in _get_single_variable aggregation=aggregation) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 183, in call return cls._variable_v1_call(*args, kwargs) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 146, in _variable_v1_call aggregation=aggregation) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 125, in previous_getter = lambda kwargs: default_variable_creator(None, *kwargs) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2444, in default_variable_creator expected_shape=expected_shape, import_scope=import_scope) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 187, in call return super(VariableMetaclass, cls).call(args, kwargs) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1329, in init constraint=constraint) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1481, in _init_from_args validate_shape=validate_shape).op File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 221, in assign validate_shape=validate_shape) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign use_locking=use_locking, name=name) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[11378432,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node fc1/weights/Adam_1/Assign (defined at /workfs/zhangjian2/cosine_metric_learning/train_app.py:286) = Assign[T=DT_FLOAT, _class=["loc:@fc1/weights/Assign"], _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fc1/weights/Adam_1, fc1/weights/Adam/Initializer/zeros)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

ERROR:tensorflow:================================== Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>): <tf.Tensor 'init_ops/report_uninitialized_variables/boolean_mask/GatherV2:0' shape=(?,) dtype=string> If you want to mark it as used call its "mark_used()" method. It was originally created here: File "train_cattledata.py", line 127, in main() File "train_cattledata.py", line 68, in main *train_kwargs) File "/workfs/zhangjian2/cosine_metric_learning/train_app.py", line 199, in train_loop save_interval_secs=save_interval_secs, number_of_steps=number_of_steps) File "/workfs/zhangjian2/cosine_metric_learning/queued_trainer.py", line 414, in run pass File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 791, in train should_retry = True File "/software/google/tensorflow/python3.6-gpu/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped return _add_should_use_warning(fn(args, **kwargs))

[mayeeeeeeee]

I resize the image size to (128, 64, 3), it works.