search

anishathalye / neural-style

Neural style in TensorFlow! 🎨
https://anishathalye.com/an-ai-that-can-mimic-any-artist/
GNU General Public License v3.0
5.54k stars 1.52k forks source link

Out of memory #169

Closed bartman081523 closed 3 years ago

bartman081523 commented 3 years ago 
Optimization started...
Iteration    1/1000
2020-10-05 10:07:39.976061: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-10-05 10:07:41.545383: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-05 10:07:47.578975: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.80GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-10-05 10:07:47.579048: W tensorflow/core/kernels/gpu_utils.cc:48] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2020-10-05 10:07:59.212287: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.78GiB (rounded to 4063297536).  Current allocation summary follows.
2020-10-05 10:07:59.212360: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (256):   Total Chunks: 24, Chunks in use: 24. 6.0KiB allocated for chunks. 6.0KiB in use in bin. 600B client-requested in use in bin.
2020-10-05 10:07:59.212400: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (512):   Total Chunks: 2, Chunks in use: 2. 1.0KiB allocated for chunks. 1.0KiB in use in bin. 1.0KiB client-requested in use in bin.
2020-10-05 10:07:59.212419: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2020-10-05 10:07:59.212434: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2048):  Total Chunks: 6, Chunks in use: 6. 12.0KiB allocated for chunks. 12.0KiB in use in bin. 12.0KiB client-requested in use in bin.
2020-10-05 10:07:59.212452: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2020-10-05 10:07:59.212460: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:07:59.212471: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16384):     Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2020-10-05 10:07:59.212481: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (32768):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:07:59.212491: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (65536):     Total Chunks: 1, Chunks in use: 1. 64.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-10-05 10:07:59.212503: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (131072):    Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2020-10-05 10:07:59.212514: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (262144):    Total Chunks: 2, Chunks in use: 2. 544.0KiB allocated for chunks. 544.0KiB in use in bin. 544.0KiB client-requested in use in bin.
2020-10-05 10:07:59.212525: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (524288):    Total Chunks: 1, Chunks in use: 1. 581.8KiB allocated for chunks. 581.8KiB in use in bin. 576.0KiB client-requested in use in bin.
2020-10-05 10:07:59.212535: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1048576):   Total Chunks: 3, Chunks in use: 3. 3.12MiB allocated for chunks. 3.12MiB in use in bin. 3.12MiB client-requested in use in bin.
2020-10-05 10:07:59.212545: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2097152):   Total Chunks: 3, Chunks in use: 3. 6.75MiB allocated for chunks. 6.75MiB in use in bin. 6.75MiB client-requested in use in bin.
2020-10-05 10:07:59.212556: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4194304):   Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2020-10-05 10:07:59.212566: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8388608):   Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2020-10-05 10:07:59.212576: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16777216):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:07:59.212586: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (33554432):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:07:59.212597: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (67108864):  Total Chunks: 2, Chunks in use: 1. 210.25MiB allocated for chunks. 121.26MiB in use in bin. 121.26MiB client-requested in use in bin.
2020-10-05 10:07:59.212607: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (134217728):     Total Chunks: 7, Chunks in use: 7. 1.33GiB allocated for chunks. 1.33GiB in use in bin. 1.24GiB client-requested in use in bin.
2020-10-05 10:07:59.212618: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (268435456):     Total Chunks: 7, Chunks in use: 5. 13.26GiB allocated for chunks. 8.54GiB in use in bin. 8.40GiB client-requested in use in bin.
2020-10-05 10:07:59.212628: I tensorflow/core/common_runtime/bfc_allocator.cc:885] Bin for 3.78GiB was 256.00MiB, Chunk State: 
2020-10-05 10:07:59.212644: I tensorflow/core/common_runtime/bfc_allocator.cc:891]   Size: 1.64GiB | Requested Size: 106.82MiB | in_use: 0 | bin_num: 20, prev:   Size: 181.61MiB | Requested Size: 181.61MiB | in_use: 1 | bin_num: -1
2020-10-05 10:07:59.212655: I tensorflow/core/common_runtime/bfc_allocator.cc:891]   Size: 3.07GiB | Requested Size: 181.64MiB | in_use: 0 | bin_num: 20, prev:   Size: 3.78GiB | Requested Size: 3.78GiB | in_use: 1 | bin_num: -1
2020-10-05 10:07:59.212664: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 7365178112
2020-10-05 10:07:59.212676: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fd9fa000000 next 68 of size 4063297536
2020-10-05 10:07:59.212685: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdaec310000 next 18446744073709551615 of size 3301880576
2020-10-05 10:07:59.212694: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 4294967296
2020-10-05 10:07:59.212703: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdbd0000000 next 71 of size 4063297536
2020-10-05 10:07:59.212715: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdcc2310000 next 18446744073709551615 of size 231669760
2020-10-05 10:07:59.212723: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 2147483648
2020-10-05 10:07:59.212732: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdcd0000000 next 66 of size 190403328
2020-10-05 10:07:59.212741: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdcdb595300 next 67 of size 190431232
2020-10-05 10:07:59.212767: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdce6b31300 next 18446744073709551615 of size 1766649088
2020-10-05 10:07:59.212795: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1073741824
2020-10-05 10:07:59.212802: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdd50000000 next 54 of size 508592128
2020-10-05 10:07:59.212809: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdd6e508000 next 57 of size 190467072
2020-10-05 10:07:59.212817: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdd79aacc00 next 59 of size 127148032
2020-10-05 10:07:59.212834: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdd813eec00 next 18446744073709551615 of size 247534592
2020-10-05 10:07:59.212840: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 536870912
2020-10-05 10:07:59.212849: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddb0000000 next 10 of size 190467072
2020-10-05 10:07:59.212858: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddbb5a4c00 next 11 of size 190467072
2020-10-05 10:07:59.212867: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6b49800 next 18 of size 512
2020-10-05 10:07:59.212876: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6b49a00 next 19 of size 1179648
2020-10-05 10:07:59.212885: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6c69a00 next 20 of size 1024
2020-10-05 10:07:59.212894: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6c69e00 next 21 of size 2359296
2020-10-05 10:07:59.212902: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6ea9e00 next 22 of size 1024
2020-10-05 10:07:59.212911: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6eaa200 next 23 of size 2359296
2020-10-05 10:07:59.212919: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc70ea200 next 24 of size 1024
2020-10-05 10:07:59.212928: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc70ea600 next 25 of size 2359296
2020-10-05 10:07:59.212937: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc732a600 next 26 of size 1024
2020-10-05 10:07:59.212946: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc732aa00 next 27 of size 4718592
2020-10-05 10:07:59.212967: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc77aaa00 next 28 of size 2048
2020-10-05 10:07:59.212975: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc77ab200 next 29 of size 9437184
2020-10-05 10:07:59.212983: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc80ab200 next 30 of size 2048
2020-10-05 10:07:59.212991: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc80aba00 next 31 of size 9437184
2020-10-05 10:07:59.212999: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc89aba00 next 32 of size 2048
2020-10-05 10:07:59.213007: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc89ac200 next 33 of size 9437184
2020-10-05 10:07:59.213015: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc92ac200 next 34 of size 2048
2020-10-05 10:07:59.213023: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc92aca00 next 35 of size 9437184
2020-10-05 10:07:59.213031: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc9baca00 next 36 of size 2048
2020-10-05 10:07:59.213040: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc9bad200 next 37 of size 9437184
2020-10-05 10:07:59.213048: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4ad200 next 38 of size 2048
2020-10-05 10:07:59.213059: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4ada00 next 39 of size 16384
2020-10-05 10:07:59.213067: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4b1a00 next 40 of size 65536
2020-10-05 10:07:59.213076: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4c1a00 next 41 of size 262144
2020-10-05 10:07:59.213084: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca501a00 next 42 of size 1048576
2020-10-05 10:07:59.213092: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca601a00 next 43 of size 1048576
2020-10-05 10:07:59.213100: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701a00 next 44 of size 256
2020-10-05 10:07:59.213108: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701b00 next 45 of size 256
2020-10-05 10:07:59.213117: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701c00 next 46 of size 256
2020-10-05 10:07:59.213125: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701d00 next 47 of size 256
2020-10-05 10:07:59.213133: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701e00 next 48 of size 256
2020-10-05 10:07:59.213168: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701f00 next 49 of size 256
2020-10-05 10:07:59.213174: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702000 next 50 of size 256
2020-10-05 10:07:59.213181: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702100 next 51 of size 256
2020-10-05 10:07:59.213189: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702200 next 52 of size 256
2020-10-05 10:07:59.213198: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702300 next 55 of size 256
2020-10-05 10:07:59.213207: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702400 next 56 of size 256
2020-10-05 10:07:59.213215: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702500 next 58 of size 256
2020-10-05 10:07:59.213224: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702600 next 60 of size 256
2020-10-05 10:07:59.213232: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702700 next 61 of size 256
2020-10-05 10:07:59.213241: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702800 next 62 of size 256
2020-10-05 10:07:59.213249: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702900 next 63 of size 256
2020-10-05 10:07:59.213258: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702a00 next 64 of size 256
2020-10-05 10:07:59.213267: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fddca702b00 next 18446744073709551615 of size 93312256
2020-10-05 10:07:59.213275: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 268435456
2020-10-05 10:07:59.213284: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddd0000000 next 18446744073709551615 of size 268435456
2020-10-05 10:07:59.213292: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 268435456
2020-10-05 10:07:59.213301: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdde0000000 next 18446744073709551615 of size 268435456
2020-10-05 10:07:59.213309: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1048576
2020-10-05 10:07:59.213318: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00000 next 1 of size 256
2020-10-05 10:07:59.213327: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00100 next 2 of size 256
2020-10-05 10:07:59.213335: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00200 next 3 of size 256
2020-10-05 10:07:59.213344: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00300 next 5 of size 1280
2020-10-05 10:07:59.213352: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00800 next 6 of size 256
2020-10-05 10:07:59.213361: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00900 next 8 of size 256
2020-10-05 10:07:59.213370: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00a00 next 12 of size 6912
2020-10-05 10:07:59.213379: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec02500 next 13 of size 256
2020-10-05 10:07:59.213387: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec02600 next 14 of size 147456
2020-10-05 10:07:59.213396: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec26600 next 15 of size 256
2020-10-05 10:07:59.213405: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec26700 next 16 of size 294912
2020-10-05 10:07:59.213413: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec6e700 next 17 of size 512
2020-10-05 10:07:59.213422: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec6e900 next 18446744073709551615 of size 595712
2020-10-05 10:07:59.213431: I tensorflow/core/common_runtime/bfc_allocator.cc:914]      Summary of in-use Chunks by size: 
2020-10-05 10:07:59.213441: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 24 Chunks of size 256 totalling 6.0KiB
2020-10-05 10:07:59.213450: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 512 totalling 1.0KiB
2020-10-05 10:07:59.213459: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 1024 totalling 4.0KiB
2020-10-05 10:07:59.213468: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1280 totalling 1.2KiB
2020-10-05 10:07:59.213477: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 6 Chunks of size 2048 totalling 12.0KiB
2020-10-05 10:07:59.213486: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 6912 totalling 6.8KiB
2020-10-05 10:07:59.213495: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 16384 totalling 16.0KiB
2020-10-05 10:07:59.213504: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 65536 totalling 64.0KiB
2020-10-05 10:07:59.213514: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 147456 totalling 144.0KiB
2020-10-05 10:07:59.213523: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 262144 totalling 256.0KiB
2020-10-05 10:07:59.213532: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 294912 totalling 288.0KiB
2020-10-05 10:07:59.213541: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 595712 totalling 581.8KiB
2020-10-05 10:07:59.213550: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 1048576 totalling 2.00MiB
2020-10-05 10:07:59.244042: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1179648 totalling 1.12MiB
2020-10-05 10:07:59.244079: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 2359296 totalling 6.75MiB
2020-10-05 10:07:59.244087: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 4718592 totalling 4.50MiB
2020-10-05 10:07:59.244097: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 5 Chunks of size 9437184 totalling 45.00MiB
2020-10-05 10:07:59.244106: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 127148032 totalling 121.26MiB
2020-10-05 10:07:59.244115: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 190403328 totalling 181.58MiB
2020-10-05 10:07:59.244124: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 190431232 totalling 181.61MiB
2020-10-05 10:07:59.244133: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 190467072 totalling 544.93MiB
2020-10-05 10:07:59.244140: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 231669760 totalling 220.94MiB
2020-10-05 10:07:59.244149: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 247534592 totalling 236.07MiB
2020-10-05 10:07:59.244160: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 268435456 totalling 512.00MiB
2020-10-05 10:07:59.244173: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 508592128 totalling 485.03MiB
2020-10-05 10:07:59.244186: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 4063297536 totalling 7.57GiB
2020-10-05 10:07:59.244199: I tensorflow/core/common_runtime/bfc_allocator.cc:921] Sum Total of in-use chunks: 10.05GiB
2020-10-05 10:07:59.244209: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_: 15956161280 memory_limit_: 15956161332 available bytes: 52 curr_region_allocation_bytes_: 17179869184
2020-10-05 10:07:59.244222: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats: 
Limit:                 15956161332
InUse:                 10794319360
MaxInUse:              12310478336
NumAllocs:                      84
MaxAllocSize:           4080074752

2020-10-05 10:07:59.244243: W tensorflow/core/common_runtime/bfc_allocator.cc:424] **************************____________________******************************__________**************
2020-10-05 10:07:59.249236: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at transpose_op.cc:198 : Resource exhausted: OOM when allocating tensor with shape[1,2988,5312,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2020-10-05 10:07:59.252406: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.91GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-10-05 10:08:09.973495: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.89GiB (rounded to 2031648768).  Current allocation summary follows.
2020-10-05 10:08:09.973555: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (256):   Total Chunks: 24, Chunks in use: 24. 6.0KiB allocated for chunks. 6.0KiB in use in bin. 600B client-requested in use in bin.
2020-10-05 10:08:09.973565: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (512):   Total Chunks: 2, Chunks in use: 2. 1.0KiB allocated for chunks. 1.0KiB in use in bin. 1.0KiB client-requested in use in bin.
2020-10-05 10:08:09.973573: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2020-10-05 10:08:09.973583: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2048):  Total Chunks: 6, Chunks in use: 6. 12.0KiB allocated for chunks. 12.0KiB in use in bin. 12.0KiB client-requested in use in bin.
2020-10-05 10:08:09.973619: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2020-10-05 10:08:09.973626: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:08:09.973634: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16384):     Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2020-10-05 10:08:09.973643: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (32768):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:08:09.973653: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (65536):     Total Chunks: 1, Chunks in use: 1. 64.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-10-05 10:08:09.973670: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (131072):    Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2020-10-05 10:08:09.973680: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (262144):    Total Chunks: 2, Chunks in use: 2. 544.0KiB allocated for chunks. 544.0KiB in use in bin. 544.0KiB client-requested in use in bin.
2020-10-05 10:08:09.973691: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (524288):    Total Chunks: 1, Chunks in use: 1. 581.8KiB allocated for chunks. 581.8KiB in use in bin. 576.0KiB client-requested in use in bin.
2020-10-05 10:08:09.973700: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1048576):   Total Chunks: 3, Chunks in use: 3. 3.12MiB allocated for chunks. 3.12MiB in use in bin. 3.12MiB client-requested in use in bin.
2020-10-05 10:08:09.973710: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2097152):   Total Chunks: 3, Chunks in use: 3. 6.75MiB allocated for chunks. 6.75MiB in use in bin. 6.75MiB client-requested in use in bin.
2020-10-05 10:08:09.973719: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4194304):   Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2020-10-05 10:08:09.973729: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8388608):   Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2020-10-05 10:08:09.973738: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16777216):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:08:09.973747: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (33554432):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:08:09.973773: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (67108864):  Total Chunks: 2, Chunks in use: 1. 210.25MiB allocated for chunks. 121.26MiB in use in bin. 121.26MiB client-requested in use in bin.
2020-10-05 10:08:09.973784: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (134217728):     Total Chunks: 6, Chunks in use: 4. 1.16GiB allocated for chunks. 726.57MiB in use in bin. 726.57MiB client-requested in use in bin.
2020-10-05 10:08:09.973794: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (268435456):     Total Chunks: 10, Chunks in use: 7. 13.44GiB allocated for chunks. 11.38GiB in use in bin. 11.23GiB client-requested in use in bin.
2020-10-05 10:08:09.973804: I tensorflow/core/common_runtime/bfc_allocator.cc:885] Bin for 1.89GiB was 256.00MiB, Chunk State: 
2020-10-05 10:08:09.973820: I tensorflow/core/common_runtime/bfc_allocator.cc:891]   Size: 363.29MiB | Requested Size: 181.64MiB | in_use: 0 | bin_num: 20, prev:   Size: 181.64MiB | Requested Size: 181.64MiB | in_use: 1 | bin_num: -1, next:   Size: 968.77MiB | Requested Size: 968.77MiB | in_use: 1 | bin_num: -1
2020-10-05 10:08:09.973837: I tensorflow/core/common_runtime/bfc_allocator.cc:891]   Size: 534.30MiB | Requested Size: 17.40MiB | in_use: 0 | bin_num: 20, prev:   Size: 968.77MiB | Requested Size: 968.77MiB | in_use: 1 | bin_num: -1
2020-10-05 10:08:09.973852: I tensorflow/core/common_runtime/bfc_allocator.cc:891]   Size: 1.18GiB | Requested Size: 1.61GiB | in_use: 0 | bin_num: 20, prev:   Size: 1.89GiB | Requested Size: 1.89GiB | in_use: 1 | bin_num: -1
2020-10-05 10:08:09.973861: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 7365178112
2020-10-05 10:08:09.973872: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fd9fa000000 next 68 of size 4063297536
2020-10-05 10:08:09.973880: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdaec310000 next 74 of size 2031648768
2020-10-05 10:08:09.973888: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdb65498000 next 18446744073709551615 of size 1270231808
2020-10-05 10:08:09.973896: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 4294967296
2020-10-05 10:08:09.973905: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdbd0000000 next 71 of size 4063297536
2020-10-05 10:08:09.973913: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdcc2310000 next 18446744073709551615 of size 231669760
2020-10-05 10:08:09.973921: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 2147483648
2020-10-05 10:08:09.973929: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdcd0000000 next 66 of size 190467072
2020-10-05 10:08:09.973937: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdcdb5a4c00 next 69 of size 380934144
2020-10-05 10:08:09.973946: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdcf20ee400 next 72 of size 1015824384
2020-10-05 10:08:09.973954: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdd2e9b2400 next 18446744073709551615 of size 560258048
2020-10-05 10:08:09.973962: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1073741824
2020-10-05 10:08:09.973971: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdd50000000 next 54 of size 508592128
2020-10-05 10:08:09.973996: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdd6e508000 next 57 of size 190467072
2020-10-05 10:08:09.974005: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdd79aacc00 next 59 of size 127148032
2020-10-05 10:08:09.974014: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdd813eec00 next 18446744073709551615 of size 247534592
2020-10-05 10:08:09.974023: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 536870912
2020-10-05 10:08:09.974031: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddb0000000 next 10 of size 190467072
2020-10-05 10:08:09.974040: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddbb5a4c00 next 11 of size 190467072
2020-10-05 10:08:09.974049: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6b49800 next 18 of size 512
2020-10-05 10:08:09.974058: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6b49a00 next 19 of size 1179648
2020-10-05 10:08:09.974067: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6c69a00 next 20 of size 1024
2020-10-05 10:08:09.974075: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6c69e00 next 21 of size 2359296
2020-10-05 10:08:09.974084: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6ea9e00 next 22 of size 1024
2020-10-05 10:08:09.974093: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6eaa200 next 23 of size 2359296
2020-10-05 10:08:09.974102: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc70ea200 next 24 of size 1024
2020-10-05 10:08:09.974110: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc70ea600 next 25 of size 2359296
2020-10-05 10:08:09.974119: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc732a600 next 26 of size 1024
2020-10-05 10:08:09.974128: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc732aa00 next 27 of size 4718592
2020-10-05 10:08:09.974137: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc77aaa00 next 28 of size 2048
2020-10-05 10:08:09.974145: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc77ab200 next 29 of size 9437184
2020-10-05 10:08:09.974154: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc80ab200 next 30 of size 2048
2020-10-05 10:08:09.974163: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc80aba00 next 31 of size 9437184
2020-10-05 10:08:09.974172: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc89aba00 next 32 of size 2048
2020-10-05 10:08:09.974180: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc89ac200 next 33 of size 9437184
2020-10-05 10:08:09.974189: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc92ac200 next 34 of size 2048
2020-10-05 10:08:09.974198: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc92aca00 next 35 of size 9437184
2020-10-05 10:08:09.974206: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc9baca00 next 36 of size 2048
2020-10-05 10:08:09.974215: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc9bad200 next 37 of size 9437184
2020-10-05 10:08:09.974224: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4ad200 next 38 of size 2048
2020-10-05 10:08:09.974232: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4ada00 next 39 of size 16384
2020-10-05 10:08:09.974241: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4b1a00 next 40 of size 65536
2020-10-05 10:08:09.974250: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4c1a00 next 41 of size 262144
2020-10-05 10:08:09.974267: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca501a00 next 42 of size 1048576
2020-10-05 10:08:09.974277: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca601a00 next 43 of size 1048576
2020-10-05 10:08:09.974287: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701a00 next 44 of size 256
2020-10-05 10:08:09.974296: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701b00 next 45 of size 256
2020-10-05 10:08:09.974304: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701c00 next 46 of size 256
2020-10-05 10:08:09.974313: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701d00 next 47 of size 256
2020-10-05 10:08:09.974322: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701e00 next 48 of size 256
2020-10-05 10:08:09.974330: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701f00 next 49 of size 256
2020-10-05 10:08:09.974339: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702000 next 50 of size 256
2020-10-05 10:08:09.974348: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702100 next 51 of size 256
2020-10-05 10:08:09.974356: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702200 next 52 of size 256
2020-10-05 10:08:09.974365: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702300 next 55 of size 256
2020-10-05 10:08:09.974373: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702400 next 56 of size 256
2020-10-05 10:08:09.974382: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702500 next 58 of size 256
2020-10-05 10:08:09.974402: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702600 next 60 of size 256
2020-10-05 10:08:09.974410: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702700 next 61 of size 256
2020-10-05 10:08:09.974418: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702800 next 62 of size 256
2020-10-05 10:08:09.974427: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702900 next 63 of size 256
2020-10-05 10:08:09.974436: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702a00 next 64 of size 256
2020-10-05 10:08:09.974444: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fddca702b00 next 18446744073709551615 of size 93312256
2020-10-05 10:08:09.974452: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 268435456
2020-10-05 10:08:09.974460: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddd0000000 next 18446744073709551615 of size 268435456
2020-10-05 10:08:09.974468: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 268435456
2020-10-05 10:08:09.974476: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdde0000000 next 18446744073709551615 of size 268435456
2020-10-05 10:08:09.974484: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1048576
2020-10-05 10:08:09.974492: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00000 next 1 of size 256
2020-10-05 10:08:09.974498: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00100 next 2 of size 256
2020-10-05 10:08:09.974503: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00200 next 3 of size 256
2020-10-05 10:08:09.974511: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00300 next 5 of size 1280
2020-10-05 10:08:09.974519: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00800 next 6 of size 256
2020-10-05 10:08:09.974527: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00900 next 8 of size 256
2020-10-05 10:08:09.974535: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00a00 next 12 of size 6912
2020-10-05 10:08:09.974543: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec02500 next 13 of size 256
2020-10-05 10:08:09.974552: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec02600 next 14 of size 147456
2020-10-05 10:08:09.974560: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec26600 next 15 of size 256
2020-10-05 10:08:09.974568: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec26700 next 16 of size 294912
2020-10-05 10:08:09.974577: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec6e700 next 17 of size 512
2020-10-05 10:08:09.974585: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec6e900 next 18446744073709551615 of size 595712
2020-10-05 10:08:09.974593: I tensorflow/core/common_runtime/bfc_allocator.cc:914]      Summary of in-use Chunks by size: 
2020-10-05 10:08:09.974602: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 24 Chunks of size 256 totalling 6.0KiB
2020-10-05 10:08:09.974611: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 512 totalling 1.0KiB
2020-10-05 10:08:09.974620: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 1024 totalling 4.0KiB
2020-10-05 10:08:09.974628: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1280 totalling 1.2KiB
2020-10-05 10:08:09.974634: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 6 Chunks of size 2048 totalling 12.0KiB
2020-10-05 10:08:09.974642: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 6912 totalling 6.8KiB
2020-10-05 10:08:09.974651: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 16384 totalling 16.0KiB
2020-10-05 10:08:09.979417: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 65536 totalling 64.0KiB
2020-10-05 10:08:09.979430: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 147456 totalling 144.0KiB
2020-10-05 10:08:09.979437: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 262144 totalling 256.0KiB
2020-10-05 10:08:09.979443: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 294912 totalling 288.0KiB
2020-10-05 10:08:09.979451: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 595712 totalling 581.8KiB
2020-10-05 10:08:09.979458: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 1048576 totalling 2.00MiB
2020-10-05 10:08:09.979465: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1179648 totalling 1.12MiB
2020-10-05 10:08:09.979474: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 2359296 totalling 6.75MiB
2020-10-05 10:08:09.979482: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 4718592 totalling 4.50MiB
2020-10-05 10:08:09.979490: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 5 Chunks of size 9437184 totalling 45.00MiB
2020-10-05 10:08:09.979498: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 127148032 totalling 121.26MiB
2020-10-05 10:08:09.979507: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 190467072 totalling 726.57MiB
2020-10-05 10:08:09.979516: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 268435456 totalling 512.00MiB
2020-10-05 10:08:09.979527: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 508592128 totalling 485.03MiB
2020-10-05 10:08:09.979539: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1015824384 totalling 968.77MiB
2020-10-05 10:08:09.979550: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 2031648768 totalling 1.89GiB
2020-10-05 10:08:09.979561: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 4063297536 totalling 7.57GiB
2020-10-05 10:08:09.979573: I tensorflow/core/common_runtime/bfc_allocator.cc:921] Sum Total of in-use chunks: 12.27GiB
2020-10-05 10:08:09.979582: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_: 15956161280 memory_limit_: 15956161332 available bytes: 52 curr_region_allocation_bytes_: 17179869184
2020-10-05 10:08:09.979595: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats: 
Limit:                 15956161332
InUse:                 13172220672
MaxInUse:              14125671424
NumAllocs:                     100
MaxAllocSize:           4080074752

2020-10-05 10:08:09.979611: W tensorflow/core/common_runtime/bfc_allocator.cc:424] ***************************************_______**************************_**_********__******_*******
2020-10-05 10:08:09.979637: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at conv_ops.cc:500 : Resource exhausted: OOM when allocating tensor with shape[1,128,1494,2656] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2020-10-05 10:08:21.046053: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.89GiB (rounded to 2031648768).  Current allocation summary follows.
2020-10-05 10:08:21.046109: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (256):   Total Chunks: 24, Chunks in use: 24. 6.0KiB allocated for chunks. 6.0KiB in use in bin. 600B client-requested in use in bin.
2020-10-05 10:08:21.046127: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (512):   Total Chunks: 2, Chunks in use: 2. 1.0KiB allocated for chunks. 1.0KiB in use in bin. 1.0KiB client-requested in use in bin.
2020-10-05 10:08:21.046135: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2020-10-05 10:08:21.046146: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2048):  Total Chunks: 6, Chunks in use: 6. 12.0KiB allocated for chunks. 12.0KiB in use in bin. 12.0KiB client-requested in use in bin.
2020-10-05 10:08:21.046156: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2020-10-05 10:08:21.046165: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:08:21.046175: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16384):     Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2020-10-05 10:08:21.046190: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (32768):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:08:21.046204: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (65536):     Total Chunks: 1, Chunks in use: 1. 64.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-10-05 10:08:21.046213: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (131072):    Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2020-10-05 10:08:21.046223: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (262144):    Total Chunks: 2, Chunks in use: 2. 544.0KiB allocated for chunks. 544.0KiB in use in bin. 544.0KiB client-requested in use in bin.
2020-10-05 10:08:21.046233: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (524288):    Total Chunks: 1, Chunks in use: 1. 581.8KiB allocated for chunks. 581.8KiB in use in bin. 576.0KiB client-requested in use in bin.
2020-10-05 10:08:21.046243: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1048576):   Total Chunks: 3, Chunks in use: 3. 3.12MiB allocated for chunks. 3.12MiB in use in bin. 3.12MiB client-requested in use in bin.
2020-10-05 10:08:21.046253: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2097152):   Total Chunks: 3, Chunks in use: 3. 6.75MiB allocated for chunks. 6.75MiB in use in bin. 6.75MiB client-requested in use in bin.
2020-10-05 10:08:21.046262: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4194304):   Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2020-10-05 10:08:21.046272: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8388608):   Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2020-10-05 10:08:21.046281: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16777216):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:08:21.046290: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (33554432):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-05 10:08:21.046300: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (67108864):  Total Chunks: 2, Chunks in use: 1. 210.25MiB allocated for chunks. 121.26MiB in use in bin. 121.26MiB client-requested in use in bin.
2020-10-05 10:08:21.046311: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (134217728):     Total Chunks: 6, Chunks in use: 4. 1.16GiB allocated for chunks. 726.57MiB in use in bin. 726.57MiB client-requested in use in bin.
2020-10-05 10:08:21.046321: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (268435456):     Total Chunks: 10, Chunks in use: 7. 13.44GiB allocated for chunks. 11.38GiB in use in bin. 11.23GiB client-requested in use in bin.
2020-10-05 10:08:21.046331: I tensorflow/core/common_runtime/bfc_allocator.cc:885] Bin for 1.89GiB was 256.00MiB, Chunk State: 
2020-10-05 10:08:21.046347: I tensorflow/core/common_runtime/bfc_allocator.cc:891]   Size: 363.29MiB | Requested Size: 181.64MiB | in_use: 0 | bin_num: 20, prev:   Size: 181.64MiB | Requested Size: 181.64MiB | in_use: 1 | bin_num: -1, next:   Size: 968.77MiB | Requested Size: 968.77MiB | in_use: 1 | bin_num: -1
2020-10-05 10:08:21.046360: I tensorflow/core/common_runtime/bfc_allocator.cc:891]   Size: 534.30MiB | Requested Size: 17.40MiB | in_use: 0 | bin_num: 20, prev:   Size: 968.77MiB | Requested Size: 968.77MiB | in_use: 1 | bin_num: -1
2020-10-05 10:08:21.046370: I tensorflow/core/common_runtime/bfc_allocator.cc:891]   Size: 1.18GiB | Requested Size: 1.61GiB | in_use: 0 | bin_num: 20, prev:   Size: 1.89GiB | Requested Size: 1.89GiB | in_use: 1 | bin_num: -1
2020-10-05 10:08:21.046378: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 7365178112
2020-10-05 10:08:21.046389: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fd9fa000000 next 68 of size 4063297536
2020-10-05 10:08:21.046397: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdaec310000 next 74 of size 2031648768
2020-10-05 10:08:21.046406: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdb65498000 next 18446744073709551615 of size 1270231808
2020-10-05 10:08:21.046413: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 4294967296
2020-10-05 10:08:21.046421: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdbd0000000 next 71 of size 4063297536
2020-10-05 10:08:21.046429: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdcc2310000 next 18446744073709551615 of size 231669760
2020-10-05 10:08:21.046437: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 2147483648
2020-10-05 10:08:21.046445: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdcd0000000 next 66 of size 190467072
2020-10-05 10:08:21.046454: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdcdb5a4c00 next 69 of size 380934144
2020-10-05 10:08:21.046462: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdcf20ee400 next 72 of size 1015824384
2020-10-05 10:08:21.046470: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdd2e9b2400 next 18446744073709551615 of size 560258048
2020-10-05 10:08:21.046478: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1073741824
2020-10-05 10:08:21.046486: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdd50000000 next 54 of size 508592128
2020-10-05 10:08:21.046494: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdd6e508000 next 57 of size 190467072
2020-10-05 10:08:21.046502: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdd79aacc00 next 59 of size 127148032
2020-10-05 10:08:21.046510: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fdd813eec00 next 18446744073709551615 of size 247534592
2020-10-05 10:08:21.046518: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 536870912
2020-10-05 10:08:21.046526: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddb0000000 next 10 of size 190467072
2020-10-05 10:08:21.046534: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddbb5a4c00 next 11 of size 190467072
2020-10-05 10:08:21.046542: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6b49800 next 18 of size 512
2020-10-05 10:08:21.046551: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6b49a00 next 19 of size 1179648
2020-10-05 10:08:21.046577: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6c69a00 next 20 of size 1024
2020-10-05 10:08:21.046585: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6c69e00 next 21 of size 2359296
2020-10-05 10:08:21.046594: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6ea9e00 next 22 of size 1024
2020-10-05 10:08:21.046603: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc6eaa200 next 23 of size 2359296
2020-10-05 10:08:21.046611: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc70ea200 next 24 of size 1024
2020-10-05 10:08:21.046620: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc70ea600 next 25 of size 2359296
2020-10-05 10:08:21.046628: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc732a600 next 26 of size 1024
2020-10-05 10:08:21.046637: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc732aa00 next 27 of size 4718592
2020-10-05 10:08:21.046646: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc77aaa00 next 28 of size 2048
2020-10-05 10:08:21.046654: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc77ab200 next 29 of size 9437184
2020-10-05 10:08:21.046663: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc80ab200 next 30 of size 2048
2020-10-05 10:08:21.046672: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc80aba00 next 31 of size 9437184
2020-10-05 10:08:21.046681: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc89aba00 next 32 of size 2048
2020-10-05 10:08:21.046689: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc89ac200 next 33 of size 9437184
2020-10-05 10:08:21.046698: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc92ac200 next 34 of size 2048
2020-10-05 10:08:21.046706: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc92aca00 next 35 of size 9437184
2020-10-05 10:08:21.046715: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc9baca00 next 36 of size 2048
2020-10-05 10:08:21.046723: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddc9bad200 next 37 of size 9437184
2020-10-05 10:08:21.046732: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4ad200 next 38 of size 2048
2020-10-05 10:08:21.046741: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4ada00 next 39 of size 16384
2020-10-05 10:08:21.046760: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4b1a00 next 40 of size 65536
2020-10-05 10:08:21.046775: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca4c1a00 next 41 of size 262144
2020-10-05 10:08:21.046794: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca501a00 next 42 of size 1048576
2020-10-05 10:08:21.046802: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca601a00 next 43 of size 1048576
2020-10-05 10:08:21.046812: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701a00 next 44 of size 256
2020-10-05 10:08:21.046821: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701b00 next 45 of size 256
2020-10-05 10:08:21.046829: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701c00 next 46 of size 256
2020-10-05 10:08:21.046838: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701d00 next 47 of size 256
2020-10-05 10:08:21.046847: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701e00 next 48 of size 256
2020-10-05 10:08:21.046855: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca701f00 next 49 of size 256
2020-10-05 10:08:21.046864: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702000 next 50 of size 256
2020-10-05 10:08:21.046872: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702100 next 51 of size 256
2020-10-05 10:08:21.046881: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702200 next 52 of size 256
2020-10-05 10:08:21.046889: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702300 next 55 of size 256
2020-10-05 10:08:21.046898: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702400 next 56 of size 256
2020-10-05 10:08:21.046906: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702500 next 58 of size 256
2020-10-05 10:08:21.046915: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702600 next 60 of size 256
2020-10-05 10:08:21.046923: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702700 next 61 of size 256
2020-10-05 10:08:21.046932: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702800 next 62 of size 256
2020-10-05 10:08:21.046940: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702900 next 63 of size 256
2020-10-05 10:08:21.046948: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddca702a00 next 64 of size 256
2020-10-05 10:08:21.046957: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free  at 0x7fddca702b00 next 18446744073709551615 of size 93312256
2020-10-05 10:08:21.046965: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 268435456
2020-10-05 10:08:21.046974: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fddd0000000 next 18446744073709551615 of size 268435456
2020-10-05 10:08:21.046982: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 268435456
2020-10-05 10:08:21.046991: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7fdde0000000 next 18446744073709551615 of size 268435456
2020-10-05 10:08:21.046999: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1048576
2020-10-05 10:08:21.047008: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00000 next 1 of size 256
2020-10-05 10:08:21.047016: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00100 next 2 of size 256
2020-10-05 10:08:21.047025: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00200 next 3 of size 256
2020-10-05 10:08:21.047033: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00300 next 5 of size 1280
2020-10-05 10:08:21.047042: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00800 next 6 of size 256
2020-10-05 10:08:21.047051: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00900 next 8 of size 256
2020-10-05 10:08:21.047059: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec00a00 next 12 of size 6912
2020-10-05 10:08:21.047068: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec02500 next 13 of size 256
2020-10-05 10:08:21.047089: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec02600 next 14 of size 147456
2020-10-05 10:08:21.047097: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec26600 next 15 of size 256
2020-10-05 10:08:21.047106: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec26700 next 16 of size 294912
2020-10-05 10:08:21.047114: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec6e700 next 17 of size 512
2020-10-05 10:08:21.047122: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7ff6eec6e900 next 18446744073709551615 of size 595712
2020-10-05 10:08:21.047130: I tensorflow/core/common_runtime/bfc_allocator.cc:914]      Summary of in-use Chunks by size: 
2020-10-05 10:08:21.047140: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 24 Chunks of size 256 totalling 6.0KiB
2020-10-05 10:08:21.047148: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 512 totalling 1.0KiB
2020-10-05 10:08:21.047157: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 1024 totalling 4.0KiB
2020-10-05 10:08:21.047165: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1280 totalling 1.2KiB
2020-10-05 10:08:21.047174: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 6 Chunks of size 2048 totalling 12.0KiB
2020-10-05 10:08:21.047182: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 6912 totalling 6.8KiB
2020-10-05 10:08:21.047196: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 16384 totalling 16.0KiB
2020-10-05 10:08:21.117247: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 65536 totalling 64.0KiB
2020-10-05 10:08:21.117287: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 147456 totalling 144.0KiB
2020-10-05 10:08:21.117297: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 262144 totalling 256.0KiB
2020-10-05 10:08:21.117309: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 294912 totalling 288.0KiB
2020-10-05 10:08:21.117320: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 595712 totalling 581.8KiB
2020-10-05 10:08:21.117337: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 1048576 totalling 2.00MiB
2020-10-05 10:08:21.117349: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1179648 totalling 1.12MiB
2020-10-05 10:08:21.117366: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 2359296 totalling 6.75MiB
2020-10-05 10:08:21.117375: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 4718592 totalling 4.50MiB
2020-10-05 10:08:21.117382: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 5 Chunks of size 9437184 totalling 45.00MiB
2020-10-05 10:08:21.117388: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 127148032 totalling 121.26MiB
2020-10-05 10:08:21.117399: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 190467072 totalling 726.57MiB
2020-10-05 10:08:21.117412: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 268435456 totalling 512.00MiB
2020-10-05 10:08:21.117424: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 508592128 totalling 485.03MiB
2020-10-05 10:08:21.117436: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1015824384 totalling 968.77MiB
2020-10-05 10:08:21.117448: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 2031648768 totalling 1.89GiB
2020-10-05 10:08:21.117460: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 4063297536 totalling 7.57GiB
2020-10-05 10:08:21.117472: I tensorflow/core/common_runtime/bfc_allocator.cc:921] Sum Total of in-use chunks: 12.27GiB
2020-10-05 10:08:21.117484: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_: 15956161280 memory_limit_: 15956161332 available bytes: 52 curr_region_allocation_bytes_: 17179869184
2020-10-05 10:08:21.117501: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats: 
Limit:                 15956161332
InUse:                 13172220672
MaxInUse:              14125671424
NumAllocs:                     100
MaxAllocSize:           4080074752

2020-10-05 10:08:21.117525: W tensorflow/core/common_runtime/bfc_allocator.cc:424] ***************************************_______**************************_**_********__******_*******
2020-10-05 10:08:21.117560: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at transpose_op.cc:198 : Resource exhausted: OOM when allocating tensor with shape[1,1494,2656,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,2988,5312,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[{{node Relu-0-1-TransposeNCHWToNHWC-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "neural_style.py", line 289, in <module>
    main()
  File "neural_style.py", line 236, in main
    checkpoint_iterations=options.checkpoint_iterations,
  File "/content/neural-style/stylize.py", line 191, in stylize
    train_step.run()
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 2439, in run
    _run_using_default_session(self, feed_dict, self.graph, session)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 5442, in _run_using_default_session
    session.run(operation, feed_dict)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,2988,5312,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[{{node Relu-0-1-TransposeNCHWToNHWC-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
anishathalye commented 3 years ago

https://github.com/anishathalye/neural-style/issues?q=out+of+memory