Ran out of memory trying to allocate XX.XX MB

[ikocev]

Hello, I was really happy to discover your port and I am really looking forward to use it a lot in future as it is very promising. However, I am facing an error and I am not sure how to resolve it. I am running it on new MacBook Pro with Intel and AMD Radeon pro 450 on it (2017 version). I also set the CL_GPUOFFSET to 1, because Intel graphics is the first one. This is the logs:

W tensorflow/core/common_runtime/bfc_allocator.cc:274] *****_**********__************************************************************************xxxxxxxxxx
W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 9.00MiB.  See logs for memory state.
W tensorflow/core/framework/op_kernel.cc:973] Resource exhausted: OOM when allocating tensor with shape[512,512,3,3]
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096):  Total Chunks: 1, Chunks in use: 0 6.8KiB allocated for chunks. 6.8KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768):     Total Chunks: 1, Chunks in use: 0 54.2KiB allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072):    Total Chunks: 1, Chunks in use: 0 144.0KiB allocated for chunks. 144.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144):    Total Chunks: 1, Chunks in use: 0 288.0KiB allocated for chunks. 288.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288):    Total Chunks: 1, Chunks in use: 0 576.0KiB allocated for chunks. 576.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576):   Total Chunks: 1, Chunks in use: 0 1.12MiB allocated for chunks. 1.12MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152):   Total Chunks: 2, Chunks in use: 0 4.50MiB allocated for chunks. 2.25MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304):   Total Chunks: 1, Chunks in use: 0 4.50MiB allocated for chunks. 4.50MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608):   Total Chunks: 1, Chunks in use: 0 9.00MiB allocated for chunks. 9.00MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 64.00MiB was 64.00MiB, Chunk State: 
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x80 of size 1280
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x580 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x680 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x780 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x880 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x980 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xb80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xc80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xd80 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1180 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1280 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1380 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1b80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1c80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1d80 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x5d80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x5e80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x5f80 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x6f80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7080 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7180 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7280 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7380 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7480 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7580 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7680 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7780 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7880 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7980 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7a80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7b80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7c80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7d80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x7e80 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x8080 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x8180 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x8280 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x8680 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x8780 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x8880 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x9580 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x9680 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x9780 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x9880 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xd880 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xd980 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xda80 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xea80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xeb80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xec80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xed80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xee80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xef80 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xf080 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xf180 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xf280 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xf380 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xf480 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x11080 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x11180 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x11280 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x11480 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x11680 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x11a80 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x11e80 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12280 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12a80 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13280 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13a80 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x14280 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x14a80 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x15280 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x19280 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1d280 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1e280 of size 6912
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x2d680 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x51780 of size 147456
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x75780 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xbd980 of size 294912
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x105980 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x195b80 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x225b80 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x225f80 of size 2359296
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x585f80 of size 1179648
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x6a5f80 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x6a6380 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x6a6780 of size 2359296
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xb26780 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0xb26f80 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1426f80 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x21a6f80 of size 2359296
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x2626f80 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x2f26f80 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x3826f80 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x3827780 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x3827f80 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x3828780 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x3828f80 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x3829780 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x382d780 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x412d780 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x4a2d780 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x532d780 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x652d780 of size 86845696
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0xf580 of size 6912
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1fd80 of size 55552
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x2d780 of size 147456
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x75980 of size 294912
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x105b80 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x465f80 of size 1179648
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x8e6780 of size 2359296
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1d26f80 of size 4718592
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x23e6f80 of size 2359296
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x5c2d780 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:693]      Summary of in-use Chunks by size: 
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 50 Chunks of size 256 totalling 12.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 512 totalling 3.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 8 Chunks of size 1024 totalling 8.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 13 Chunks of size 2048 totalling 26.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 3328 totalling 3.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 4096 totalling 12.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 6912 totalling 6.8KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 16384 totalling 80.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 147456 totalling 144.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 294912 totalling 288.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 589824 totalling 576.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1179648 totalling 1.12MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 2359296 totalling 6.75MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 8 Chunks of size 9437184 totalling 72.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 86845696 totalling 82.82MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 163.83MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats: 
Limit:                   192937984
InUse:                   171789312
MaxInUse:                192937984
NumAllocs:                     141
MaxAllocSize:            134031616

W tensorflow/core/common_runtime/bfc_allocator.cc:274] *****_**********__*********************************___************************************xxxxxxxxxx
W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 64.00MiB.  See logs for memory state.
W tensorflow/core/framework/op_kernel.cc:973] Resource exhausted: OOM when allocating tensor with shape[4096,4096]

[mihailescu2m]

Limit: 192MB InUse: 171MB Ran out of memory trying to allocate 64.00MiB 171+64 > 192

For some reason, it limits the memory to 192MB.... have you found a way to increase that?