GPU runs out of memory Issue

[seo-dev]

Hi Anjul Tyagi

I'm using single GPU ( RTX 2070 SUPER )

Changing the batch size from 4 to 2 seems to solve the problem.

Below is the complete log where the problem occurs. (It's pretty long...)

2020-10-01 02:17:37.628699: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-10-01 02:17:39.691247: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1 2020-10-01 02:17:39.722748: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 02:17:39.723410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5 coreClock: 1.815GHz coreCount: 40 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s 2020-10-01 02:17:39.723428: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-10-01 02:17:39.727685: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2020-10-01 02:17:39.730038: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10 2020-10-01 02:17:39.732899: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10 2020-10-01 02:17:39.736958: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10 2020-10-01 02:17:39.738226: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10 2020-10-01 02:17:39.743865: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2020-10-01 02:17:39.744026: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 02:17:39.744413: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 02:17:39.744921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2020-10-01 02:17:39.745117: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-10-01 02:17:39.752118: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3600000000 Hz 2020-10-01 02:17:39.752324: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558a181f3d20 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-10-01 02:17:39.752339: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-10-01 02:17:39.822090: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 02:17:39.822442: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558a1701aa50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-10-01 02:17:39.822457: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2070 SUPER, Compute Capability 7.5 2020-10-01 02:17:39.822595: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 02:17:39.822863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5 coreClock: 1.815GHz coreCount: 40 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s 2020-10-01 02:17:39.822882: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-10-01 02:17:39.822897: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2020-10-01 02:17:39.822908: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10 2020-10-01 02:17:39.822918: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10 2020-10-01 02:17:39.822927: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10 2020-10-01 02:17:39.822960: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10 2020-10-01 02:17:39.822970: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2020-10-01 02:17:39.823024: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 02:17:39.823299: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 02:17:39.823545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2020-10-01 02:17:39.823565: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-10-01 02:17:40.336218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-10-01 02:17:40.336260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0 2020-10-01 02:17:40.336268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N 2020-10-01 02:17:40.336419: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 02:17:40.336756: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 02:17:40.337045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6829 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5) Amount of training samples: 7481 2020-10-01 02:17:40.855393: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started. 2020-10-01 02:17:40.855431: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1391] Profiler found 1 GPUs 2020-10-01 02:17:40.859362: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcupti.so.10.1 2020-10-01 02:17:40.959867: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1441] function cuptiinterface->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES 2020-10-01 02:17:41.448717: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 134400000 exceeds 10% of free system memory.

Epoch 00001: LearningRateScheduler reducing learning rate to 0.00019999999494757503. Epoch 1/160 2020-10-01 02:17:45.859263: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 134400000 exceeds 10% of free system memory. 2020-10-01 02:17:45.965246: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2020-10-01 02:17:47.726050: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2020-10-01 02:17:48.489603: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.80GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-10-01 02:17:48.506336: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-10-01 02:17:48.599886: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-10-01 02:17:48.599986: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-10-01 02:17:48.619758: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-10-01 02:17:48.658248: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 801.94MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-10-01 02:17:48.675404: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 801.94MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-10-01 02:17:48.687687: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.07GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-10-01 02:17:48.709399: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 388.09MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-10-01 02:17:48.709431: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2020-10-01 02:17:48.728799: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.31GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-10-01 02:17:58.718398: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_0_bfc) ran out of memory trying to allocate 372.09MiB (rounded to 390168576)requested by op gradient_tape/functional_1/angle/conv2d/Conv2D/Conv2DBackpropInput Current allocation summary follows. 2020-10-01 02:17:58.718495: I tensorflow/core/common_runtime/bfc_allocator.cc:970] BFCAllocator dump for GPU_0_bfc 2020-10-01 02:17:58.718539: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (256): Total Chunks: 155, Chunks in use: 155. 38.8KiB allocated for chunks. 38.8KiB in use in bin. 16.5KiB client-requested in use in bin. 2020-10-01 02:17:58.718589: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (512): Total Chunks: 196, Chunks in use: 195. 98.0KiB allocated for chunks. 97.5KiB in use in bin. 97.5KiB client-requested in use in bin. 2020-10-01 02:17:58.718621: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1024): Total Chunks: 4, Chunks in use: 4. 6.5KiB allocated for chunks. 6.5KiB in use in bin. 6.3KiB client-requested in use in bin. 2020-10-01 02:17:58.718649: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2048): Total Chunks: 3, Chunks in use: 0. 7.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-10-01 02:17:58.718682: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4096): Total Chunks: 13, Chunks in use: 12. 78.2KiB allocated for chunks. 72.2KiB in use in bin. 72.0KiB client-requested in use in bin. 2020-10-01 02:17:58.718710: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-10-01 02:17:58.718744: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16384): Total Chunks: 12, Chunks in use: 11. 258.5KiB allocated for chunks. 228.5KiB in use in bin. 216.0KiB client-requested in use in bin. 2020-10-01 02:17:58.718777: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (32768): Total Chunks: 1, Chunks in use: 0. 48.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-10-01 02:17:58.718809: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-10-01 02:17:58.718839: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (131072): Total Chunks: 11, Chunks in use: 11. 1.55MiB allocated for chunks. 1.55MiB in use in bin. 1.55MiB client-requested in use in bin. 2020-10-01 02:17:58.718870: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (262144): Total Chunks: 7, Chunks in use: 7. 2.11MiB allocated for chunks. 2.11MiB in use in bin. 1.83MiB client-requested in use in bin. 2020-10-01 02:17:58.718904: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (524288): Total Chunks: 40, Chunks in use: 40. 22.49MiB allocated for chunks. 22.49MiB in use in bin. 22.49MiB client-requested in use in bin. 2020-10-01 02:17:58.718934: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-10-01 02:17:58.718964: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2097152): Total Chunks: 4, Chunks in use: 2. 14.53MiB allocated for chunks. 7.75MiB in use in bin. 7.75MiB client-requested in use in bin. 2020-10-01 02:17:58.718995: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4194304): Total Chunks: 13, Chunks in use: 13. 100.77MiB allocated for chunks. 100.77MiB in use in bin. 96.90MiB client-requested in use in bin. 2020-10-01 02:17:58.719034: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8388608): Total Chunks: 4, Chunks in use: 3. 45.61MiB allocated for chunks. 34.97MiB in use in bin. 34.97MiB client-requested in use in bin. 2020-10-01 02:17:58.719067: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16777216): Total Chunks: 13, Chunks in use: 12. 388.42MiB allocated for chunks. 372.09MiB in use in bin. 372.09MiB client-requested in use in bin. 2020-10-01 02:17:58.719098: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (33554432): Total Chunks: 8, Chunks in use: 7. 468.99MiB allocated for chunks. 434.11MiB in use in bin. 434.11MiB client-requested in use in bin. 2020-10-01 02:17:58.719134: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (67108864): Total Chunks: 6, Chunks in use: 4. 721.11MiB allocated for chunks. 488.55MiB in use in bin. 434.11MiB client-requested in use in bin. 2020-10-01 02:17:58.719168: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (134217728): Total Chunks: 4, Chunks in use: 2. 843.65MiB allocated for chunks. 376.24MiB in use in bin. 376.24MiB client-requested in use in bin. 2020-10-01 02:17:58.719199: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (268435456): Total Chunks: 7, Chunks in use: 7. 4.12GiB allocated for chunks. 4.12GiB in use in bin. 4.11GiB client-requested in use in bin. 2020-10-01 02:17:58.719230: I tensorflow/core/common_runtime/bfc_allocator.cc:993] Bin for 372.09MiB was 256.00MiB, Chunk State: 2020-10-01 02:17:58.719253: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 7161754368 2020-10-01 02:17:58.719285: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000000 of size 1280 next 1 2020-10-01 02:17:58.719311: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000500 of size 256 next 5 2020-10-01 02:17:58.719334: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000600 of size 256 next 8 2020-10-01 02:17:58.719357: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000700 of size 256 next 9 2020-10-01 02:17:58.719380: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000800 of size 256 next 10 2020-10-01 02:17:58.719403: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000900 of size 256 next 410 2020-10-01 02:17:58.719426: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000a00 of size 256 next 12 2020-10-01 02:17:58.719449: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000b00 of size 256 next 2 2020-10-01 02:17:58.719473: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000c00 of size 256 next 3 2020-10-01 02:17:58.719496: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000d00 of size 256 next 4 2020-10-01 02:17:58.719519: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000e00 of size 256 next 14 2020-10-01 02:17:58.719541: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000f00 of size 256 next 13 2020-10-01 02:17:58.719565: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001000 of size 256 next 17 2020-10-01 02:17:58.719588: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001100 of size 256 next 18 2020-10-01 02:17:58.719611: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001200 of size 256 next 414 2020-10-01 02:17:58.719634: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001300 of size 256 next 20 2020-10-01 02:17:58.719657: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001400 of size 256 next 21 2020-10-01 02:17:58.719680: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001500 of size 256 next 6 2020-10-01 02:17:58.719704: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001600 of size 1792 next 7 2020-10-01 02:17:58.719727: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001d00 of size 256 next 23 2020-10-01 02:17:58.719750: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001e00 of size 256 next 417 2020-10-01 02:17:58.719773: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001f00 of size 256 next 25 2020-10-01 02:17:58.719798: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268002000 of size 294144 next 16 2020-10-01 02:17:58.719822: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268049d00 of size 147456 next 15 2020-10-01 02:17:58.719845: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126806dd00 of size 147456 next 22 2020-10-01 02:17:58.719868: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268091d00 of size 256 next 26 2020-10-01 02:17:58.719910: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268091e00 of size 256 next 29 2020-10-01 02:17:58.719935: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268091f00 of size 256 next 30 2020-10-01 02:17:58.719958: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092000 of size 256 next 421 2020-10-01 02:17:58.719982: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092100 of size 256 next 32 2020-10-01 02:17:58.720005: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092200 of size 256 next 35 2020-10-01 02:17:58.720028: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092300 of size 256 next 36 2020-10-01 02:17:58.720052: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092400 of size 256 next 37 2020-10-01 02:17:58.720076: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092500 of size 512 next 39 2020-10-01 02:17:58.720099: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092700 of size 256 next 41 2020-10-01 02:17:58.720122: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092800 of size 256 next 42 2020-10-01 02:17:58.720146: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092900 of size 512 next 40 2020-10-01 02:17:58.720169: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092b00 of size 512 next 45 2020-10-01 02:17:58.720192: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092d00 of size 512 next 46 2020-10-01 02:17:58.720215: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092f00 of size 512 next 431 2020-10-01 02:17:58.720239: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093100 of size 512 next 48 2020-10-01 02:17:58.720262: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093300 of size 256 next 49 2020-10-01 02:17:58.720285: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093400 of size 256 next 50 2020-10-01 02:17:58.720309: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093500 of size 512 next 53 2020-10-01 02:17:58.720332: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093700 of size 512 next 54 2020-10-01 02:17:58.720356: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093900 of size 512 next 55 2020-10-01 02:17:58.720379: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093b00 of size 512 next 438 2020-10-01 02:17:58.720402: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093d00 of size 512 next 57 2020-10-01 02:17:58.720425: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093f00 of size 512 next 59 2020-10-01 02:17:58.720448: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094100 of size 512 next 60 2020-10-01 02:17:58.720472: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094300 of size 512 next 61 2020-10-01 02:17:58.720495: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094500 of size 512 next 441 2020-10-01 02:17:58.720518: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094700 of size 512 next 63 2020-10-01 02:17:58.720541: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094900 of size 512 next 65 2020-10-01 02:17:58.720564: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094b00 of size 512 next 66 2020-10-01 02:17:58.720587: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094d00 of size 512 next 67 2020-10-01 02:17:58.720611: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094f00 of size 512 next 445 2020-10-01 02:17:58.720634: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095100 of size 512 next 69 2020-10-01 02:17:58.720657: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095300 of size 512 next 71 2020-10-01 02:17:58.720681: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095500 of size 512 next 72 2020-10-01 02:17:58.720705: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095700 of size 512 next 73 2020-10-01 02:17:58.720728: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095900 of size 512 next 449 2020-10-01 02:17:58.720751: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095b00 of size 512 next 75 2020-10-01 02:17:58.720774: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095d00 of size 512 next 77 2020-10-01 02:17:58.720797: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095f00 of size 512 next 78 2020-10-01 02:17:58.720820: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096100 of size 512 next 79 2020-10-01 02:17:58.720844: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096300 of size 512 next 452 2020-10-01 02:17:58.720866: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096500 of size 512 next 81 2020-10-01 02:17:58.720890: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096700 of size 512 next 83 2020-10-01 02:17:58.720913: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096900 of size 512 next 84 2020-10-01 02:17:58.720936: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096b00 of size 512 next 85 2020-10-01 02:17:58.720959: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096d00 of size 512 next 459 2020-10-01 02:17:58.720983: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096f00 of size 512 next 87 2020-10-01 02:17:58.721006: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097100 of size 512 next 89 2020-10-01 02:17:58.721030: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097300 of size 512 next 90 2020-10-01 02:17:58.721053: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097500 of size 512 next 91 2020-10-01 02:17:58.721076: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097700 of size 512 next 466 2020-10-01 02:17:58.721099: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097900 of size 512 next 93 2020-10-01 02:17:58.721122: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097b00 of size 512 next 95 2020-10-01 02:17:58.721146: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097d00 of size 512 next 96 2020-10-01 02:17:58.721169: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097f00 of size 512 next 97 2020-10-01 02:17:58.721192: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098100 of size 512 next 469 2020-10-01 02:17:58.721215: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098300 of size 512 next 99 2020-10-01 02:17:58.721238: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098500 of size 512 next 101 2020-10-01 02:17:58.721262: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098700 of size 512 next 102 2020-10-01 02:17:58.721285: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098900 of size 512 next 103 2020-10-01 02:17:58.721308: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098b00 of size 512 next 473 2020-10-01 02:17:58.721331: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098d00 of size 512 next 105 2020-10-01 02:17:58.721354: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098f00 of size 512 next 107 2020-10-01 02:17:58.721378: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099100 of size 512 next 108 2020-10-01 02:17:58.721401: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099300 of size 512 next 109 2020-10-01 02:17:58.721424: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099500 of size 512 next 477 2020-10-01 02:17:58.721447: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099700 of size 512 next 111 2020-10-01 02:17:58.721470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099900 of size 512 next 113 2020-10-01 02:17:58.721493: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099b00 of size 512 next 114 2020-10-01 02:17:58.721516: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099d00 of size 512 next 115 2020-10-01 02:17:58.721540: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099f00 of size 512 next 480 2020-10-01 02:17:58.721563: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809a100 of size 512 next 117 2020-10-01 02:17:58.721586: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809a300 of size 512 next 118 2020-10-01 02:17:58.721609: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809a500 of size 512 next 120 2020-10-01 02:17:58.721632: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809a700 of size 512 next 121 2020-10-01 02:17:58.721655: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809a900 of size 512 next 439 2020-10-01 02:17:58.721678: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809ab00 of size 512 next 123 2020-10-01 02:17:58.721702: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809ad00 of size 512 next 126 2020-10-01 02:17:58.721725: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809af00 of size 512 next 127 2020-10-01 02:17:58.721748: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809b100 of size 512 next 128 2020-10-01 02:17:58.721771: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809b300 of size 512 next 467 2020-10-01 02:17:58.721794: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809b500 of size 512 next 130 2020-10-01 02:17:58.721818: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809b700 of size 512 next 132 2020-10-01 02:17:58.721841: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809b900 of size 512 next 133 2020-10-01 02:17:58.721864: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809bb00 of size 512 next 134 2020-10-01 02:17:58.721888: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809bd00 of size 256 next 513 2020-10-01 02:17:58.721911: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809be00 of size 256 next 509 2020-10-01 02:17:58.721934: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f126809bf00 of size 512 next 136 2020-10-01 02:17:58.721957: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c100 of size 256 next 140 2020-10-01 02:17:58.721981: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c200 of size 256 next 144 2020-10-01 02:17:58.722004: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c300 of size 256 next 143 2020-10-01 02:17:58.722028: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c400 of size 256 next 148 2020-10-01 02:17:58.722050: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c500 of size 256 next 151 2020-10-01 02:17:58.722074: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c600 of size 256 next 152 2020-10-01 02:17:58.722097: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c700 of size 256 next 153 2020-10-01 02:17:58.722120: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c800 of size 256 next 154 2020-10-01 02:17:58.722144: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c900 of size 256 next 157 2020-10-01 02:17:58.722166: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809ca00 of size 256 next 158 2020-10-01 02:17:58.722190: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809cb00 of size 256 next 159 2020-10-01 02:17:58.722213: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809cc00 of size 256 next 160 2020-10-01 02:17:58.722236: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809cd00 of size 256 next 161 2020-10-01 02:17:58.722259: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809ce00 of size 256 next 162 2020-10-01 02:17:58.722283: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809cf00 of size 256 next 163 2020-10-01 02:17:58.722306: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d000 of size 256 next 164 2020-10-01 02:17:58.722329: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d100 of size 256 next 165 2020-10-01 02:17:58.722352: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d200 of size 256 next 166 2020-10-01 02:17:58.722375: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d300 of size 256 next 167 2020-10-01 02:17:58.722398: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d400 of size 256 next 168 2020-10-01 02:17:58.722421: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d500 of size 256 next 169 2020-10-01 02:17:58.722444: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d600 of size 256 next 170 2020-10-01 02:17:58.722467: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d700 of size 256 next 171 2020-10-01 02:17:58.722490: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d800 of size 256 next 137 2020-10-01 02:17:58.722513: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d900 of size 256 next 138 2020-10-01 02:17:58.722537: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809da00 of size 256 next 139 2020-10-01 02:17:58.722560: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809db00 of size 6400 next 141 2020-10-01 02:17:58.722585: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809f400 of size 6144 next 142 2020-10-01 02:17:58.722608: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a0c00 of size 256 next 172 2020-10-01 02:17:58.722631: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a0d00 of size 256 next 173 2020-10-01 02:17:58.722654: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a0e00 of size 256 next 174 2020-10-01 02:17:58.722678: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a0f00 of size 256 next 175 2020-10-01 02:17:58.722701: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1000 of size 256 next 176 2020-10-01 02:17:58.722724: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1100 of size 256 next 177 2020-10-01 02:17:58.722747: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1200 of size 256 next 178 2020-10-01 02:17:58.722770: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1300 of size 256 next 179 2020-10-01 02:17:58.722793: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1400 of size 256 next 180 2020-10-01 02:17:58.722817: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1500 of size 1792 next 181 2020-10-01 02:17:58.722841: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1c00 of size 256 next 182 2020-10-01 02:17:58.722864: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1d00 of size 256 next 183 2020-10-01 02:17:58.722888: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1e00 of size 256 next 184 2020-10-01 02:17:58.722914: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1f00 of size 256 next 185 2020-10-01 02:17:58.722937: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a2000 of size 256 next 186 2020-10-01 02:17:58.722959: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a2100 of size 256 next 188 2020-10-01 02:17:58.722982: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a2200 of size 256 next 189 2020-10-01 02:17:58.723007: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a2300 of size 256 next 150 2020-10-01 02:17:58.723030: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a2400 of size 6144 next 149 2020-10-01 02:17:58.723054: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a3c00 of size 256 next 190 2020-10-01 02:17:58.723077: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a3d00 of size 256 next 191 2020-10-01 02:17:58.723100: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a3e00 of size 256 next 192 2020-10-01 02:17:58.723124: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a3f00 of size 256 next 194 2020-10-01 02:17:58.723147: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4000 of size 256 next 195 2020-10-01 02:17:58.723171: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4100 of size 256 next 196 2020-10-01 02:17:58.723194: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4200 of size 512 next 197 2020-10-01 02:17:58.723217: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4400 of size 512 next 198 2020-10-01 02:17:58.723240: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4600 of size 512 next 199 2020-10-01 02:17:58.723264: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4800 of size 512 next 200 2020-10-01 02:17:58.723287: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4a00 of size 512 next 201 2020-10-01 02:17:58.723310: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4c00 of size 512 next 202 2020-10-01 02:17:58.723333: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4e00 of size 512 next 204 2020-10-01 02:17:58.723357: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a5000 of size 512 next 205 2020-10-01 02:17:58.723380: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a5200 of size 512 next 147 2020-10-01 02:17:58.723404: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a5400 of size 18432 next 145 2020-10-01 02:17:58.723427: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a9c00 of size 18432 next 146 2020-10-01 02:17:58.723452: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680ae400 of size 31232 next 27 2020-10-01 02:17:58.723475: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680b5e00 of size 147456 next 28 2020-10-01 02:17:58.723498: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680d9e00 of size 512 next 207 2020-10-01 02:17:58.723522: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680da000 of size 512 next 208 2020-10-01 02:17:58.723545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680da200 of size 512 next 209 2020-10-01 02:17:58.723568: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680da400 of size 512 next 211 2020-10-01 02:17:58.723591: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680da600 of size 512 next 212 2020-10-01 02:17:58.723614: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680da800 of size 512 next 213 2020-10-01 02:17:58.723637: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680daa00 of size 512 next 215 2020-10-01 02:17:58.723660: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dac00 of size 512 next 216 2020-10-01 02:17:58.723683: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dae00 of size 512 next 217 2020-10-01 02:17:58.723706: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680db000 of size 512 next 219 2020-10-01 02:17:58.723729: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680db200 of size 512 next 220 2020-10-01 02:17:58.723752: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680db400 of size 512 next 221 2020-10-01 02:17:58.723776: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680db600 of size 512 next 223 2020-10-01 02:17:58.723799: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680db800 of size 512 next 224 2020-10-01 02:17:58.723822: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dba00 of size 512 next 225 2020-10-01 02:17:58.723845: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dbc00 of size 512 next 227 2020-10-01 02:17:58.723868: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dbe00 of size 512 next 228 2020-10-01 02:17:58.723907: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dc000 of size 512 next 229 2020-10-01 02:17:58.723930: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dc200 of size 512 next 231 2020-10-01 02:17:58.723953: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dc400 of size 512 next 232 2020-10-01 02:17:58.723976: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dc600 of size 512 next 233 2020-10-01 02:17:58.724000: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dc800 of size 512 next 235 2020-10-01 02:17:58.724023: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dca00 of size 512 next 236 2020-10-01 02:17:58.724046: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dcc00 of size 512 next 237 2020-10-01 02:17:58.724069: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dce00 of size 512 next 239 2020-10-01 02:17:58.724092: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dd000 of size 512 next 240 2020-10-01 02:17:58.724116: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dd200 of size 512 next 241 2020-10-01 02:17:58.724139: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dd400 of size 512 next 243 2020-10-01 02:17:58.724162: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dd600 of size 512 next 245 2020-10-01 02:17:58.724186: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dd800 of size 512 next 247 2020-10-01 02:17:58.724209: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dda00 of size 512 next 248 2020-10-01 02:17:58.724233: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680ddc00 of size 512 next 249 2020-10-01 02:17:58.724256: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dde00 of size 512 next 250 2020-10-01 02:17:58.724280: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de000 of size 512 next 251 2020-10-01 02:17:58.724302: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de200 of size 512 next 252 2020-10-01 02:17:58.724307: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de400 of size 512 next 253 2020-10-01 02:17:58.724311: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de600 of size 256 next 254 2020-10-01 02:17:58.724316: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de700 of size 256 next 256 2020-10-01 02:17:58.724320: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de800 of size 256 next 258 2020-10-01 02:17:58.724325: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de900 of size 256 next 260 2020-10-01 02:17:58.724329: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dea00 of size 256 next 262 2020-10-01 02:17:58.724334: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680deb00 of size 256 next 264 2020-10-01 02:17:58.724338: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dec00 of size 1792 next 265 2020-10-01 02:17:58.724343: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df300 of size 256 next 266 2020-10-01 02:17:58.724347: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df400 of size 256 next 267 2020-10-01 02:17:58.724352: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df500 of size 256 next 269 2020-10-01 02:17:58.724356: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df600 of size 256 next 270 2020-10-01 02:17:58.724361: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df700 of size 256 next 271 2020-10-01 02:17:58.724365: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df800 of size 256 next 273 2020-10-01 02:17:58.724370: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df900 of size 256 next 274 2020-10-01 02:17:58.724374: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dfa00 of size 256 next 275 2020-10-01 02:17:58.724379: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dfb00 of size 256 next 277 2020-10-01 02:17:58.724383: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dfc00 of size 256 next 278 2020-10-01 02:17:58.724388: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dfd00 of size 256 next 156 2020-10-01 02:17:58.724392: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dfe00 of size 24576 next 155 2020-10-01 02:17:58.724397: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680e5e00 of size 18432 next 255 2020-10-01 02:17:58.724401: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680ea600 of size 24576 next 257 2020-10-01 02:17:58.724406: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f0600 of size 6144 next 259 2020-10-01 02:17:58.724410: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f1e00 of size 6144 next 261 2020-10-01 02:17:58.724415: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f3600 of size 6144 next 263 2020-10-01 02:17:58.724419: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f4e00 of size 256 next 280 2020-10-01 02:17:58.724424: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f4f00 of size 256 next 281 2020-10-01 02:17:58.724428: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5000 of size 256 next 282 2020-10-01 02:17:58.724433: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5100 of size 512 next 284 2020-10-01 02:17:58.724438: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5300 of size 512 next 285 2020-10-01 02:17:58.724442: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5500 of size 512 next 286 2020-10-01 02:17:58.724446: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5700 of size 512 next 288 2020-10-01 02:17:58.724451: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5900 of size 512 next 289 2020-10-01 02:17:58.724455: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5b00 of size 512 next 290 2020-10-01 02:17:58.724460: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5d00 of size 512 next 292 2020-10-01 02:17:58.724464: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5f00 of size 512 next 293 2020-10-01 02:17:58.724469: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6100 of size 512 next 294 2020-10-01 02:17:58.724473: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6300 of size 512 next 296 2020-10-01 02:17:58.724478: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6500 of size 512 next 297 2020-10-01 02:17:58.724482: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6700 of size 512 next 298 2020-10-01 02:17:58.724487: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6900 of size 512 next 300 2020-10-01 02:17:58.724491: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6b00 of size 512 next 301 2020-10-01 02:17:58.724496: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6d00 of size 512 next 302 2020-10-01 02:17:58.724500: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6f00 of size 512 next 304 2020-10-01 02:17:58.724505: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7100 of size 512 next 305 2020-10-01 02:17:58.724509: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7300 of size 512 next 306 2020-10-01 02:17:58.724514: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7500 of size 512 next 308 2020-10-01 02:17:58.724518: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7700 of size 512 next 309 2020-10-01 02:17:58.724523: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7900 of size 512 next 310 2020-10-01 02:17:58.724527: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7b00 of size 512 next 312 2020-10-01 02:17:58.724531: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7d00 of size 512 next 313 2020-10-01 02:17:58.724536: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7f00 of size 512 next 314 2020-10-01 02:17:58.724541: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8100 of size 512 next 316 2020-10-01 02:17:58.724545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8300 of size 512 next 317 2020-10-01 02:17:58.724549: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8500 of size 512 next 318 2020-10-01 02:17:58.724554: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8700 of size 512 next 320 2020-10-01 02:17:58.724558: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8900 of size 512 next 321 2020-10-01 02:17:58.724563: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8b00 of size 512 next 322 2020-10-01 02:17:58.724567: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8d00 of size 512 next 324 2020-10-01 02:17:58.724572: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8f00 of size 512 next 325 2020-10-01 02:17:58.724576: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9100 of size 512 next 326 2020-10-01 02:17:58.724581: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9300 of size 512 next 328 2020-10-01 02:17:58.724585: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9500 of size 512 next 329 2020-10-01 02:17:58.724590: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9700 of size 512 next 330 2020-10-01 02:17:58.724594: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9900 of size 512 next 332 2020-10-01 02:17:58.724599: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9b00 of size 512 next 334 2020-10-01 02:17:58.724603: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9d00 of size 512 next 336 2020-10-01 02:17:58.724608: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9f00 of size 512 next 337 2020-10-01 02:17:58.724612: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fa100 of size 512 next 338 2020-10-01 02:17:58.724617: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fa300 of size 512 next 339 2020-10-01 02:17:58.724621: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fa500 of size 512 next 340 2020-10-01 02:17:58.724626: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fa700 of size 512 next 341 2020-10-01 02:17:58.724630: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fa900 of size 512 next 342 2020-10-01 02:17:58.724635: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fab00 of size 256 next 344 2020-10-01 02:17:58.724639: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fac00 of size 256 next 346 2020-10-01 02:17:58.724644: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fad00 of size 256 next 348 2020-10-01 02:17:58.724648: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fae00 of size 6144 next 349 2020-10-01 02:17:58.724653: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fc600 of size 256 next 350 2020-10-01 02:17:58.724657: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fc700 of size 256 next 352 2020-10-01 02:17:58.724662: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fc800 of size 256 next 354 2020-10-01 02:17:58.724666: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fc900 of size 256 next 355 2020-10-01 02:17:58.724671: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fca00 of size 256 next 356 2020-10-01 02:17:58.724675: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fcb00 of size 256 next 357 2020-10-01 02:17:58.724680: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fcc00 of size 256 next 358 2020-10-01 02:17:58.724684: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fcd00 of size 256 next 360 2020-10-01 02:17:58.724689: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fce00 of size 256 next 361 2020-10-01 02:17:58.724693: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fcf00 of size 256 next 362 2020-10-01 02:17:58.724698: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd000 of size 256 next 363 2020-10-01 02:17:58.724702: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd100 of size 256 next 364 2020-10-01 02:17:58.724707: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd200 of size 256 next 365 2020-10-01 02:17:58.724711: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd300 of size 256 next 366 2020-10-01 02:17:58.724716: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd400 of size 256 next 367 2020-10-01 02:17:58.724720: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd500 of size 256 next 368 2020-10-01 02:17:58.724725: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd600 of size 256 next 369 2020-10-01 02:17:58.724729: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd700 of size 256 next 370 2020-10-01 02:17:58.724734: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd800 of size 256 next 371 2020-10-01 02:17:58.724738: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd900 of size 256 next 372 2020-10-01 02:17:58.724742: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fda00 of size 256 next 373 2020-10-01 02:17:58.724747: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fdb00 of size 256 next 375 2020-10-01 02:17:58.724752: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fdc00 of size 256 next 376 2020-10-01 02:17:58.724756: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fdd00 of size 256 next 34 2020-10-01 02:17:58.724761: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fde00 of size 147456 next 33 2020-10-01 02:17:58.724765: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268121e00 of size 147456 next 193 2020-10-01 02:17:58.724770: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268145e00 of size 442368 next 44 2020-10-01 02:17:58.724775: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12681b1e00 of size 294912 next 43 2020-10-01 02:17:58.724779: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12681f9e00 of size 294912 next 119 2020-10-01 02:17:58.724784: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268241e00 of size 147456 next 187 2020-10-01 02:17:58.724788: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268265e00 of size 147456 next 52 2020-10-01 02:17:58.724793: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268289e00 of size 589824 next 51 2020-10-01 02:17:58.724797: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268319e00 of size 589824 next 58 2020-10-01 02:17:58.724802: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12683a9e00 of size 589824 next 64 2020-10-01 02:17:58.724806: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268439e00 of size 589824 next 70 2020-10-01 02:17:58.724811: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12684c9e00 of size 589824 next 76 2020-10-01 02:17:58.724815: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268559e00 of size 589824 next 82 2020-10-01 02:17:58.724820: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12685e9e00 of size 589824 next 88 2020-10-01 02:17:58.724824: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268679e00 of size 589824 next 94 2020-10-01 02:17:58.724829: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268709e00 of size 589824 next 100 2020-10-01 02:17:58.724833: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268799e00 of size 589824 next 106 2020-10-01 02:17:58.724838: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268829e00 of size 589824 next 112 2020-10-01 02:17:58.724842: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12688b9e00 of size 589824 next 125 2020-10-01 02:17:58.724847: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268949e00 of size 589824 next 124 2020-10-01 02:17:58.724851: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12689d9e00 of size 589824 next 131 2020-10-01 02:17:58.724856: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268a69e00 of size 589824 next 203 2020-10-01 02:17:58.724860: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268af9e00 of size 589824 next 206 2020-10-01 02:17:58.724865: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268b89e00 of size 589824 next 210 2020-10-01 02:17:58.724869: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268c19e00 of size 589824 next 214 2020-10-01 02:17:58.724874: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268ca9e00 of size 589824 next 218 2020-10-01 02:17:58.724878: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268d39e00 of size 589824 next 222 2020-10-01 02:17:58.724883: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268dc9e00 of size 589824 next 226 2020-10-01 02:17:58.724887: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268e59e00 of size 589824 next 230 2020-10-01 02:17:58.724892: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268ee9e00 of size 589824 next 234 2020-10-01 02:17:58.724896: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268f79e00 of size 589824 next 238 2020-10-01 02:17:58.724901: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269009e00 of size 294912 next 242 2020-10-01 02:17:58.724905: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269051e00 of size 589824 next 244 2020-10-01 02:17:58.724910: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12690e1e00 of size 589824 next 246 2020-10-01 02:17:58.724914: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269171e00 of size 147456 next 268 2020-10-01 02:17:58.724919: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269195e00 of size 147456 next 272 2020-10-01 02:17:58.724923: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12691b9e00 of size 147456 next 276 2020-10-01 02:17:58.724928: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12691dde00 of size 147456 next 279 2020-10-01 02:17:58.724933: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269201e00 of size 294912 next 283 2020-10-01 02:17:58.724937: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269249e00 of size 589824 next 287 2020-10-01 02:17:58.724942: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12692d9e00 of size 589824 next 291 2020-10-01 02:17:58.724946: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269369e00 of size 589824 next 295 2020-10-01 02:17:58.724950: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12693f9e00 of size 589824 next 299 2020-10-01 02:17:58.724955: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269489e00 of size 589824 next 303 2020-10-01 02:17:58.724959: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269519e00 of size 589824 next 307 2020-10-01 02:17:58.724964: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12695a9e00 of size 589824 next 311 2020-10-01 02:17:58.724968: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269639e00 of size 589824 next 315 2020-10-01 02:17:58.724973: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12696c9e00 of size 589824 next 319 2020-10-01 02:17:58.724977: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269759e00 of size 589824 next 323 2020-10-01 02:17:58.724982: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12697e9e00 of size 589824 next 327 2020-10-01 02:17:58.724987: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269879e00 of size 294912 next 331 2020-10-01 02:17:58.724991: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12698c1e00 of size 589824 next 333 2020-10-01 02:17:58.724996: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269951e00 of size 589824 next 335 2020-10-01 02:17:58.725000: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699e1e00 of size 18432 next 343 2020-10-01 02:17:58.725005: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699e6600 of size 18432 next 345 2020-10-01 02:17:58.725009: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699eae00 of size 24576 next 347 2020-10-01 02:17:58.725014: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699f0e00 of size 6144 next 351 2020-10-01 02:17:58.725018: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699f2600 of size 6144 next 353 2020-10-01 02:17:58.725023: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699f3e00 of size 12192768 next 359 2020-10-01 02:17:58.725028: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126a594a00 of size 4064256 next 374 2020-10-01 02:17:58.725032: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126a974e00 of size 256 next 377 2020-10-01 02:17:58.725037: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126a974f00 of size 256 next 378 2020-10-01 02:17:58.725041: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126a975000 of size 256 next 379 2020-10-01 02:17:58.725046: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126a975100 of size 256 next 380 2020-10-01 02:17:58.725050: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f126a975200 of size 4064256 next 381 2020-10-01 02:17:58.725055: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126ad55600 of size 12288000 next 11 2020-10-01 02:17:58.725060: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126b90d600 of size 122112000 next 382 2020-10-01 02:17:58.725064: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f1272d81e00 of size 3048192 next 501 2020-10-01 02:17:58.725069: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306a100 of size 256 next 517 2020-10-01 02:17:58.725073: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f127306a200 of size 2304 next 510 2020-10-01 02:17:58.725078: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306ab00 of size 256 next 504 2020-10-01 02:17:58.725082: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306ac00 of size 256 next 505 2020-10-01 02:17:58.725087: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f127306ad00 of size 2816 next 519 2020-10-01 02:17:58.725091: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306b800 of size 256 next 520 2020-10-01 02:17:58.725096: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306b900 of size 256 next 521 2020-10-01 02:17:58.725100: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306ba00 of size 256 next 515 2020-10-01 02:17:58.725104: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306bb00 of size 256 next 522 2020-10-01 02:17:58.725109: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306bc00 of size 256 next 523 2020-10-01 02:17:58.725113: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306bd00 of size 18432 next 484 2020-10-01 02:17:58.725118: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f1273070500 of size 30720 next 400 2020-10-01 02:17:58.725122: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1273077d00 of size 18432 next 494 2020-10-01 02:17:58.725127: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127307c500 of size 6144 next 527 2020-10-01 02:17:58.725131: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f127307dd00 of size 49152 next 512 2020-10-01 02:17:58.725136: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1273089d00 of size 6144 next 499 2020-10-01 02:17:58.725140: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f127308b500 of size 6144 next 503 2020-10-01 02:17:58.725145: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127308cd00 of size 6144 next 526 2020-10-01 02:17:58.725149: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f127308e500 of size 17124608 next 385 2020-10-01 02:17:58.725154: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12740e3200 of size 256 next 386 2020-10-01 02:17:58.725158: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f12740e3300 of size 36578304 next 390 2020-10-01 02:17:58.725163: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12763c5700 of size 576000 next 391 2020-10-01 02:17:58.725167: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f1276452100 of size 2816 next 392 2020-10-01 02:17:58.725172: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276452c00 of size 256 next 404 2020-10-01 02:17:58.725176: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276452d00 of size 256 next 405 2020-10-01 02:17:58.725181: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276452e00 of size 256 next 406 2020-10-01 02:17:58.725185: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276452f00 of size 256 next 411 2020-10-01 02:17:58.725190: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453000 of size 256 next 412 2020-10-01 02:17:58.725194: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453100 of size 256 next 415 2020-10-01 02:17:58.725199: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453200 of size 256 next 416 2020-10-01 02:17:58.725203: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453300 of size 256 next 418 2020-10-01 02:17:58.725208: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453400 of size 256 next 419 2020-10-01 02:17:58.725212: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453500 of size 256 next 422 2020-10-01 02:17:58.725217: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453600 of size 256 next 423 2020-10-01 02:17:58.725221: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453700 of size 512 next 427 2020-10-01 02:17:58.725226: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453900 of size 512 next 428 2020-10-01 02:17:58.725231: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453b00 of size 512 next 429 2020-10-01 02:17:58.725235: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453d00 of size 512 next 432 2020-10-01 02:17:58.725240: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453f00 of size 512 next 433 2020-10-01 02:17:58.725244: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454100 of size 512 next 434 2020-10-01 02:17:58.725248: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454300 of size 512 next 435 2020-10-01 02:17:58.725253: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454500 of size 512 next 436 2020-10-01 02:17:58.725257: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454700 of size 512 next 437 2020-10-01 02:17:58.725262: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454900 of size 512 next 442 2020-10-01 02:17:58.725266: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454b00 of size 512 next 443 2020-10-01 02:17:58.725271: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454d00 of size 512 next 446 2020-10-01 02:17:58.725275: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454f00 of size 512 next 447 2020-10-01 02:17:58.725280: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455100 of size 512 next 450 2020-10-01 02:17:58.725284: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455300 of size 512 next 451 2020-10-01 02:17:58.725289: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455500 of size 512 next 456 2020-10-01 02:17:58.725293: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455700 of size 512 next 457 2020-10-01 02:17:58.725298: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455900 of size 512 next 460 2020-10-01 02:17:58.725302: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455b00 of size 512 next 461 2020-10-01 02:17:58.725307: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455d00 of size 512 next 462 2020-10-01 02:17:58.725311: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455f00 of size 512 next 463 2020-10-01 02:17:58.725316: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456100 of size 512 next 464 2020-10-01 02:17:58.725320: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456300 of size 512 next 465 2020-10-01 02:17:58.725325: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456500 of size 512 next 470 2020-10-01 02:17:58.725329: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456700 of size 512 next 471 2020-10-01 02:17:58.725334: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456900 of size 512 next 474 2020-10-01 02:17:58.725338: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456b00 of size 512 next 475 2020-10-01 02:17:58.725343: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456d00 of size 512 next 478 2020-10-01 02:17:58.725347: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456f00 of size 512 next 479 2020-10-01 02:17:58.725352: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276457100 of size 512 next 482 2020-10-01 02:17:58.725356: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276457300 of size 512 next 483 2020-10-01 02:17:58.725360: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f1276457500 of size 11155200 next 396 2020-10-01 02:17:58.725365: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276efac00 of size 256 next 397 2020-10-01 02:17:58.725369: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276efad00 of size 8128000 next 399 2020-10-01 02:17:58.725374: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12776bb300 of size 134400000 next 401 2020-10-01 02:17:58.725379: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127f6e7b00 of size 1228800000 next 402 2020-10-01 02:17:58.725384: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12c8ac7b00 of size 1228800000 next 403 2020-10-01 02:17:58.725388: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1311ea7b00 of size 65028096 next 409 2020-10-01 02:17:58.725393: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1315cabb00 of size 65028096 next 413 2020-10-01 02:17:58.725397: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1319aafb00 of size 65028096 next 19 2020-10-01 02:17:58.725402: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f131d8b3b00 of size 65028096 next 408 2020-10-01 02:17:58.725406: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13216b7b00 of size 260112384 next 407 2020-10-01 02:17:58.725411: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1330ec7b00 of size 65028096 next 24 2020-10-01 02:17:58.725415: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1334ccbb00 of size 65028096 next 31 2020-10-01 02:17:58.725420: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1338acfb00 of size 65028096 next 420 2020-10-01 02:17:58.725424: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f133c8d3b00 of size 32514048 next 425 2020-10-01 02:17:58.725429: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f133e7d5b00 of size 32514048 next 38 2020-10-01 02:17:58.725434: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13406d7b00 of size 32514048 next 47 2020-10-01 02:17:58.725438: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13425d9b00 of size 32514048 next 424 2020-10-01 02:17:58.725443: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13444dbb00 of size 130056192 next 426 2020-10-01 02:17:58.725447: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f134c0e3b00 of size 130056192 next 430 2020-10-01 02:17:58.725452: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1353cebb00 of size 32514048 next 122 2020-10-01 02:17:58.725456: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1355bedb00 of size 32514048 next 56 2020-10-01 02:17:58.725461: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1357aefb00 of size 32514048 next 62 2020-10-01 02:17:58.725465: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13599f1b00 of size 32514048 next 440 2020-10-01 02:17:58.725470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f135b8f3b00 of size 32514048 next 444 2020-10-01 02:17:58.725475: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f135d7f5b00 of size 32514048 next 68 2020-10-01 02:17:58.725479: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f135f6f7b00 of size 32514048 next 448 2020-10-01 02:17:58.725484: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13615f9b00 of size 32514048 next 74 2020-10-01 02:17:58.725488: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13634fbb00 of size 8128512 next 80 2020-10-01 02:17:58.725493: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1363cbc300 of size 8128512 next 453 2020-10-01 02:17:58.725497: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f136447cb00 of size 8128512 next 458 2020-10-01 02:17:58.725501: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1364c3d300 of size 8128512 next 86 2020-10-01 02:17:58.725506: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f13653fdb00 of size 236761088 next 454 2020-10-01 02:17:58.725509: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13735c8b00 of size 130056192 next 455 2020-10-01 02:17:58.725514: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137b1d0b00 of size 8128512 next 92 2020-10-01 02:17:58.725519: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137b991300 of size 8128512 next 129 2020-10-01 02:17:58.725522: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137c151b00 of size 8128512 next 98 2020-10-01 02:17:58.725527: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137c912300 of size 8128512 next 468 2020-10-01 02:17:58.725531: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137d0d2b00 of size 8128512 next 104 2020-10-01 02:17:58.725536: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137d893300 of size 8128512 next 472 2020-10-01 02:17:58.725540: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137e053b00 of size 8128512 next 476 2020-10-01 02:17:58.725545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137e814300 of size 8128512 next 110 2020-10-01 02:17:58.725549: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137efd4b00 of size 4064256 next 516 2020-10-01 02:17:58.725554: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137f3b4f00 of size 12192768 next 490 2020-10-01 02:17:58.725558: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f137ff55b00 of size 113799168 next 116 2020-10-01 02:17:58.725563: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1386bdcb00 of size 130056192 next 481 2020-10-01 02:17:58.725568: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f138e7e4b00 of size 390168576 next 135 2020-10-01 02:17:58.725572: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13a5bfcb00 of size 390168576 next 389 2020-10-01 02:17:58.725577: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13bd014b00 of size 390168576 next 395 2020-10-01 02:17:58.725582: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13d442cb00 of size 406945792 next 383 2020-10-01 02:17:58.725586: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13ec844b00 of size 390168576 next 384 2020-10-01 02:17:58.725590: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f1403c5cb00 of size 253356032 next 18446744073709551615 2020-10-01 02:17:58.725595: I tensorflow/core/common_runtime/bfc_allocator.cc:1031] Summary of in-use Chunks by size: 2020-10-01 02:17:58.725601: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 155 Chunks of size 256 totalling 38.8KiB 2020-10-01 02:17:58.725606: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 195 Chunks of size 512 totalling 97.5KiB 2020-10-01 02:17:58.725611: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1280 totalling 1.2KiB 2020-10-01 02:17:58.725615: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 1792 totalling 5.2KiB 2020-10-01 02:17:58.725620: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 11 Chunks of size 6144 totalling 66.0KiB 2020-10-01 02:17:58.725625: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 6400 totalling 6.2KiB 2020-10-01 02:17:58.725630: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 7 Chunks of size 18432 totalling 126.0KiB 2020-10-01 02:17:58.725635: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 24576 totalling 72.0KiB 2020-10-01 02:17:58.725640: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 31232 totalling 30.5KiB 2020-10-01 02:17:58.725644: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 11 Chunks of size 147456 totalling 1.55MiB 2020-10-01 02:17:58.725649: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 294144 totalling 287.2KiB 2020-10-01 02:17:58.725654: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 5 Chunks of size 294912 totalling 1.41MiB 2020-10-01 02:17:58.725659: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 442368 totalling 432.0KiB 2020-10-01 02:17:58.725664: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 576000 totalling 562.5KiB 2020-10-01 02:17:58.725669: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 39 Chunks of size 589824 totalling 21.94MiB 2020-10-01 02:17:58.725674: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 4064256 totalling 7.75MiB 2020-10-01 02:17:58.725678: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 8128000 totalling 7.75MiB 2020-10-01 02:17:58.725683: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 12 Chunks of size 8128512 totalling 93.02MiB 2020-10-01 02:17:58.725689: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 12192768 totalling 23.26MiB 2020-10-01 02:17:58.725694: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12288000 totalling 11.72MiB 2020-10-01 02:17:58.725699: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 12 Chunks of size 32514048 totalling 372.09MiB 2020-10-01 02:17:58.725704: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 7 Chunks of size 65028096 totalling 434.11MiB 2020-10-01 02:17:58.725709: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 122112000 totalling 116.46MiB 2020-10-01 02:17:58.725714: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 130056192 totalling 372.09MiB 2020-10-01 02:17:58.725719: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 134400000 totalling 128.17MiB 2020-10-01 02:17:58.725724: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 260112384 totalling 248.06MiB 2020-10-01 02:17:58.725728: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 4 Chunks of size 390168576 totalling 1.45GiB 2020-10-01 02:17:58.725733: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 406945792 totalling 388.09MiB 2020-10-01 02:17:58.725738: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 1228800000 totalling 2.29GiB 2020-10-01 02:17:58.725743: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 5.92GiB 2020-10-01 02:17:58.725748: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocatedbytes: 7161754368 memorylimit: 7161754432 available bytes: 64 curr_region_allocationbytes: 14323509248 2020-10-01 02:17:58.725755: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats: Limit: 7161754432 InUse: 6355716864 MaxInUse: 6658663936 NumAllocs: 1626 MaxAllocSize: 2317877248 Reserved: 0 PeakReserved: 0 LargestFreeBlock: 0

2020-10-01 02:17:58.725779: W tensorflow/core/common_runtime/bfcallocator.cc:439] ***___**___ 2020-10-01 02:17:58.725795: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_grad_input_ops.cc:1103 : Resource exhausted: OOM when allocating tensor with shape[4,384,252,252] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc 2020-10-01 02:17:58.755897: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_0_bfc) ran out of memory trying to allocate 372.09MiB (rounded to 390168576)requested by op gradient_tape/functional_1/loc/conv2d/Conv2D/Conv2DBackpropInput Current allocation summary follows. 2020-10-01 02:17:58.755933: I tensorflow/core/common_runtime/bfc_allocator.cc:970] BFCAllocator dump for GPU_0_bfc 2020-10-01 02:17:58.755946: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (256): Total Chunks: 154, Chunks in use: 154. 38.5KiB allocated for chunks. 38.5KiB in use in bin. 16.5KiB client-requested in use in bin. 2020-10-01 02:17:58.755958: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (512): Total Chunks: 196, Chunks in use: 195. 98.0KiB allocated for chunks. 97.5KiB in use in bin. 97.5KiB client-requested in use in bin. 2020-10-01 02:17:58.755969: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1024): Total Chunks: 4, Chunks in use: 4. 6.5KiB allocated for chunks. 6.5KiB in use in bin. 6.3KiB client-requested in use in bin. 2020-10-01 02:17:58.755978: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2048): Total Chunks: 3, Chunks in use: 0. 8.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-10-01 02:17:58.755987: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4096): Total Chunks: 11, Chunks in use: 11. 66.2KiB allocated for chunks. 66.2KiB in use in bin. 66.0KiB client-requested in use in bin. 2020-10-01 02:17:58.755996: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-10-01 02:17:58.756010: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16384): Total Chunks: 12, Chunks in use: 11. 258.5KiB allocated for chunks. 228.5KiB in use in bin. 216.0KiB client-requested in use in bin. 2020-10-01 02:17:58.756021: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (32768): Total Chunks: 1, Chunks in use: 0. 54.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-10-01 02:17:58.756029: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-10-01 02:17:58.756039: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (131072): Total Chunks: 11, Chunks in use: 11. 1.55MiB allocated for chunks. 1.55MiB in use in bin. 1.55MiB client-requested in use in bin. 2020-10-01 02:17:58.756050: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (262144): Total Chunks: 7, Chunks in use: 7. 2.11MiB allocated for chunks. 2.11MiB in use in bin. 1.83MiB client-requested in use in bin. 2020-10-01 02:17:58.756060: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (524288): Total Chunks: 40, Chunks in use: 40. 22.49MiB allocated for chunks. 22.49MiB in use in bin. 22.49MiB client-requested in use in bin. 2020-10-01 02:17:58.756070: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-10-01 02:17:58.756081: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2097152): Total Chunks: 4, Chunks in use: 1. 14.53MiB allocated for chunks. 3.88MiB in use in bin. 3.88MiB client-requested in use in bin. 2020-10-01 02:17:58.756092: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4194304): Total Chunks: 13, Chunks in use: 13. 100.77MiB allocated for chunks. 100.77MiB in use in bin. 96.90MiB client-requested in use in bin. 2020-10-01 02:17:58.756102: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8388608): Total Chunks: 4, Chunks in use: 3. 45.61MiB allocated for chunks. 34.97MiB in use in bin. 34.97MiB client-requested in use in bin. 2020-10-01 02:17:58.756112: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16777216): Total Chunks: 13, Chunks in use: 12. 388.43MiB allocated for chunks. 372.09MiB in use in bin. 372.09MiB client-requested in use in bin. 2020-10-01 02:17:58.756122: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (33554432): Total Chunks: 8, Chunks in use: 7. 468.99MiB allocated for chunks. 434.11MiB in use in bin. 434.11MiB client-requested in use in bin. 2020-10-01 02:17:58.756134: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (67108864): Total Chunks: 6, Chunks in use: 4. 721.11MiB allocated for chunks. 488.55MiB in use in bin. 434.11MiB client-requested in use in bin. 2020-10-01 02:17:58.756144: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (134217728): Total Chunks: 4, Chunks in use: 2. 843.65MiB allocated for chunks. 376.24MiB in use in bin. 376.24MiB client-requested in use in bin. 2020-10-01 02:17:58.756153: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (268435456): Total Chunks: 7, Chunks in use: 7. 4.12GiB allocated for chunks. 4.12GiB in use in bin. 4.11GiB client-requested in use in bin. 2020-10-01 02:17:58.756162: I tensorflow/core/common_runtime/bfc_allocator.cc:993] Bin for 372.09MiB was 256.00MiB, Chunk State: 2020-10-01 02:17:58.756169: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 7161754368 2020-10-01 02:17:58.756179: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000000 of size 1280 next 1 2020-10-01 02:17:58.756187: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000500 of size 256 next 5 2020-10-01 02:17:58.756195: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000600 of size 256 next 8 2020-10-01 02:17:58.756202: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000700 of size 256 next 9 2020-10-01 02:17:58.756210: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000800 of size 256 next 10 2020-10-01 02:17:58.756218: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000900 of size 256 next 410 2020-10-01 02:17:58.756225: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000a00 of size 256 next 12 2020-10-01 02:17:58.756233: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000b00 of size 256 next 2 2020-10-01 02:17:58.756241: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000c00 of size 256 next 3 2020-10-01 02:17:58.756249: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000d00 of size 256 next 4 2020-10-01 02:17:58.756256: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000e00 of size 256 next 14 2020-10-01 02:17:58.756264: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268000f00 of size 256 next 13 2020-10-01 02:17:58.756272: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001000 of size 256 next 17 2020-10-01 02:17:58.756280: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001100 of size 256 next 18 2020-10-01 02:17:58.756288: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001200 of size 256 next 414 2020-10-01 02:17:58.756295: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001300 of size 256 next 20 2020-10-01 02:17:58.756303: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001400 of size 256 next 21 2020-10-01 02:17:58.756311: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001500 of size 256 next 6 2020-10-01 02:17:58.756319: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001600 of size 1792 next 7 2020-10-01 02:17:58.756327: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001d00 of size 256 next 23 2020-10-01 02:17:58.756335: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001e00 of size 256 next 417 2020-10-01 02:17:58.756343: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268001f00 of size 256 next 25 2020-10-01 02:17:58.756351: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268002000 of size 294144 next 16 2020-10-01 02:17:58.756359: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268049d00 of size 147456 next 15 2020-10-01 02:17:58.756367: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126806dd00 of size 147456 next 22 2020-10-01 02:17:58.756375: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268091d00 of size 256 next 26 2020-10-01 02:17:58.756384: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268091e00 of size 256 next 29 2020-10-01 02:17:58.756392: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268091f00 of size 256 next 30 2020-10-01 02:17:58.756400: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092000 of size 256 next 421 2020-10-01 02:17:58.756408: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092100 of size 256 next 32 2020-10-01 02:17:58.756416: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092200 of size 256 next 35 2020-10-01 02:17:58.756424: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092300 of size 256 next 36 2020-10-01 02:17:58.756431: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092400 of size 256 next 37 2020-10-01 02:17:58.756445: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092500 of size 512 next 39 2020-10-01 02:17:58.756453: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092700 of size 256 next 41 2020-10-01 02:17:58.756460: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092800 of size 256 next 42 2020-10-01 02:17:58.756466: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092900 of size 512 next 40 2020-10-01 02:17:58.756473: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092b00 of size 512 next 45 2020-10-01 02:17:58.756479: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092d00 of size 512 next 46 2020-10-01 02:17:58.756487: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268092f00 of size 512 next 431 2020-10-01 02:17:58.756495: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093100 of size 512 next 48 2020-10-01 02:17:58.756503: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093300 of size 256 next 49 2020-10-01 02:17:58.756511: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093400 of size 256 next 50 2020-10-01 02:17:58.756518: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093500 of size 512 next 53 2020-10-01 02:17:58.756526: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093700 of size 512 next 54 2020-10-01 02:17:58.756534: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093900 of size 512 next 55 2020-10-01 02:17:58.756542: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093b00 of size 512 next 438 2020-10-01 02:17:58.756550: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093d00 of size 512 next 57 2020-10-01 02:17:58.756559: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268093f00 of size 512 next 59 2020-10-01 02:17:58.756567: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094100 of size 512 next 60 2020-10-01 02:17:58.756575: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094300 of size 512 next 61 2020-10-01 02:17:58.756582: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094500 of size 512 next 441 2020-10-01 02:17:58.756590: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094700 of size 512 next 63 2020-10-01 02:17:58.756598: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094900 of size 512 next 65 2020-10-01 02:17:58.756606: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094b00 of size 512 next 66 2020-10-01 02:17:58.756614: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094d00 of size 512 next 67 2020-10-01 02:17:58.756622: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268094f00 of size 512 next 445 2020-10-01 02:17:58.756630: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095100 of size 512 next 69 2020-10-01 02:17:58.756638: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095300 of size 512 next 71 2020-10-01 02:17:58.756645: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095500 of size 512 next 72 2020-10-01 02:17:58.756653: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095700 of size 512 next 73 2020-10-01 02:17:58.756661: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095900 of size 512 next 449 2020-10-01 02:17:58.756669: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095b00 of size 512 next 75 2020-10-01 02:17:58.756677: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095d00 of size 512 next 77 2020-10-01 02:17:58.756685: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268095f00 of size 512 next 78 2020-10-01 02:17:58.756693: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096100 of size 512 next 79 2020-10-01 02:17:58.756701: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096300 of size 512 next 452 2020-10-01 02:17:58.756709: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096500 of size 512 next 81 2020-10-01 02:17:58.756717: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096700 of size 512 next 83 2020-10-01 02:17:58.756725: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096900 of size 512 next 84 2020-10-01 02:17:58.756733: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096b00 of size 512 next 85 2020-10-01 02:17:58.756740: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096d00 of size 512 next 459 2020-10-01 02:17:58.756748: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268096f00 of size 512 next 87 2020-10-01 02:17:58.756756: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097100 of size 512 next 89 2020-10-01 02:17:58.756764: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097300 of size 512 next 90 2020-10-01 02:17:58.756772: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097500 of size 512 next 91 2020-10-01 02:17:58.756780: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097700 of size 512 next 466 2020-10-01 02:17:58.756788: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097900 of size 512 next 93 2020-10-01 02:17:58.756795: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097b00 of size 512 next 95 2020-10-01 02:17:58.756803: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097d00 of size 512 next 96 2020-10-01 02:17:58.756811: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268097f00 of size 512 next 97 2020-10-01 02:17:58.756819: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098100 of size 512 next 469 2020-10-01 02:17:58.756827: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098300 of size 512 next 99 2020-10-01 02:17:58.756835: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098500 of size 512 next 101 2020-10-01 02:17:58.756842: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098700 of size 512 next 102 2020-10-01 02:17:58.756850: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098900 of size 512 next 103 2020-10-01 02:17:58.756858: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098b00 of size 512 next 473 2020-10-01 02:17:58.756866: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098d00 of size 512 next 105 2020-10-01 02:17:58.756874: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268098f00 of size 512 next 107 2020-10-01 02:17:58.756882: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099100 of size 512 next 108 2020-10-01 02:17:58.756889: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099300 of size 512 next 109 2020-10-01 02:17:58.756897: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099500 of size 512 next 477 2020-10-01 02:17:58.756905: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099700 of size 512 next 111 2020-10-01 02:17:58.756913: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099900 of size 512 next 113 2020-10-01 02:17:58.756921: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099b00 of size 512 next 114 2020-10-01 02:17:58.756929: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099d00 of size 512 next 115 2020-10-01 02:17:58.756937: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268099f00 of size 512 next 480 2020-10-01 02:17:58.756944: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809a100 of size 512 next 117 2020-10-01 02:17:58.757021: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809a300 of size 512 next 118 2020-10-01 02:17:58.757032: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809a500 of size 512 next 120 2020-10-01 02:17:58.757040: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809a700 of size 512 next 121 2020-10-01 02:17:58.757048: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809a900 of size 512 next 439 2020-10-01 02:17:58.757056: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809ab00 of size 512 next 123 2020-10-01 02:17:58.757064: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809ad00 of size 512 next 126 2020-10-01 02:17:58.757072: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809af00 of size 512 next 127 2020-10-01 02:17:58.757080: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809b100 of size 512 next 128 2020-10-01 02:17:58.757088: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809b300 of size 512 next 467 2020-10-01 02:17:58.757096: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809b500 of size 512 next 130 2020-10-01 02:17:58.757104: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809b700 of size 512 next 132 2020-10-01 02:17:58.757111: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809b900 of size 512 next 133 2020-10-01 02:17:58.757119: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809bb00 of size 512 next 134 2020-10-01 02:17:58.757127: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809bd00 of size 256 next 513 2020-10-01 02:17:58.757135: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809be00 of size 256 next 509 2020-10-01 02:17:58.757143: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f126809bf00 of size 512 next 136 2020-10-01 02:17:58.757151: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c100 of size 256 next 140 2020-10-01 02:17:58.757159: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c200 of size 256 next 144 2020-10-01 02:17:58.757166: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c300 of size 256 next 143 2020-10-01 02:17:58.757174: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c400 of size 256 next 148 2020-10-01 02:17:58.757182: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c500 of size 256 next 151 2020-10-01 02:17:58.757190: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c600 of size 256 next 152 2020-10-01 02:17:58.757198: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c700 of size 256 next 153 2020-10-01 02:17:58.757205: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c800 of size 256 next 154 2020-10-01 02:17:58.757213: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809c900 of size 256 next 157 2020-10-01 02:17:58.757221: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809ca00 of size 256 next 158 2020-10-01 02:17:58.757229: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809cb00 of size 256 next 159 2020-10-01 02:17:58.757237: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809cc00 of size 256 next 160 2020-10-01 02:17:58.757245: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809cd00 of size 256 next 161 2020-10-01 02:17:58.757252: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809ce00 of size 256 next 162 2020-10-01 02:17:58.757260: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809cf00 of size 256 next 163 2020-10-01 02:17:58.757268: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d000 of size 256 next 164 2020-10-01 02:17:58.757276: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d100 of size 256 next 165 2020-10-01 02:17:58.757284: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d200 of size 256 next 166 2020-10-01 02:17:58.757291: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d300 of size 256 next 167 2020-10-01 02:17:58.757299: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d400 of size 256 next 168 2020-10-01 02:17:58.757307: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d500 of size 256 next 169 2020-10-01 02:17:58.757315: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d600 of size 256 next 170 2020-10-01 02:17:58.757323: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d700 of size 256 next 171 2020-10-01 02:17:58.757330: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d800 of size 256 next 137 2020-10-01 02:17:58.757338: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809d900 of size 256 next 138 2020-10-01 02:17:58.757346: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809da00 of size 256 next 139 2020-10-01 02:17:58.757354: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809db00 of size 6400 next 141 2020-10-01 02:17:58.757363: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126809f400 of size 6144 next 142 2020-10-01 02:17:58.757370: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a0c00 of size 256 next 172 2020-10-01 02:17:58.757378: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a0d00 of size 256 next 173 2020-10-01 02:17:58.757386: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a0e00 of size 256 next 174 2020-10-01 02:17:58.757394: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a0f00 of size 256 next 175 2020-10-01 02:17:58.757402: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1000 of size 256 next 176 2020-10-01 02:17:58.757410: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1100 of size 256 next 177 2020-10-01 02:17:58.757418: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1200 of size 256 next 178 2020-10-01 02:17:58.757426: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1300 of size 256 next 179 2020-10-01 02:17:58.757434: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1400 of size 256 next 180 2020-10-01 02:17:58.757442: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1500 of size 1792 next 181 2020-10-01 02:17:58.757450: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1c00 of size 256 next 182 2020-10-01 02:17:58.757458: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1d00 of size 256 next 183 2020-10-01 02:17:58.757466: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1e00 of size 256 next 184 2020-10-01 02:17:58.757473: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a1f00 of size 256 next 185 2020-10-01 02:17:58.757481: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a2000 of size 256 next 186 2020-10-01 02:17:58.757489: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a2100 of size 256 next 188 2020-10-01 02:17:58.757497: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a2200 of size 256 next 189 2020-10-01 02:17:58.757505: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a2300 of size 256 next 150 2020-10-01 02:17:58.757513: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a2400 of size 6144 next 149 2020-10-01 02:17:58.757521: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a3c00 of size 256 next 190 2020-10-01 02:17:58.757529: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a3d00 of size 256 next 191 2020-10-01 02:17:58.757536: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a3e00 of size 256 next 192 2020-10-01 02:17:58.757544: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a3f00 of size 256 next 194 2020-10-01 02:17:58.757552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4000 of size 256 next 195 2020-10-01 02:17:58.757560: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4100 of size 256 next 196 2020-10-01 02:17:58.757568: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4200 of size 512 next 197 2020-10-01 02:17:58.757576: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4400 of size 512 next 198 2020-10-01 02:17:58.757584: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4600 of size 512 next 199 2020-10-01 02:17:58.757592: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4800 of size 512 next 200 2020-10-01 02:17:58.757600: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4a00 of size 512 next 201 2020-10-01 02:17:58.757607: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4c00 of size 512 next 202 2020-10-01 02:17:58.757615: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a4e00 of size 512 next 204 2020-10-01 02:17:58.757623: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a5000 of size 512 next 205 2020-10-01 02:17:58.757631: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a5200 of size 512 next 147 2020-10-01 02:17:58.757639: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a5400 of size 18432 next 145 2020-10-01 02:17:58.757647: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680a9c00 of size 18432 next 146 2020-10-01 02:17:58.757656: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680ae400 of size 31232 next 27 2020-10-01 02:17:58.757664: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680b5e00 of size 147456 next 28 2020-10-01 02:17:58.757672: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680d9e00 of size 512 next 207 2020-10-01 02:17:58.757680: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680da000 of size 512 next 208 2020-10-01 02:17:58.757688: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680da200 of size 512 next 209 2020-10-01 02:17:58.757695: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680da400 of size 512 next 211 2020-10-01 02:17:58.757703: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680da600 of size 512 next 212 2020-10-01 02:17:58.757711: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680da800 of size 512 next 213 2020-10-01 02:17:58.757719: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680daa00 of size 512 next 215 2020-10-01 02:17:58.757727: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dac00 of size 512 next 216 2020-10-01 02:17:58.757735: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dae00 of size 512 next 217 2020-10-01 02:17:58.757743: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680db000 of size 512 next 219 2020-10-01 02:17:58.757751: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680db200 of size 512 next 220 2020-10-01 02:17:58.757759: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680db400 of size 512 next 221 2020-10-01 02:17:58.757766: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680db600 of size 512 next 223 2020-10-01 02:17:58.757774: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680db800 of size 512 next 224 2020-10-01 02:17:58.757782: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dba00 of size 512 next 225 2020-10-01 02:17:58.757790: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dbc00 of size 512 next 227 2020-10-01 02:17:58.757798: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dbe00 of size 512 next 228 2020-10-01 02:17:58.757806: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dc000 of size 512 next 229 2020-10-01 02:17:58.757814: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dc200 of size 512 next 231 2020-10-01 02:17:58.757822: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dc400 of size 512 next 232 2020-10-01 02:17:58.757830: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dc600 of size 512 next 233 2020-10-01 02:17:58.757838: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dc800 of size 512 next 235 2020-10-01 02:17:58.757845: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dca00 of size 512 next 236 2020-10-01 02:17:58.757853: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dcc00 of size 512 next 237 2020-10-01 02:17:58.757861: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dce00 of size 512 next 239 2020-10-01 02:17:58.757869: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dd000 of size 512 next 240 2020-10-01 02:17:58.757877: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dd200 of size 512 next 241 2020-10-01 02:17:58.757885: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dd400 of size 512 next 243 2020-10-01 02:17:58.757893: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dd600 of size 512 next 245 2020-10-01 02:17:58.757901: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dd800 of size 512 next 247 2020-10-01 02:17:58.757909: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dda00 of size 512 next 248 2020-10-01 02:17:58.757917: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680ddc00 of size 512 next 249 2020-10-01 02:17:58.757924: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dde00 of size 512 next 250 2020-10-01 02:17:58.757932: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de000 of size 512 next 251 2020-10-01 02:17:58.757940: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de200 of size 512 next 252 2020-10-01 02:17:58.757948: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de400 of size 512 next 253 2020-10-01 02:17:58.757956: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de600 of size 256 next 254 2020-10-01 02:17:58.757963: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de700 of size 256 next 256 2020-10-01 02:17:58.757971: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de800 of size 256 next 258 2020-10-01 02:17:58.757979: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680de900 of size 256 next 260 2020-10-01 02:17:58.757987: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dea00 of size 256 next 262 2020-10-01 02:17:58.757995: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680deb00 of size 256 next 264 2020-10-01 02:17:58.758003: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dec00 of size 1792 next 265 2020-10-01 02:17:58.758011: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df300 of size 256 next 266 2020-10-01 02:17:58.758019: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df400 of size 256 next 267 2020-10-01 02:17:58.758026: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df500 of size 256 next 269 2020-10-01 02:17:58.758034: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df600 of size 256 next 270 2020-10-01 02:17:58.758042: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df700 of size 256 next 271 2020-10-01 02:17:58.758050: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df800 of size 256 next 273 2020-10-01 02:17:58.758058: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680df900 of size 256 next 274 2020-10-01 02:17:58.758065: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dfa00 of size 256 next 275 2020-10-01 02:17:58.758073: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dfb00 of size 256 next 277 2020-10-01 02:17:58.758081: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dfc00 of size 256 next 278 2020-10-01 02:17:58.758089: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dfd00 of size 256 next 156 2020-10-01 02:17:58.758097: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680dfe00 of size 24576 next 155 2020-10-01 02:17:58.758105: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680e5e00 of size 18432 next 255 2020-10-01 02:17:58.758113: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680ea600 of size 24576 next 257 2020-10-01 02:17:58.758121: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f0600 of size 6144 next 259 2020-10-01 02:17:58.758129: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f1e00 of size 6144 next 261 2020-10-01 02:17:58.758137: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f3600 of size 6144 next 263 2020-10-01 02:17:58.758145: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f4e00 of size 256 next 280 2020-10-01 02:17:58.758152: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f4f00 of size 256 next 281 2020-10-01 02:17:58.758160: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5000 of size 256 next 282 2020-10-01 02:17:58.758168: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5100 of size 512 next 284 2020-10-01 02:17:58.758176: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5300 of size 512 next 285 2020-10-01 02:17:58.758184: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5500 of size 512 next 286 2020-10-01 02:17:58.758192: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5700 of size 512 next 288 2020-10-01 02:17:58.758200: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5900 of size 512 next 289 2020-10-01 02:17:58.758208: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5b00 of size 512 next 290 2020-10-01 02:17:58.758215: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5d00 of size 512 next 292 2020-10-01 02:17:58.758223: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f5f00 of size 512 next 293 2020-10-01 02:17:58.758231: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6100 of size 512 next 294 2020-10-01 02:17:58.758239: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6300 of size 512 next 296 2020-10-01 02:17:58.758247: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6500 of size 512 next 297 2020-10-01 02:17:58.758255: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6700 of size 512 next 298 2020-10-01 02:17:58.758263: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6900 of size 512 next 300 2020-10-01 02:17:58.758271: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6b00 of size 512 next 301 2020-10-01 02:17:58.758278: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6d00 of size 512 next 302 2020-10-01 02:17:58.758286: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f6f00 of size 512 next 304 2020-10-01 02:17:58.758294: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7100 of size 512 next 305 2020-10-01 02:17:58.758302: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7300 of size 512 next 306 2020-10-01 02:17:58.758310: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7500 of size 512 next 308 2020-10-01 02:17:58.758318: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7700 of size 512 next 309 2020-10-01 02:17:58.758325: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7900 of size 512 next 310 2020-10-01 02:17:58.758333: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7b00 of size 512 next 312 2020-10-01 02:17:58.758341: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7d00 of size 512 next 313 2020-10-01 02:17:58.758349: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f7f00 of size 512 next 314 2020-10-01 02:17:58.758357: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8100 of size 512 next 316 2020-10-01 02:17:58.758365: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8300 of size 512 next 317 2020-10-01 02:17:58.758373: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8500 of size 512 next 318 2020-10-01 02:17:58.758380: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8700 of size 512 next 320 2020-10-01 02:17:58.758388: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8900 of size 512 next 321 2020-10-01 02:17:58.758396: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8b00 of size 512 next 322 2020-10-01 02:17:58.758404: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8d00 of size 512 next 324 2020-10-01 02:17:58.758412: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f8f00 of size 512 next 325 2020-10-01 02:17:58.758420: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9100 of size 512 next 326 2020-10-01 02:17:58.758428: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9300 of size 512 next 328 2020-10-01 02:17:58.758435: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9500 of size 512 next 329 2020-10-01 02:17:58.758443: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9700 of size 512 next 330 2020-10-01 02:17:58.758451: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9900 of size 512 next 332 2020-10-01 02:17:58.758459: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9b00 of size 512 next 334 2020-10-01 02:17:58.758467: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9d00 of size 512 next 336 2020-10-01 02:17:58.758475: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680f9f00 of size 512 next 337 2020-10-01 02:17:58.758482: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fa100 of size 512 next 338 2020-10-01 02:17:58.758490: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fa300 of size 512 next 339 2020-10-01 02:17:58.758498: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fa500 of size 512 next 340 2020-10-01 02:17:58.758506: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fa700 of size 512 next 341 2020-10-01 02:17:58.758514: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fa900 of size 512 next 342 2020-10-01 02:17:58.758522: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fab00 of size 256 next 344 2020-10-01 02:17:58.758530: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fac00 of size 256 next 346 2020-10-01 02:17:58.758538: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fad00 of size 256 next 348 2020-10-01 02:17:58.758545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fae00 of size 6144 next 349 2020-10-01 02:17:58.758553: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fc600 of size 256 next 350 2020-10-01 02:17:58.758561: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fc700 of size 256 next 352 2020-10-01 02:17:58.758569: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fc800 of size 256 next 354 2020-10-01 02:17:58.758577: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fc900 of size 256 next 355 2020-10-01 02:17:58.758584: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fca00 of size 256 next 356 2020-10-01 02:17:58.758592: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fcb00 of size 256 next 357 2020-10-01 02:17:58.758600: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fcc00 of size 256 next 358 2020-10-01 02:17:58.758608: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fcd00 of size 256 next 360 2020-10-01 02:17:58.758616: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fce00 of size 256 next 361 2020-10-01 02:17:58.758624: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fcf00 of size 256 next 362 2020-10-01 02:17:58.758632: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd000 of size 256 next 363 2020-10-01 02:17:58.758639: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd100 of size 256 next 364 2020-10-01 02:17:58.758647: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd200 of size 256 next 365 2020-10-01 02:17:58.758655: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd300 of size 256 next 366 2020-10-01 02:17:58.758663: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd400 of size 256 next 367 2020-10-01 02:17:58.758671: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd500 of size 256 next 368 2020-10-01 02:17:58.758679: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd600 of size 256 next 369 2020-10-01 02:17:58.758686: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd700 of size 256 next 370 2020-10-01 02:17:58.758694: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd800 of size 256 next 371 2020-10-01 02:17:58.758702: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fd900 of size 256 next 372 2020-10-01 02:17:58.758710: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fda00 of size 256 next 373 2020-10-01 02:17:58.758718: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fdb00 of size 256 next 375 2020-10-01 02:17:58.758726: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fdc00 of size 256 next 376 2020-10-01 02:17:58.758733: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fdd00 of size 256 next 34 2020-10-01 02:17:58.758741: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12680fde00 of size 147456 next 33 2020-10-01 02:17:58.758749: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268121e00 of size 147456 next 193 2020-10-01 02:17:58.758757: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268145e00 of size 442368 next 44 2020-10-01 02:17:58.758766: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12681b1e00 of size 294912 next 43 2020-10-01 02:17:58.758774: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12681f9e00 of size 294912 next 119 2020-10-01 02:17:58.758782: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268241e00 of size 147456 next 187 2020-10-01 02:17:58.758790: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268265e00 of size 147456 next 52 2020-10-01 02:17:58.758798: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268289e00 of size 589824 next 51 2020-10-01 02:17:58.758806: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268319e00 of size 589824 next 58 2020-10-01 02:17:58.758814: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12683a9e00 of size 589824 next 64 2020-10-01 02:17:58.758822: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268439e00 of size 589824 next 70 2020-10-01 02:17:58.758829: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12684c9e00 of size 589824 next 76 2020-10-01 02:17:58.758837: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268559e00 of size 589824 next 82 2020-10-01 02:17:58.758845: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12685e9e00 of size 589824 next 88 2020-10-01 02:17:58.758853: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268679e00 of size 589824 next 94 2020-10-01 02:17:58.758861: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268709e00 of size 589824 next 100 2020-10-01 02:17:58.758869: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268799e00 of size 589824 next 106 2020-10-01 02:17:58.758877: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268829e00 of size 589824 next 112 2020-10-01 02:17:58.758884: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12688b9e00 of size 589824 next 125 2020-10-01 02:17:58.758892: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268949e00 of size 589824 next 124 2020-10-01 02:17:58.758900: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12689d9e00 of size 589824 next 131 2020-10-01 02:17:58.758908: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268a69e00 of size 589824 next 203 2020-10-01 02:17:58.758915: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268af9e00 of size 589824 next 206 2020-10-01 02:17:58.758923: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268b89e00 of size 589824 next 210 2020-10-01 02:17:58.758931: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268c19e00 of size 589824 next 214 2020-10-01 02:17:58.758939: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268ca9e00 of size 589824 next 218 2020-10-01 02:17:58.758947: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268d39e00 of size 589824 next 222 2020-10-01 02:17:58.758954: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268dc9e00 of size 589824 next 226 2020-10-01 02:17:58.758962: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268e59e00 of size 589824 next 230 2020-10-01 02:17:58.758970: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268ee9e00 of size 589824 next 234 2020-10-01 02:17:58.758978: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1268f79e00 of size 589824 next 238 2020-10-01 02:17:58.758986: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269009e00 of size 294912 next 242 2020-10-01 02:17:58.758994: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269051e00 of size 589824 next 244 2020-10-01 02:17:58.759002: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12690e1e00 of size 589824 next 246 2020-10-01 02:17:58.759009: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269171e00 of size 147456 next 268 2020-10-01 02:17:58.759017: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269195e00 of size 147456 next 272 2020-10-01 02:17:58.759025: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12691b9e00 of size 147456 next 276 2020-10-01 02:17:58.759033: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12691dde00 of size 147456 next 279 2020-10-01 02:17:58.759041: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269201e00 of size 294912 next 283 2020-10-01 02:17:58.759048: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269249e00 of size 589824 next 287 2020-10-01 02:17:58.759056: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12692d9e00 of size 589824 next 291 2020-10-01 02:17:58.759064: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269369e00 of size 589824 next 295 2020-10-01 02:17:58.759072: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12693f9e00 of size 589824 next 299 2020-10-01 02:17:58.759079: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269489e00 of size 589824 next 303 2020-10-01 02:17:58.759087: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269519e00 of size 589824 next 307 2020-10-01 02:17:58.759095: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12695a9e00 of size 589824 next 311 2020-10-01 02:17:58.759103: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269639e00 of size 589824 next 315 2020-10-01 02:17:58.759111: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12696c9e00 of size 589824 next 319 2020-10-01 02:17:58.759119: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269759e00 of size 589824 next 323 2020-10-01 02:17:58.759127: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12697e9e00 of size 589824 next 327 2020-10-01 02:17:58.759134: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269879e00 of size 294912 next 331 2020-10-01 02:17:58.759142: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12698c1e00 of size 589824 next 333 2020-10-01 02:17:58.759150: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1269951e00 of size 589824 next 335 2020-10-01 02:17:58.759158: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699e1e00 of size 18432 next 343 2020-10-01 02:17:58.759166: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699e6600 of size 18432 next 345 2020-10-01 02:17:58.759174: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699eae00 of size 24576 next 347 2020-10-01 02:17:58.759182: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699f0e00 of size 6144 next 351 2020-10-01 02:17:58.759189: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699f2600 of size 6144 next 353 2020-10-01 02:17:58.759198: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12699f3e00 of size 12192768 next 359 2020-10-01 02:17:58.759206: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126a594a00 of size 4064256 next 374 2020-10-01 02:17:58.759214: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126a974e00 of size 256 next 377 2020-10-01 02:17:58.759221: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126a974f00 of size 256 next 378 2020-10-01 02:17:58.759229: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126a975000 of size 256 next 379 2020-10-01 02:17:58.759237: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126a975100 of size 256 next 380 2020-10-01 02:17:58.759245: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f126a975200 of size 4064256 next 381 2020-10-01 02:17:58.759253: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126ad55600 of size 12288000 next 11 2020-10-01 02:17:58.759261: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f126b90d600 of size 122112000 next 382 2020-10-01 02:17:58.759269: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f1272d81e00 of size 3048192 next 501 2020-10-01 02:17:58.759277: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306a100 of size 256 next 517 2020-10-01 02:17:58.759285: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f127306a200 of size 2560 next 504 2020-10-01 02:17:58.759293: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306ac00 of size 256 next 505 2020-10-01 02:17:58.759301: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f127306ad00 of size 2816 next 519 2020-10-01 02:17:58.759308: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306b800 of size 256 next 520 2020-10-01 02:17:58.759316: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306b900 of size 256 next 521 2020-10-01 02:17:58.759324: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306ba00 of size 256 next 515 2020-10-01 02:17:58.759332: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306bb00 of size 256 next 522 2020-10-01 02:17:58.759340: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306bc00 of size 256 next 523 2020-10-01 02:17:58.759348: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127306bd00 of size 18432 next 484 2020-10-01 02:17:58.759356: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f1273070500 of size 30720 next 400 2020-10-01 02:17:58.759363: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1273077d00 of size 18432 next 494 2020-10-01 02:17:58.759371: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f127307c500 of size 55296 next 512 2020-10-01 02:17:58.759379: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1273089d00 of size 6144 next 499 2020-10-01 02:17:58.759387: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127308b500 of size 6144 next 503 2020-10-01 02:17:58.759395: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f127308cd00 of size 17130752 next 385 2020-10-01 02:17:58.759403: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12740e3200 of size 256 next 386 2020-10-01 02:17:58.759410: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f12740e3300 of size 36578304 next 390 2020-10-01 02:17:58.759419: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12763c5700 of size 576000 next 391 2020-10-01 02:17:58.759427: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f1276452100 of size 2816 next 392 2020-10-01 02:17:58.759434: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276452c00 of size 256 next 404 2020-10-01 02:17:58.759442: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276452d00 of size 256 next 405 2020-10-01 02:17:58.759450: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276452e00 of size 256 next 406 2020-10-01 02:17:58.759458: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276452f00 of size 256 next 411 2020-10-01 02:17:58.759466: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453000 of size 256 next 412 2020-10-01 02:17:58.759474: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453100 of size 256 next 415 2020-10-01 02:17:58.759481: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453200 of size 256 next 416 2020-10-01 02:17:58.759489: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453300 of size 256 next 418 2020-10-01 02:17:58.759497: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453400 of size 256 next 419 2020-10-01 02:17:58.759505: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453500 of size 256 next 422 2020-10-01 02:17:58.759513: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453600 of size 256 next 423 2020-10-01 02:17:58.759520: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453700 of size 512 next 427 2020-10-01 02:17:58.759528: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453900 of size 512 next 428 2020-10-01 02:17:58.759536: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453b00 of size 512 next 429 2020-10-01 02:17:58.759544: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453d00 of size 512 next 432 2020-10-01 02:17:58.759552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276453f00 of size 512 next 433 2020-10-01 02:17:58.759559: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454100 of size 512 next 434 2020-10-01 02:17:58.759567: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454300 of size 512 next 435 2020-10-01 02:17:58.759575: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454500 of size 512 next 436 2020-10-01 02:17:58.759583: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454700 of size 512 next 437 2020-10-01 02:17:58.759591: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454900 of size 512 next 442 2020-10-01 02:17:58.759599: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454b00 of size 512 next 443 2020-10-01 02:17:58.759607: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454d00 of size 512 next 446 2020-10-01 02:17:58.759614: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276454f00 of size 512 next 447 2020-10-01 02:17:58.759622: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455100 of size 512 next 450 2020-10-01 02:17:58.759630: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455300 of size 512 next 451 2020-10-01 02:17:58.759638: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455500 of size 512 next 456 2020-10-01 02:17:58.759645: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455700 of size 512 next 457 2020-10-01 02:17:58.759653: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455900 of size 512 next 460 2020-10-01 02:17:58.759661: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455b00 of size 512 next 461 2020-10-01 02:17:58.759669: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455d00 of size 512 next 462 2020-10-01 02:17:58.759677: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276455f00 of size 512 next 463 2020-10-01 02:17:58.759685: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456100 of size 512 next 464 2020-10-01 02:17:58.759693: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456300 of size 512 next 465 2020-10-01 02:17:58.759700: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456500 of size 512 next 470 2020-10-01 02:17:58.759708: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456700 of size 512 next 471 2020-10-01 02:17:58.759716: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456900 of size 512 next 474 2020-10-01 02:17:58.759724: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456b00 of size 512 next 475 2020-10-01 02:17:58.759732: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456d00 of size 512 next 478 2020-10-01 02:17:58.759740: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276456f00 of size 512 next 479 2020-10-01 02:17:58.759747: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276457100 of size 512 next 482 2020-10-01 02:17:58.759755: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276457300 of size 512 next 483 2020-10-01 02:17:58.759763: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f1276457500 of size 11155200 next 396 2020-10-01 02:17:58.759771: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276efac00 of size 256 next 397 2020-10-01 02:17:58.759779: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1276efad00 of size 8128000 next 399 2020-10-01 02:17:58.759787: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12776bb300 of size 134400000 next 401 2020-10-01 02:17:58.759795: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f127f6e7b00 of size 1228800000 next 402 2020-10-01 02:17:58.759803: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f12c8ac7b00 of size 1228800000 next 403 2020-10-01 02:17:58.759812: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1311ea7b00 of size 65028096 next 409 2020-10-01 02:17:58.759820: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1315cabb00 of size 65028096 next 413 2020-10-01 02:17:58.759827: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1319aafb00 of size 65028096 next 19 2020-10-01 02:17:58.759835: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f131d8b3b00 of size 65028096 next 408 2020-10-01 02:17:58.759843: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13216b7b00 of size 260112384 next 407 2020-10-01 02:17:58.759851: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1330ec7b00 of size 65028096 next 24 2020-10-01 02:17:58.759859: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1334ccbb00 of size 65028096 next 31 2020-10-01 02:17:58.759867: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1338acfb00 of size 65028096 next 420 2020-10-01 02:17:58.759881: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f133c8d3b00 of size 32514048 next 425 2020-10-01 02:17:58.759890: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f133e7d5b00 of size 32514048 next 38 2020-10-01 02:17:58.759898: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13406d7b00 of size 32514048 next 47 2020-10-01 02:17:58.759906: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13425d9b00 of size 32514048 next 424 2020-10-01 02:17:58.759914: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13444dbb00 of size 130056192 next 426 2020-10-01 02:17:58.759922: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f134c0e3b00 of size 130056192 next 430 2020-10-01 02:17:58.759929: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1353cebb00 of size 32514048 next 122 2020-10-01 02:17:58.759937: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1355bedb00 of size 32514048 next 56 2020-10-01 02:17:58.759945: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1357aefb00 of size 32514048 next 62 2020-10-01 02:17:58.759953: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13599f1b00 of size 32514048 next 440 2020-10-01 02:17:58.759961: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f135b8f3b00 of size 32514048 next 444 2020-10-01 02:17:58.759969: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f135d7f5b00 of size 32514048 next 68 2020-10-01 02:17:58.759976: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f135f6f7b00 of size 32514048 next 448 2020-10-01 02:17:58.759984: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13615f9b00 of size 32514048 next 74 2020-10-01 02:17:58.759992: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13634fbb00 of size 8128512 next 80 2020-10-01 02:17:58.760000: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1363cbc300 of size 8128512 next 453 2020-10-01 02:17:58.760008: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f136447cb00 of size 8128512 next 458 2020-10-01 02:17:58.760016: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1364c3d300 of size 8128512 next 86 2020-10-01 02:17:58.760024: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f13653fdb00 of size 236761088 next 454 2020-10-01 02:17:58.760032: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13735c8b00 of size 130056192 next 455 2020-10-01 02:17:58.760040: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137b1d0b00 of size 8128512 next 92 2020-10-01 02:17:58.760047: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137b991300 of size 8128512 next 129 2020-10-01 02:17:58.760055: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137c151b00 of size 8128512 next 98 2020-10-01 02:17:58.760063: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137c912300 of size 8128512 next 468 2020-10-01 02:17:58.760071: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137d0d2b00 of size 8128512 next 104 2020-10-01 02:17:58.760079: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137d893300 of size 8128512 next 472 2020-10-01 02:17:58.760087: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137e053b00 of size 8128512 next 476 2020-10-01 02:17:58.760094: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137e814300 of size 8128512 next 110 2020-10-01 02:17:58.760102: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f137efd4b00 of size 4064256 next 516 2020-10-01 02:17:58.760110: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f137f3b4f00 of size 12192768 next 490 2020-10-01 02:17:58.760118: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f137ff55b00 of size 113799168 next 116 2020-10-01 02:17:58.760126: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1386bdcb00 of size 130056192 next 481 2020-10-01 02:17:58.760134: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f138e7e4b00 of size 390168576 next 135 2020-10-01 02:17:58.760142: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13a5bfcb00 of size 390168576 next 389 2020-10-01 02:17:58.760150: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13bd014b00 of size 390168576 next 395 2020-10-01 02:17:58.760158: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13d442cb00 of size 406945792 next 383 2020-10-01 02:17:58.760165: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f13ec844b00 of size 390168576 next 384 2020-10-01 02:17:58.760174: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 7f1403c5cb00 of size 253356032 next 18446744073709551615 2020-10-01 02:17:58.760195: I tensorflow/core/common_runtime/bfc_allocator.cc:1031] Summary of in-use Chunks by size: 2020-10-01 02:17:58.760208: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 154 Chunks of size 256 totalling 38.5KiB 2020-10-01 02:17:58.760219: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 195 Chunks of size 512 totalling 97.5KiB 2020-10-01 02:17:58.760229: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1280 totalling 1.2KiB 2020-10-01 02:17:58.760238: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 1792 totalling 5.2KiB 2020-10-01 02:17:58.760248: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 10 Chunks of size 6144 totalling 60.0KiB 2020-10-01 02:17:58.760258: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 6400 totalling 6.2KiB 2020-10-01 02:17:58.760268: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 7 Chunks of size 18432 totalling 126.0KiB 2020-10-01 02:17:58.760278: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 24576 totalling 72.0KiB 2020-10-01 02:17:58.760288: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 31232 totalling 30.5KiB 2020-10-01 02:17:58.760298: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 11 Chunks of size 147456 totalling 1.55MiB 2020-10-01 02:17:58.760307: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 294144 totalling 287.2KiB 2020-10-01 02:17:58.760315: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 5 Chunks of size 294912 totalling 1.41MiB 2020-10-01 02:17:58.760323: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 442368 totalling 432.0KiB 2020-10-01 02:17:58.760333: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 576000 totalling 562.5KiB 2020-10-01 02:17:58.760344: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 39 Chunks of size 589824 totalling 21.94MiB 2020-10-01 02:17:58.760354: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4064256 totalling 3.88MiB 2020-10-01 02:17:58.760363: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 8128000 totalling 7.75MiB 2020-10-01 02:17:58.760373: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 12 Chunks of size 8128512 totalling 93.02MiB 2020-10-01 02:17:58.760382: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 12192768 totalling 23.26MiB 2020-10-01 02:17:58.760390: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12288000 totalling 11.72MiB 2020-10-01 02:17:58.760399: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 12 Chunks of size 32514048 totalling 372.09MiB 2020-10-01 02:17:58.760408: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 7 Chunks of size 65028096 totalling 434.11MiB 2020-10-01 02:17:58.760419: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 122112000 totalling 116.46MiB 2020-10-01 02:17:58.760429: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 130056192 totalling 372.09MiB 2020-10-01 02:17:58.760439: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 134400000 totalling 128.17MiB 2020-10-01 02:17:58.760449: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 260112384 totalling 248.06MiB 2020-10-01 02:17:58.760459: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 4 Chunks of size 390168576 totalling 1.45GiB 2020-10-01 02:17:58.760469: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 406945792 totalling 388.09MiB 2020-10-01 02:17:58.760479: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 1228800000 totalling 2.29GiB 2020-10-01 02:17:58.760489: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 5.92GiB 2020-10-01 02:17:58.760498: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocatedbytes: 7161754368 memorylimit: 7161754432 available bytes: 64 curr_region_allocationbytes: 14323509248 2020-10-01 02:17:58.760513: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats: Limit: 7161754432 InUse: 6351646208 MaxInUse: 6658663936 NumAllocs: 1628 MaxAllocSize: 2317877248 Reserved: 0 PeakReserved: 0 LargestFreeBlock: 0

2020-10-01 02:17:58.760558: W tensorflow/core/common_runtime/bfcallocator.cc:439] ***__**_**___ 2020-10-01 02:17:58.760583: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_grad_input_ops.cc:1103 : Resource exhausted: OOM when allocating tensor with shape[4,384,252,252] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "point_pillars_training_run.py", line 55, in pillar_net.fit(training_gen, File "/home/yg-ubuntu1804/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper return method(self, *args, kwargs) File "/home/yg-ubuntu1804/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit tmp_logs = train_function(iterator) File "/home/yg-ubuntu1804/.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 780, in call result = self._call(*args, *kwds) File "/home/yg-ubuntu1804/.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 840, in _call return self._stateless_fn(args, **kwds) File "/home/yg-ubuntu1804/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2829, in call return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access File "/home/yg-ubuntu1804/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1843, in _filtered_call return self._call_flat( File "/home/yg-ubuntu1804/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1923, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/home/yg-ubuntu1804/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 545, in call outputs = execute.execute( File "/home/yg-ubuntu1804/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4,384,252,252] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node gradient_tape/functional_1/angle/conv2d/Conv2D/Conv2DBackpropInput (defined at point_pillars_training_run.py:55) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [Op:__inference_train_function_8410]

Function call stack: train_function

2020-10-01 02:17:58.925699: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated. [[{{node PyFunc}}]]

[tyagi-iiitv]

This seems to be related to your GPU memory. I have seen a similar error before #14 but that was because Tf doesn't free up the memory after we stop running the model. More details about this issue can be found in https://github.com/tensorflow/tensorflow/issues/36465 and https://github.com/tensorflow/tensorflow/issues/1757

I think the only resolution is to use a smaller batch size in this case, because this is supposedly a large network.

[XanderKehoe]

While i'm not done thoroughly testing it yet, I believe a combination of the solutions above has worked for me:

if __name__ == "__main__":
    for learning_rate in np.linspace(min_learning_rate, max_learning_rate, num_of_learning_rate_iterations):
        for dense_layer_count in range(min_num_of_dense_layers, max_num_of_dense_layers):
            for bs in range(batch_size_min, batch_size_max, int((batch_size_max - batch_size_min) / num_of_batch_size_iterations)):
                for dense_layer_size in dense_layer_sizes:
                    p = multiprocessing.Process(target=train(dense_layer_count, learning_rate, bs, dense_layer_size))
                    p.start()
                    p.join()
                    p.terminate()
                    tf.keras.backend.clear_session()

Will update if I experience any issues with this.