search

IBM / tensorflow-large-model-support

Large Model Support in Tensorflow
Apache License 2.0
202 stars 38 forks source link

LMS does not take into account swap size on linux #33

Closed zpxp closed 4 years ago

zpxp commented 4 years ago

GPU: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] 4032MiB Memory

CPU: Intel(R) Core(TM) i7-6850K CPU @ 3.60GHz

RAM: 23 Gb

OS: Docker running on linux host

===

You can create swap files on linux meaning you can create huge amounts of swap to use as ram space for training see here but the LMS doesnt seem to take into account swap size when allocating memory on the host.

Using ibmcom/powerai:1.7.0-tensorflow-ubuntu18.04-py37 docker image and running the example found on the main branch of this repo you can reproduce this by starting the docker image like this

docker run --rm -it \
    --gpus all \
   --workdir=/app \
   --oom-kill-disable \
   --memory="10gb" \
   --memory-swap="10gb" \
  --env TF_GPU_HOST_MEM_LIMIT_IN_MB=700000 \
   --volume="$PWD":/app \
   $containerId  bash

and then running the example python ManyModel.py --lms --image_size 2000 (or w/e image size runs on 10gb of memory) then reduce memory and increase swap space (same total memory size) while keeping the same image size

docker run --rm -it \
    --gpus all \
   --workdir=/app \
   --oom-kill-disable \
   --memory="5gb" \
   --memory-swap="15gb" \
  --env TF_GPU_HOST_MEM_LIMIT_IN_MB=700000 \
   --volume="$PWD":/app \
   $containerId  bash

but doing this crashes out of memory even tho the total memory remains the same as before

Click to Expand ``` 2020-04-26 06:22:20.016242: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (2097152): Total Chunks: 24, Chunks in use: 24. 52.50MiB allocated for chunks. 52.50MiB in use in bin. 52.50MiB client-requested in use in bin. 2020-04-26 06:22:20.016256: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (4194304): Total Chunks: 15, Chunks in use: 15. 60.00MiB allocated for chunks. 60.00MiB in use in bin. 60.00MiB client-requested in use in bin. 2020-04-26 06:22:20.016268: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (8388608): Total Chunks: 12, Chunks in use: 12. 106.14MiB allocated for chunks. 106.14MiB in use in bin. 105.00MiB client-requested in use in bin. 2020-04-26 06:22:20.016280: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (16777216): Total Chunks: 16, Chunks in use: 16. 488.28MiB allocated for chunks. 488.28MiB in use in bin. 488.28MiB client-requested in use in bin. 2020-04-26 06:22:20.016450: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (33554432): Total Chunks: 12, Chunks in use: 12. 717.44MiB allocated for chunks. 717.44MiB in use in bin. 717.44MiB client-requested in use in bin. 2020-04-26 06:22:20.016467: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (67108864): Total Chunks: 8, Chunks in use: 7. 886.06MiB allocated for chunks. 803.54MiB in use in bin. 732.42MiB client-requested in use in bin. 2020-04-26 06:22:20.016477: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (134217728): Total Chunks: 10, Chunks in use: 10. 2.36GiB allocated for chunks. 2.36GiB in use in bin. 2.27GiB client-requested in use in bin. 2020-04-26 06:22:20.016488: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (268435456): Total Chunks: 1, Chunks in use: 1. 256.00MiB allocated for chunks. 256.00MiB in use in bin. 244.14MiB client-requested in use in bin. 2020-04-26 06:22:20.016499: I tensorflow/core/common_runtime/bfc_allocator.cc:1311] Bin for 122.07MiB was 64.00MiB, Chunk State: 2020-04-26 06:22:20.016699: I tensorflow/core/common_runtime/bfc_allocator.cc:1317] Size: 82.52MiB | Requested Size: 0B | in_use: 0 | bin_num: 18, prev: Size: 30.52MiB | Requested Size: 30.52MiB | in_use: 1 | bin_num: -1 2020-04-26 06:22:20.017061: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 982550528 2020-04-26 06:22:20.017283: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad7e000000 of size 2048 next 873 2020-04-26 06:22:20.017327: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad7e000800 of size 2048 next 874 2020-04-26 06:22:20.017347: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad7e001000 of size 32000000 next 875 2020-04-26 06:22:20.017370: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad7fe85800 of size 512 next 876 2020-04-26 06:22:20.017392: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad7fe85a00 of size 512 next 877 2020-04-26 06:22:20.017413: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad7fe85c00 of size 32000000 next 878 2020-04-26 06:22:20.017439: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad81d0a400 of size 32000000 next 879 2020-04-26 06:22:20.017460: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad83b8ec00 of size 512 next 880 2020-04-26 06:22:20.017486: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad83b8ee00 of size 512 next 881 2020-04-26 06:22:20.017517: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad83b8f000 of size 32000000 next 882 2020-04-26 06:22:20.017547: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad85a13800 of size 128000000 next 883 2020-04-26 06:22:20.017574: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad8d425800 of size 2048 next 884 2020-04-26 06:22:20.017604: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad8d426000 of size 2048 next 885 2020-04-26 06:22:20.017633: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad8d426800 of size 128000000 next 886 2020-04-26 06:22:20.017662: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad94e38800 of size 32000000 next 887 2020-04-26 06:22:20.017690: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad96cbd000 of size 512 next 888 2020-04-26 06:22:20.017712: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad96cbd200 of size 512 next 889 2020-04-26 06:22:20.017739: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad96cbd400 of size 32000000 next 890 2020-04-26 06:22:20.017770: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad98b41c00 of size 32000000 next 891 2020-04-26 06:22:20.017799: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad9a9c6400 of size 512 next 892 2020-04-26 06:22:20.017827: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad9a9c6600 of size 512 next 893 2020-04-26 06:22:20.017858: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad9a9c6800 of size 32000000 next 894 2020-04-26 06:22:20.017887: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fad9c84b000 of size 128000000 next 895 2020-04-26 06:22:20.017911: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fada425d000 of size 2048 next 896 2020-04-26 06:22:20.017929: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fada425d800 of size 2048 next 897 2020-04-26 06:22:20.017954: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fada425e000 of size 128000000 next 898 2020-04-26 06:22:20.017980: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadabc70000 of size 32000000 next 899 2020-04-26 06:22:20.018005: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadadaf4800 of size 512 next 900 2020-04-26 06:22:20.018030: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadadaf4a00 of size 512 next 901 2020-04-26 06:22:20.018054: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadadaf4c00 of size 32000000 next 902 2020-04-26 06:22:20.018071: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadaf979400 of size 32000000 next 903 2020-04-26 06:22:20.018089: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadb17fdc00 of size 512 next 904 2020-04-26 06:22:20.018118: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadb17fde00 of size 512 next 905 2020-04-26 06:22:20.018147: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadb17fe000 of size 32000000 next 906 2020-04-26 06:22:20.018172: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free at 7fadb3682800 of size 86532096 next 18446744073709551615 2020-04-26 06:22:20.018197: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 2147483648 2020-04-26 06:22:20.018229: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadc0000000 of size 64000000 next 841 2020-04-26 06:22:20.018256: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadc3d09000 of size 64000000 next 843 2020-04-26 06:22:20.018275: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadc7a12000 of size 256000000 next 844 2020-04-26 06:22:20.018303: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadd6e36000 of size 1024 next 845 2020-04-26 06:22:20.018339: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadd6e36400 of size 1024 next 846 2020-04-26 06:22:20.018369: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadd6e36800 of size 256000000 next 847 2020-04-26 06:22:20.018396: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fade625a800 of size 64000000 next 848 2020-04-26 06:22:20.018425: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fade9f63800 of size 256 next 849 2020-04-26 06:22:20.018453: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fade9f63900 of size 256 next 850 2020-04-26 06:22:20.018482: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fade9f63a00 of size 64000000 next 851 2020-04-26 06:22:20.018509: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadedc6ca00 of size 64000000 next 852 2020-04-26 06:22:20.018538: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadf1975a00 of size 256 next 853 2020-04-26 06:22:20.018564: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadf1975b00 of size 256 next 854 2020-04-26 06:22:20.018592: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadf1975c00 of size 64000000 next 855 2020-04-26 06:22:20.018619: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fadf567ec00 of size 256000000 next 856 2020-04-26 06:22:20.018647: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae04aa2c00 of size 1024 next 857 2020-04-26 06:22:20.018677: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae04aa3000 of size 1024 next 858 2020-04-26 06:22:20.018706: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae04aa3400 of size 256000000 next 859 2020-04-26 06:22:20.018736: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae13ec7400 of size 256000000 next 860 2020-04-26 06:22:20.018764: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae232eb400 of size 128000000 next 861 2020-04-26 06:22:20.018784: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae2acfd400 of size 2048 next 862 2020-04-26 06:22:20.018804: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae2acfdc00 of size 2048 next 863 2020-04-26 06:22:20.018822: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae2acfe400 of size 32000000 next 864 2020-04-26 06:22:20.018842: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae2cb82c00 of size 512 next 865 2020-04-26 06:22:20.018860: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae2cb82e00 of size 512 next 866 2020-04-26 06:22:20.018878: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae2cb83000 of size 32000000 next 867 2020-04-26 06:22:20.018896: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae2ea07800 of size 32000000 next 870 2020-04-26 06:22:20.018914: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae3088c000 of size 512 next 871 2020-04-26 06:22:20.018932: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae3088c200 of size 512 next 869 2020-04-26 06:22:20.018953: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae3088c400 of size 32000000 next 868 2020-04-26 06:22:20.018978: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae32710c00 of size 227472384 next 18446744073709551615 2020-04-26 06:22:20.019007: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 1073741824 2020-04-26 06:22:20.019034: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae40000000 of size 257025024 next 801 2020-04-26 06:22:20.019068: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae4f51e400 of size 256000000 next 807 2020-04-26 06:22:20.019088: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae5e942400 of size 147456 next 817 2020-04-26 06:22:20.019108: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae5e966400 of size 256 next 818 2020-04-26 06:22:20.019126: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae5e966500 of size 64000000 next 819 2020-04-26 06:22:20.019153: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae6266f500 of size 256 next 820 2020-04-26 06:22:20.019172: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae6266f600 of size 256 next 821 2020-04-26 06:22:20.019190: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae6266f700 of size 256 next 822 2020-04-26 06:22:20.019207: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae6266f800 of size 256 next 823 2020-04-26 06:22:20.019234: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae6266f900 of size 64000000 next 824 2020-04-26 06:22:20.019262: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae66378900 of size 65536 next 825 2020-04-26 06:22:20.019281: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae66388900 of size 1024 next 826 2020-04-26 06:22:20.019302: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae66388d00 of size 256000000 next 827 2020-04-26 06:22:20.019332: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757acd00 of size 1024 next 828 2020-04-26 06:22:20.019355: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757ad100 of size 1024 next 829 2020-04-26 06:22:20.019388: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757ad500 of size 1024 next 830 2020-04-26 06:22:20.019411: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757ad900 of size 1024 next 831 2020-04-26 06:22:20.019429: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757add00 of size 1024 next 832 2020-04-26 06:22:20.019447: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757ae100 of size 1024 next 833 2020-04-26 06:22:20.019477: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757ae500 of size 256 next 838 2020-04-26 06:22:20.019498: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757ae600 of size 256 next 839 2020-04-26 06:22:20.019516: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757ae700 of size 256 next 842 2020-04-26 06:22:20.019533: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757ae800 of size 256 next 834 2020-04-26 06:22:20.019563: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757ae900 of size 256 next 835 2020-04-26 06:22:20.019583: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757aea00 of size 65536 next 836 2020-04-26 06:22:20.019601: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae757bea00 of size 64000000 next 837 2020-04-26 06:22:20.019621: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae794c7a00 of size 112428544 next 18446744073709551615 2020-04-26 06:22:20.019650: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 536870912 2020-04-26 06:22:20.019671: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80000000 of size 8192 next 707 2020-04-26 06:22:20.019689: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80002000 of size 8192 next 708 2020-04-26 06:22:20.019710: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80004000 of size 1024 next 709 2020-04-26 06:22:20.019738: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80004400 of size 1024 next 710 2020-04-26 06:22:20.019757: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80004800 of size 2048 next 711 2020-04-26 06:22:20.019775: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80005000 of size 2048 next 712 2020-04-26 06:22:20.019796: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80005800 of size 2048 next 713 2020-04-26 06:22:20.019825: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80006000 of size 2048 next 714 2020-04-26 06:22:20.019853: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80006800 of size 4096 next 715 2020-04-26 06:22:20.019874: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80007800 of size 4096 next 716 2020-04-26 06:22:20.019904: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80008800 of size 4096 next 717 2020-04-26 06:22:20.019929: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80009800 of size 4096 next 718 2020-04-26 06:22:20.019958: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8000a800 of size 1024 next 719 2020-04-26 06:22:20.019986: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8000ac00 of size 1024 next 720 2020-04-26 06:22:20.020015: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8000b000 of size 1024 next 721 2020-04-26 06:22:20.020036: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8000b400 of size 1024 next 722 2020-04-26 06:22:20.020058: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8000b800 of size 2048 next 723 2020-04-26 06:22:20.020087: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8000c000 of size 2048 next 724 2020-04-26 06:22:20.020107: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8000c800 of size 1048576 next 725 2020-04-26 06:22:20.020126: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8010c800 of size 1048576 next 726 2020-04-26 06:22:20.020162: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8020c800 of size 1024 next 727 2020-04-26 06:22:20.020192: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8020cc00 of size 1024 next 728 2020-04-26 06:22:20.020216: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8020d000 of size 4096 next 729 2020-04-26 06:22:20.020245: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8020e000 of size 4096 next 730 2020-04-26 06:22:20.020278: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae8020f000 of size 4096 next 731 2020-04-26 06:22:20.020311: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80210000 of size 4096 next 732 2020-04-26 06:22:20.020336: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80211000 of size 1024 next 733 2020-04-26 06:22:20.020366: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80211400 of size 1024 next 734 2020-04-26 06:22:20.020396: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80211800 of size 1024 next 735 2020-04-26 06:22:20.020447: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80211c00 of size 1024 next 736 2020-04-26 06:22:20.020476: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80212000 of size 4096 next 737 2020-04-26 06:22:20.020499: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80213000 of size 4096 next 738 2020-04-26 06:22:20.020519: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80214000 of size 2097152 next 739 2020-04-26 06:22:20.020545: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80414000 of size 2097152 next 740 2020-04-26 06:22:20.020567: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80614000 of size 1024 next 741 2020-04-26 06:22:20.020593: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80614400 of size 1024 next 742 2020-04-26 06:22:20.020611: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80614800 of size 4096 next 743 2020-04-26 06:22:20.020629: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80615800 of size 4096 next 744 2020-04-26 06:22:20.020655: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80616800 of size 1024 next 745 2020-04-26 06:22:20.020684: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80616c00 of size 1024 next 746 2020-04-26 06:22:20.020708: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80617000 of size 2359296 next 747 2020-04-26 06:22:20.020747: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80857000 of size 2359296 next 748 2020-04-26 06:22:20.020776: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80a97000 of size 1048576 next 749 2020-04-26 06:22:20.020796: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80b97000 of size 1048576 next 750 2020-04-26 06:22:20.020814: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c97000 of size 1024 next 751 2020-04-26 06:22:20.020842: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c97400 of size 1024 next 752 2020-04-26 06:22:20.020861: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c97800 of size 1024 next 753 2020-04-26 06:22:20.020883: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c97c00 of size 1024 next 754 2020-04-26 06:22:20.020904: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c98000 of size 2048 next 755 2020-04-26 06:22:20.020925: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c98800 of size 2048 next 756 2020-04-26 06:22:20.020947: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c99000 of size 4096 next 757 2020-04-26 06:22:20.020964: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c9a000 of size 4096 next 758 2020-04-26 06:22:20.020980: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c9b000 of size 1024 next 759 2020-04-26 06:22:20.020996: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c9b400 of size 1024 next 760 2020-04-26 06:22:20.021022: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c9b800 of size 4096 next 761 2020-04-26 06:22:20.021039: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c9c800 of size 4096 next 762 2020-04-26 06:22:20.021056: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c9d800 of size 4096 next 763 2020-04-26 06:22:20.021072: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c9e800 of size 4096 next 764 2020-04-26 06:22:20.021095: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c9f800 of size 512 next 765 2020-04-26 06:22:20.021115: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c9fa00 of size 512 next 766 2020-04-26 06:22:20.021131: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80c9fc00 of size 2048 next 767 2020-04-26 06:22:20.021147: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca0400 of size 2048 next 768 2020-04-26 06:22:20.021164: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca0c00 of size 256 next 769 2020-04-26 06:22:20.021180: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca0d00 of size 256 next 770 2020-04-26 06:22:20.021196: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca0e00 of size 2048 next 771 2020-04-26 06:22:20.021212: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca1600 of size 2048 next 772 2020-04-26 06:22:20.021228: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca1e00 of size 2048 next 773 2020-04-26 06:22:20.021244: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca2600 of size 2048 next 774 2020-04-26 06:22:20.021260: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca2e00 of size 2048 next 775 2020-04-26 06:22:20.021276: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca3600 of size 2048 next 776 2020-04-26 06:22:20.021293: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca3e00 of size 512 next 777 2020-04-26 06:22:20.021309: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca4000 of size 512 next 778 2020-04-26 06:22:20.021325: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80ca4200 of size 524288 next 779 2020-04-26 06:22:20.021343: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80d24200 of size 524288 next 780 2020-04-26 06:22:20.021359: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da4200 of size 512 next 781 2020-04-26 06:22:20.021375: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da4400 of size 512 next 782 2020-04-26 06:22:20.021392: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da4600 of size 2048 next 783 2020-04-26 06:22:20.021408: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da4e00 of size 2048 next 784 2020-04-26 06:22:20.021424: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da5600 of size 512 next 785 2020-04-26 06:22:20.021440: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da5800 of size 512 next 786 2020-04-26 06:22:20.021456: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da5a00 of size 1024 next 787 2020-04-26 06:22:20.021472: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da5e00 of size 1024 next 788 2020-04-26 06:22:20.021489: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da6200 of size 256 next 789 2020-04-26 06:22:20.021505: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da6300 of size 1024 next 790 2020-04-26 06:22:20.021521: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da6700 of size 1024 next 791 2020-04-26 06:22:20.021538: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae80da6b00 of size 48288512 next 792 2020-04-26 06:22:20.021555: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae83bb3e00 of size 37632 next 793 2020-04-26 06:22:20.021571: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae83bbd100 of size 256 next 794 2020-04-26 06:22:20.021589: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae83bbd200 of size 256 next 795 2020-04-26 06:22:20.021605: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae83bbd300 of size 256 next 796 2020-04-26 06:22:20.021621: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae83bbd400 of size 256 next 797 2020-04-26 06:22:20.021638: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae83bbd500 of size 256 next 798 2020-04-26 06:22:20.021658: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae83bbd600 of size 256000000 next 799 2020-04-26 06:22:20.021684: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae92fe1600 of size 65536 next 802 2020-04-26 06:22:20.021712: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae92ff1600 of size 64000000 next 803 2020-04-26 06:22:20.021742: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae96cfa600 of size 16384 next 804 2020-04-26 06:22:20.021762: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae96cfe600 of size 1024 next 805 2020-04-26 06:22:20.021778: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae96cfea00 of size 256 next 806 2020-04-26 06:22:20.021795: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae96cfeb00 of size 1024 next 808 2020-04-26 06:22:20.021810: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae96cfef00 of size 1024 next 809 2020-04-26 06:22:20.021826: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae96cff300 of size 1024 next 810 2020-04-26 06:22:20.021842: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae96cff700 of size 1024 next 811 2020-04-26 06:22:20.021858: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae96cffb00 of size 64000000 next 812 2020-04-26 06:22:20.021881: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae9aa08b00 of size 256 next 813 2020-04-26 06:22:20.021897: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae9aa08c00 of size 256 next 814 2020-04-26 06:22:20.021913: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae9aa08d00 of size 256 next 815 2020-04-26 06:22:20.021928: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae9aa08e00 of size 256 next 816 2020-04-26 06:22:20.021945: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fae9aa08f00 of size 90140928 next 18446744073709551615 2020-04-26 06:22:20.021963: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 268435456 2020-04-26 06:22:20.021991: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0000000 of size 2048 next 103 2020-04-26 06:22:20.022020: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0000800 of size 2048 next 104 2020-04-26 06:22:20.022045: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0001000 of size 2048 next 105 2020-04-26 06:22:20.022062: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0001800 of size 2048 next 106 2020-04-26 06:22:20.022093: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0002000 of size 2048 next 107 2020-04-26 06:22:20.022129: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0002800 of size 131072 next 108 2020-04-26 06:22:20.022164: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0022800 of size 512 next 109 2020-04-26 06:22:20.022196: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0022a00 of size 512 next 110 2020-04-26 06:22:20.022228: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0022c00 of size 512 next 111 2020-04-26 06:22:20.022259: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0022e00 of size 512 next 112 2020-04-26 06:22:20.022290: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0023000 of size 512 next 113 2020-04-26 06:22:20.022321: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0023200 of size 589824 next 114 2020-04-26 06:22:20.022352: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00b3200 of size 512 next 115 2020-04-26 06:22:20.022381: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00b3400 of size 512 next 116 2020-04-26 06:22:20.022411: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00b3600 of size 512 next 117 2020-04-26 06:22:20.022456: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00b3800 of size 512 next 118 2020-04-26 06:22:20.022477: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00b3a00 of size 512 next 119 2020-04-26 06:22:20.022500: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00b3c00 of size 262144 next 120 2020-04-26 06:22:20.022525: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00f3c00 of size 2048 next 121 2020-04-26 06:22:20.022549: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00f4400 of size 2048 next 122 2020-04-26 06:22:20.022571: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00f4c00 of size 2048 next 123 2020-04-26 06:22:20.022595: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00f5400 of size 2048 next 124 2020-04-26 06:22:20.022618: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00f5c00 of size 2048 next 125 2020-04-26 06:22:20.022643: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea00f6400 of size 262144 next 126 2020-04-26 06:22:20.022664: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0136400 of size 512 next 127 2020-04-26 06:22:20.022686: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0136600 of size 512 next 128 2020-04-26 06:22:20.022707: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0136800 of size 512 next 129 2020-04-26 06:22:20.022731: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0136a00 of size 512 next 130 2020-04-26 06:22:20.022754: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0136c00 of size 512 next 131 2020-04-26 06:22:20.022778: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0136e00 of size 589824 next 132 2020-04-26 06:22:20.022806: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea01c6e00 of size 512 next 133 2020-04-26 06:22:20.022830: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea01c7000 of size 512 next 134 2020-04-26 06:22:20.022852: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea01c7200 of size 512 next 135 2020-04-26 06:22:20.022874: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea01c7400 of size 512 next 136 2020-04-26 06:22:20.022896: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea01c7600 of size 512 next 137 2020-04-26 06:22:20.022918: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea01c7800 of size 262144 next 138 2020-04-26 06:22:20.022939: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0207800 of size 2048 next 139 2020-04-26 06:22:20.022964: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0208000 of size 2048 next 140 2020-04-26 06:22:20.022986: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0208800 of size 2048 next 141 2020-04-26 06:22:20.023010: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0209000 of size 2048 next 142 2020-04-26 06:22:20.023037: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0209800 of size 2048 next 143 2020-04-26 06:22:20.023062: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea020a000 of size 262144 next 144 2020-04-26 06:22:20.023086: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea024a000 of size 512 next 145 2020-04-26 06:22:20.023105: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea024a200 of size 512 next 146 2020-04-26 06:22:20.023127: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea024a400 of size 512 next 147 2020-04-26 06:22:20.023142: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea024a600 of size 512 next 148 2020-04-26 06:22:20.023163: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea024a800 of size 512 next 149 2020-04-26 06:22:20.023182: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea024aa00 of size 589824 next 150 2020-04-26 06:22:20.023201: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea02daa00 of size 512 next 151 2020-04-26 06:22:20.023215: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea02dac00 of size 512 next 152 2020-04-26 06:22:20.023232: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea02dae00 of size 512 next 153 2020-04-26 06:22:20.023255: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea02db000 of size 512 next 154 2020-04-26 06:22:20.023275: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea02db200 of size 512 next 155 2020-04-26 06:22:20.023293: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea02db400 of size 262144 next 156 2020-04-26 06:22:20.023310: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea031b400 of size 2048 next 157 2020-04-26 06:22:20.023333: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea031bc00 of size 2048 next 158 2020-04-26 06:22:20.023349: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea031c400 of size 2048 next 159 2020-04-26 06:22:20.023362: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea031cc00 of size 2048 next 160 2020-04-26 06:22:20.023375: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea031d400 of size 2048 next 161 2020-04-26 06:22:20.023388: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea031dc00 of size 262144 next 162 2020-04-26 06:22:20.023403: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea035dc00 of size 512 next 163 2020-04-26 06:22:20.023423: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea035de00 of size 512 next 164 2020-04-26 06:22:20.023443: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea035e000 of size 512 next 165 2020-04-26 06:22:20.023458: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea035e200 of size 512 next 166 2020-04-26 06:22:20.023478: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea035e400 of size 512 next 167 2020-04-26 06:22:20.023495: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea035e600 of size 589824 next 168 2020-04-26 06:22:20.023523: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea03ee600 of size 512 next 169 2020-04-26 06:22:20.023542: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea03ee800 of size 512 next 170 2020-04-26 06:22:20.023560: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea03eea00 of size 512 next 171 2020-04-26 06:22:20.023577: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea03eec00 of size 512 next 172 2020-04-26 06:22:20.023597: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea03eee00 of size 512 next 173 2020-04-26 06:22:20.023618: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea03ef000 of size 262144 next 174 2020-04-26 06:22:20.023638: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea042f000 of size 2048 next 175 2020-04-26 06:22:20.023654: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea042f800 of size 2048 next 176 2020-04-26 06:22:20.023670: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0430000 of size 2048 next 177 2020-04-26 06:22:20.023689: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0430800 of size 2048 next 178 2020-04-26 06:22:20.023705: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0431000 of size 2048 next 179 2020-04-26 06:22:20.023723: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0431800 of size 2097152 next 180 2020-04-26 06:22:20.023754: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0631800 of size 4096 next 181 2020-04-26 06:22:20.023772: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0632800 of size 4096 next 182 2020-04-26 06:22:20.023790: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0633800 of size 4096 next 183 2020-04-26 06:22:20.023809: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0634800 of size 4096 next 184 2020-04-26 06:22:20.023825: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0635800 of size 4096 next 185 2020-04-26 06:22:20.023843: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0636800 of size 524288 next 186 2020-04-26 06:22:20.023860: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea06b6800 of size 1024 next 187 2020-04-26 06:22:20.023877: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea06b6c00 of size 1024 next 188 2020-04-26 06:22:20.023894: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea06b7000 of size 1024 next 189 2020-04-26 06:22:20.023912: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea06b7400 of size 1024 next 190 2020-04-26 06:22:20.023929: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea06b7800 of size 1024 next 191 2020-04-26 06:22:20.023947: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea06b7c00 of size 2359296 next 192 2020-04-26 06:22:20.023963: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea08f7c00 of size 1024 next 193 2020-04-26 06:22:20.023985: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea08f8000 of size 1024 next 194 2020-04-26 06:22:20.024006: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea08f8400 of size 1024 next 195 2020-04-26 06:22:20.024019: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea08f8800 of size 1024 next 196 2020-04-26 06:22:20.024043: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea08f8c00 of size 1024 next 197 2020-04-26 06:22:20.024049: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea08f9000 of size 1048576 next 198 2020-04-26 06:22:20.024055: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea09f9000 of size 4096 next 199 2020-04-26 06:22:20.024061: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea09fa000 of size 4096 next 200 2020-04-26 06:22:20.024067: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea09fb000 of size 4096 next 201 2020-04-26 06:22:20.024073: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea09fc000 of size 4096 next 202 2020-04-26 06:22:20.024079: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea09fd000 of size 4096 next 203 2020-04-26 06:22:20.024085: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea09fe000 of size 1048576 next 204 2020-04-26 06:22:20.024091: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0afe000 of size 1024 next 205 2020-04-26 06:22:20.024097: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0afe400 of size 1024 next 206 2020-04-26 06:22:20.024103: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0afe800 of size 1024 next 207 2020-04-26 06:22:20.024109: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0afec00 of size 1024 next 208 2020-04-26 06:22:20.024115: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0aff000 of size 1024 next 209 2020-04-26 06:22:20.024122: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0aff400 of size 2359296 next 210 2020-04-26 06:22:20.024128: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0d3f400 of size 1024 next 211 2020-04-26 06:22:20.024134: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0d3f800 of size 1024 next 212 2020-04-26 06:22:20.024140: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0d3fc00 of size 1024 next 213 2020-04-26 06:22:20.024146: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0d40000 of size 1024 next 214 2020-04-26 06:22:20.024152: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0d40400 of size 1024 next 215 2020-04-26 06:22:20.024158: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0d40800 of size 1048576 next 216 2020-04-26 06:22:20.024164: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0e40800 of size 4096 next 217 2020-04-26 06:22:20.024170: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0e41800 of size 4096 next 218 2020-04-26 06:22:20.024176: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0e42800 of size 4096 next 219 2020-04-26 06:22:20.024182: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0e43800 of size 4096 next 220 2020-04-26 06:22:20.024188: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0e44800 of size 4096 next 221 2020-04-26 06:22:20.024194: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0e45800 of size 1048576 next 222 2020-04-26 06:22:20.024200: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0f45800 of size 1024 next 223 2020-04-26 06:22:20.024207: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0f45c00 of size 1024 next 224 2020-04-26 06:22:20.024213: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0f46000 of size 1024 next 225 2020-04-26 06:22:20.024219: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0f46400 of size 1024 next 226 2020-04-26 06:22:20.024225: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0f46800 of size 1024 next 227 2020-04-26 06:22:20.024231: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea0f46c00 of size 2359296 next 228 2020-04-26 06:22:20.024239: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1186c00 of size 1024 next 229 2020-04-26 06:22:20.024250: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1187000 of size 1024 next 230 2020-04-26 06:22:20.024257: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1187400 of size 1024 next 231 2020-04-26 06:22:20.024265: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1187800 of size 1024 next 232 2020-04-26 06:22:20.024277: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1187c00 of size 1024 next 233 2020-04-26 06:22:20.024288: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1188000 of size 1048576 next 234 2020-04-26 06:22:20.024300: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1288000 of size 4096 next 235 2020-04-26 06:22:20.024308: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1289000 of size 4096 next 236 2020-04-26 06:22:20.024316: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea128a000 of size 4096 next 237 2020-04-26 06:22:20.024323: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea128b000 of size 4096 next 238 2020-04-26 06:22:20.024330: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea128c000 of size 4096 next 239 2020-04-26 06:22:20.024338: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea128d000 of size 1048576 next 240 2020-04-26 06:22:20.024345: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea138d000 of size 1024 next 241 2020-04-26 06:22:20.024355: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea138d400 of size 1024 next 242 2020-04-26 06:22:20.024368: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea138d800 of size 1024 next 243 2020-04-26 06:22:20.024381: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea138dc00 of size 1024 next 244 2020-04-26 06:22:20.024394: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea138e000 of size 1024 next 245 2020-04-26 06:22:20.024406: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea138e400 of size 2359296 next 246 2020-04-26 06:22:20.024415: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea15ce400 of size 1024 next 247 2020-04-26 06:22:20.024427: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea15ce800 of size 1024 next 248 2020-04-26 06:22:20.024437: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea15cec00 of size 1024 next 249 2020-04-26 06:22:20.024449: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea15cf000 of size 1024 next 250 2020-04-26 06:22:20.024460: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea15cf400 of size 1024 next 251 2020-04-26 06:22:20.024473: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea15cf800 of size 1048576 next 252 2020-04-26 06:22:20.024484: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea16cf800 of size 4096 next 253 2020-04-26 06:22:20.024495: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea16d0800 of size 4096 next 254 2020-04-26 06:22:20.024505: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea16d1800 of size 4096 next 255 2020-04-26 06:22:20.024516: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea16d2800 of size 4096 next 256 2020-04-26 06:22:20.024526: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea16d3800 of size 4096 next 257 2020-04-26 06:22:20.024537: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea16d4800 of size 1048576 next 258 2020-04-26 06:22:20.024547: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea17d4800 of size 1024 next 259 2020-04-26 06:22:20.024560: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea17d4c00 of size 1024 next 260 2020-04-26 06:22:20.024570: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea17d5000 of size 1024 next 261 2020-04-26 06:22:20.024591: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea17d5400 of size 1024 next 262 2020-04-26 06:22:20.024602: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea17d5800 of size 1024 next 263 2020-04-26 06:22:20.024615: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea17d5c00 of size 2359296 next 264 2020-04-26 06:22:20.024626: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1a15c00 of size 1024 next 265 2020-04-26 06:22:20.024638: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1a16000 of size 1024 next 266 2020-04-26 06:22:20.024647: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1a16400 of size 1024 next 267 2020-04-26 06:22:20.024659: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1a16800 of size 1024 next 268 2020-04-26 06:22:20.024670: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1a16c00 of size 1024 next 269 2020-04-26 06:22:20.024682: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1a17000 of size 1048576 next 270 2020-04-26 06:22:20.024691: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1b17000 of size 4096 next 271 2020-04-26 06:22:20.024703: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1b18000 of size 4096 next 272 2020-04-26 06:22:20.024713: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1b19000 of size 4096 next 273 2020-04-26 06:22:20.024725: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1b1a000 of size 4096 next 274 2020-04-26 06:22:20.024736: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1b1b000 of size 4096 next 275 2020-04-26 06:22:20.024748: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1b1c000 of size 1048576 next 276 2020-04-26 06:22:20.024756: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1c1c000 of size 1024 next 277 2020-04-26 06:22:20.024768: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1c1c400 of size 1024 next 278 2020-04-26 06:22:20.024777: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1c1c800 of size 1024 next 279 2020-04-26 06:22:20.024787: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1c1cc00 of size 1024 next 280 2020-04-26 06:22:20.024795: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1c1d000 of size 1024 next 281 2020-04-26 06:22:20.024803: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1c1d400 of size 2359296 next 282 2020-04-26 06:22:20.024810: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1e5d400 of size 1024 next 283 2020-04-26 06:22:20.024817: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1e5d800 of size 1024 next 284 2020-04-26 06:22:20.024824: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1e5dc00 of size 1024 next 285 2020-04-26 06:22:20.024831: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1e5e000 of size 1024 next 286 2020-04-26 06:22:20.024840: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1e5e400 of size 1024 next 287 2020-04-26 06:22:20.024853: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1e5e800 of size 1048576 next 288 2020-04-26 06:22:20.024864: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1f5e800 of size 4096 next 289 2020-04-26 06:22:20.024878: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1f5f800 of size 4096 next 290 2020-04-26 06:22:20.024892: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1f60800 of size 4096 next 291 2020-04-26 06:22:20.024904: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1f61800 of size 4096 next 292 2020-04-26 06:22:20.024918: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1f62800 of size 4096 next 293 2020-04-26 06:22:20.024933: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea1f63800 of size 8388608 next 294 2020-04-26 06:22:20.024949: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea2763800 of size 8192 next 295 2020-04-26 06:22:20.024967: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea2765800 of size 8192 next 296 2020-04-26 06:22:20.024979: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea2767800 of size 8192 next 297 2020-04-26 06:22:20.024987: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea2769800 of size 8192 next 298 2020-04-26 06:22:20.024994: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea276b800 of size 8192 next 299 2020-04-26 06:22:20.025002: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea276d800 of size 2097152 next 300 2020-04-26 06:22:20.025009: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea296d800 of size 2048 next 301 2020-04-26 06:22:20.025017: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea296e000 of size 2048 next 302 2020-04-26 06:22:20.025024: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea296e800 of size 2048 next 303 2020-04-26 06:22:20.025032: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea296f000 of size 2048 next 304 2020-04-26 06:22:20.025039: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea296f800 of size 2048 next 305 2020-04-26 06:22:20.025048: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea2970000 of size 9437184 next 306 2020-04-26 06:22:20.025056: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3270000 of size 2048 next 307 2020-04-26 06:22:20.025063: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3270800 of size 2048 next 308 2020-04-26 06:22:20.025074: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3271000 of size 2048 next 309 2020-04-26 06:22:20.025087: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3271800 of size 2048 next 310 2020-04-26 06:22:20.025098: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3272000 of size 2048 next 311 2020-04-26 06:22:20.025109: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3272800 of size 4194304 next 312 2020-04-26 06:22:20.025121: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3672800 of size 8192 next 313 2020-04-26 06:22:20.025132: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3674800 of size 8192 next 314 2020-04-26 06:22:20.025145: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3676800 of size 8192 next 315 2020-04-26 06:22:20.025157: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3678800 of size 8192 next 316 2020-04-26 06:22:20.025169: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea367a800 of size 8192 next 317 2020-04-26 06:22:20.025181: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea367c800 of size 4194304 next 318 2020-04-26 06:22:20.025195: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3a7c800 of size 2048 next 319 2020-04-26 06:22:20.025205: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3a7d000 of size 2048 next 320 2020-04-26 06:22:20.025216: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3a7d800 of size 2048 next 321 2020-04-26 06:22:20.025230: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3a7e000 of size 2048 next 322 2020-04-26 06:22:20.025244: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3a7e800 of size 2048 next 323 2020-04-26 06:22:20.025255: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea3a7f000 of size 9437184 next 324 2020-04-26 06:22:20.025267: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea437f000 of size 2048 next 325 2020-04-26 06:22:20.025279: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea437f800 of size 2048 next 326 2020-04-26 06:22:20.025291: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4380000 of size 2048 next 327 2020-04-26 06:22:20.025303: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4380800 of size 2048 next 328 2020-04-26 06:22:20.025316: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4381000 of size 2048 next 329 2020-04-26 06:22:20.025329: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4381800 of size 4194304 next 330 2020-04-26 06:22:20.025345: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4781800 of size 8192 next 331 2020-04-26 06:22:20.025355: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4783800 of size 8192 next 332 2020-04-26 06:22:20.025369: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4785800 of size 8192 next 333 2020-04-26 06:22:20.025384: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4787800 of size 8192 next 334 2020-04-26 06:22:20.025396: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4789800 of size 8192 next 335 2020-04-26 06:22:20.025404: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea478b800 of size 4194304 next 336 2020-04-26 06:22:20.025412: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4b8b800 of size 2048 next 337 2020-04-26 06:22:20.025420: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4b8c000 of size 2048 next 338 2020-04-26 06:22:20.025432: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4b8c800 of size 2048 next 339 2020-04-26 06:22:20.025444: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4b8d000 of size 2048 next 340 2020-04-26 06:22:20.025457: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4b8d800 of size 2048 next 341 2020-04-26 06:22:20.025468: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea4b8e000 of size 9437184 next 342 2020-04-26 06:22:20.025480: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea548e000 of size 2048 next 343 2020-04-26 06:22:20.025491: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea548e800 of size 2048 next 344 2020-04-26 06:22:20.025503: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea548f000 of size 2048 next 345 2020-04-26 06:22:20.025514: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea548f800 of size 2048 next 346 2020-04-26 06:22:20.025526: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea5490000 of size 2048 next 347 2020-04-26 06:22:20.025541: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea5490800 of size 4194304 next 348 2020-04-26 06:22:20.025554: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea5890800 of size 8192 next 349 2020-04-26 06:22:20.025566: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea5892800 of size 8192 next 350 2020-04-26 06:22:20.025577: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea5894800 of size 8192 next 351 2020-04-26 06:22:20.025589: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea5896800 of size 8192 next 352 2020-04-26 06:22:20.025600: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea5898800 of size 8192 next 353 2020-04-26 06:22:20.025612: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea589a800 of size 122880 next 354 2020-04-26 06:22:20.025624: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea58b8800 of size 256 next 355 2020-04-26 06:22:20.025636: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea58b8900 of size 256 next 356 2020-04-26 06:22:20.025649: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea58b8a00 of size 256 next 357 2020-04-26 06:22:20.025661: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea58b8b00 of size 256 next 358 2020-04-26 06:22:20.025674: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea58b8c00 of size 256 next 359 2020-04-26 06:22:20.025685: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea58b8d00 of size 256 next 360 2020-04-26 06:22:20.025695: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea58b8e00 of size 4194304 next 361 2020-04-26 06:22:20.025704: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea5cb8e00 of size 4194304 next 362 2020-04-26 06:22:20.025714: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60b8e00 of size 8192 next 363 2020-04-26 06:22:20.025723: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60bae00 of size 8192 next 364 2020-04-26 06:22:20.025732: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60bce00 of size 8192 next 365 2020-04-26 06:22:20.025744: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60bee00 of size 8192 next 366 2020-04-26 06:22:20.025756: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60c0e00 of size 256 next 367 2020-04-26 06:22:20.025769: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60c0f00 of size 256 next 368 2020-04-26 06:22:20.025781: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60c1000 of size 256 next 369 2020-04-26 06:22:20.025793: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60c1100 of size 256 next 370 2020-04-26 06:22:20.025805: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60c1200 of size 65536 next 371 2020-04-26 06:22:20.025817: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60d1200 of size 65536 next 372 2020-04-26 06:22:20.025830: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea60e1200 of size 147456 next 373 2020-04-26 06:22:20.025843: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6105200 of size 147456 next 374 2020-04-26 06:22:20.025857: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6129200 of size 16384 next 375 2020-04-26 06:22:20.025870: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea612d200 of size 16384 next 376 2020-04-26 06:22:20.025883: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6131200 of size 1024 next 377 2020-04-26 06:22:20.025896: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6131600 of size 1024 next 378 2020-04-26 06:22:20.025909: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6131a00 of size 131072 next 379 2020-04-26 06:22:20.025921: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6151a00 of size 131072 next 380 2020-04-26 06:22:20.025933: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6171a00 of size 65536 next 381 2020-04-26 06:22:20.025946: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6181a00 of size 65536 next 382 2020-04-26 06:22:20.025958: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6191a00 of size 122880 next 383 2020-04-26 06:22:20.025970: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61afa00 of size 122880 next 384 2020-04-26 06:22:20.025983: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61cda00 of size 512 next 385 2020-04-26 06:22:20.025997: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61cdc00 of size 512 next 386 2020-04-26 06:22:20.026010: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61cde00 of size 65536 next 387 2020-04-26 06:22:20.026024: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61dde00 of size 65536 next 388 2020-04-26 06:22:20.026032: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61ede00 of size 1024 next 389 2020-04-26 06:22:20.026040: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61ee200 of size 1024 next 390 2020-04-26 06:22:20.026047: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61ee600 of size 8192 next 391 2020-04-26 06:22:20.026057: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61f0600 of size 8192 next 392 2020-04-26 06:22:20.026068: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61f2600 of size 256 next 393 2020-04-26 06:22:20.026079: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61f2700 of size 256 next 394 2020-04-26 06:22:20.026090: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61f2800 of size 256 next 395 2020-04-26 06:22:20.026101: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61f2900 of size 256 next 396 2020-04-26 06:22:20.026112: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61f2a00 of size 2048 next 397 2020-04-26 06:22:20.026123: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61f3200 of size 2048 next 398 2020-04-26 06:22:20.026135: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea61f3a00 of size 589824 next 399 2020-04-26 06:22:20.026146: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6283a00 of size 589824 next 400 2020-04-26 06:22:20.026157: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6313a00 of size 256 next 401 2020-04-26 06:22:20.026169: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6313b00 of size 256 next 402 2020-04-26 06:22:20.026179: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6313c00 of size 256 next 403 2020-04-26 06:22:20.026191: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6313d00 of size 256 next 404 2020-04-26 06:22:20.026202: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6313e00 of size 256 next 405 2020-04-26 06:22:20.026214: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6313f00 of size 256 next 406 2020-04-26 06:22:20.026226: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6314000 of size 256 next 407 2020-04-26 06:22:20.026237: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6314100 of size 256 next 408 2020-04-26 06:22:20.026249: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6314200 of size 512 next 409 2020-04-26 06:22:20.026260: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6314400 of size 512 next 410 2020-04-26 06:22:20.026271: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6314600 of size 2048 next 411 2020-04-26 06:22:20.026282: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6314e00 of size 2048 next 412 2020-04-26 06:22:20.026293: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6315600 of size 256 next 413 2020-04-26 06:22:20.026305: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6315700 of size 256 next 414 2020-04-26 06:22:20.026317: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6315800 of size 256 next 415 2020-04-26 06:22:20.026328: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6315900 of size 256 next 416 2020-04-26 06:22:20.026339: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6315a00 of size 1024 next 417 2020-04-26 06:22:20.026351: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6315e00 of size 1024 next 418 2020-04-26 06:22:20.026362: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6316200 of size 512 next 419 2020-04-26 06:22:20.026374: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6316400 of size 512 next 420 2020-04-26 06:22:20.026386: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6316600 of size 147456 next 421 2020-04-26 06:22:20.026398: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea633a600 of size 147456 next 422 2020-04-26 06:22:20.026409: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea635e600 of size 262144 next 423 2020-04-26 06:22:20.026420: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea639e600 of size 262144 next 424 2020-04-26 06:22:20.026431: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63de600 of size 256 next 425 2020-04-26 06:22:20.026442: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63de700 of size 256 next 426 2020-04-26 06:22:20.026453: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63de800 of size 256 next 427 2020-04-26 06:22:20.026466: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63de900 of size 256 next 428 2020-04-26 06:22:20.026478: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63dea00 of size 256 next 429 2020-04-26 06:22:20.026489: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63deb00 of size 256 next 430 2020-04-26 06:22:20.026500: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63dec00 of size 1024 next 431 2020-04-26 06:22:20.026511: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63df000 of size 1024 next 432 2020-04-26 06:22:20.026522: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63df400 of size 512 next 433 2020-04-26 06:22:20.026533: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63df600 of size 512 next 434 2020-04-26 06:22:20.026546: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63df800 of size 256 next 435 2020-04-26 06:22:20.026558: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63df900 of size 256 next 436 2020-04-26 06:22:20.026569: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63dfa00 of size 256 next 437 2020-04-26 06:22:20.026580: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63dfb00 of size 256 next 438 2020-04-26 06:22:20.026588: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea63dfc00 of size 262144 next 439 2020-04-26 06:22:20.026597: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea641fc00 of size 262144 next 440 2020-04-26 06:22:20.026604: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea645fc00 of size 1024 next 441 2020-04-26 06:22:20.026611: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6460000 of size 1024 next 442 2020-04-26 06:22:20.026619: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6460400 of size 512 next 443 2020-04-26 06:22:20.026626: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6460600 of size 512 next 444 2020-04-26 06:22:20.026636: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6460800 of size 256 next 445 2020-04-26 06:22:20.026648: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6460900 of size 256 next 446 2020-04-26 06:22:20.026660: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6460a00 of size 512 next 447 2020-04-26 06:22:20.026671: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6460c00 of size 512 next 448 2020-04-26 06:22:20.026683: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6460e00 of size 589824 next 449 2020-04-26 06:22:20.026694: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea64f0e00 of size 589824 next 450 2020-04-26 06:22:20.026707: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6580e00 of size 2048 next 451 2020-04-26 06:22:20.026718: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6581600 of size 2048 next 452 2020-04-26 06:22:20.026730: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6581e00 of size 1024 next 453 2020-04-26 06:22:20.026741: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6582200 of size 1024 next 454 2020-04-26 06:22:20.026753: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6582600 of size 512 next 455 2020-04-26 06:22:20.026764: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6582800 of size 512 next 456 2020-04-26 06:22:20.026776: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6582a00 of size 2048 next 457 2020-04-26 06:22:20.026787: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6583200 of size 2048 next 458 2020-04-26 06:22:20.026799: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6583a00 of size 512 next 459 2020-04-26 06:22:20.026810: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6583c00 of size 512 next 460 2020-04-26 06:22:20.026822: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6583e00 of size 512 next 461 2020-04-26 06:22:20.026833: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6584000 of size 512 next 462 2020-04-26 06:22:20.026845: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6584200 of size 147456 next 463 2020-04-26 06:22:20.026858: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65a8200 of size 147456 next 464 2020-04-26 06:22:20.026870: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cc200 of size 512 next 465 2020-04-26 06:22:20.026883: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cc400 of size 512 next 466 2020-04-26 06:22:20.026895: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cc600 of size 2048 next 467 2020-04-26 06:22:20.026908: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cce00 of size 2048 next 468 2020-04-26 06:22:20.026921: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cd600 of size 256 next 469 2020-04-26 06:22:20.026933: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cd700 of size 256 next 470 2020-04-26 06:22:20.026943: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cd800 of size 512 next 471 2020-04-26 06:22:20.026955: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cda00 of size 512 next 472 2020-04-26 06:22:20.026966: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cdc00 of size 2048 next 473 2020-04-26 06:22:20.026979: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65ce400 of size 2048 next 474 2020-04-26 06:22:20.026989: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cec00 of size 512 next 475 2020-04-26 06:22:20.027000: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cee00 of size 512 next 476 2020-04-26 06:22:20.027012: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cf000 of size 512 next 477 2020-04-26 06:22:20.027025: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cf200 of size 512 next 478 2020-04-26 06:22:20.027034: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea65cf400 of size 262144 next 479 2020-04-26 06:22:20.027045: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea660f400 of size 262144 next 480 2020-04-26 06:22:20.027057: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea664f400 of size 262144 next 481 2020-04-26 06:22:20.027067: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea668f400 of size 262144 next 482 2020-04-26 06:22:20.027089: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea66cf400 of size 589824 next 483 2020-04-26 06:22:20.027100: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea675f400 of size 589824 next 484 2020-04-26 06:22:20.027113: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea67ef400 of size 262144 next 485 2020-04-26 06:22:20.027122: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea682f400 of size 262144 next 486 2020-04-26 06:22:20.027133: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea686f400 of size 512 next 487 2020-04-26 06:22:20.027145: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea686f600 of size 512 next 488 2020-04-26 06:22:20.027153: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea686f800 of size 512 next 489 2020-04-26 06:22:20.027164: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea686fa00 of size 512 next 490 2020-04-26 06:22:20.027176: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea686fc00 of size 512 next 491 2020-04-26 06:22:20.027185: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea686fe00 of size 512 next 492 2020-04-26 06:22:20.027197: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6870000 of size 512 next 493 2020-04-26 06:22:20.027209: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6870200 of size 512 next 494 2020-04-26 06:22:20.027221: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6870400 of size 512 next 495 2020-04-26 06:22:20.027232: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6870600 of size 512 next 496 2020-04-26 06:22:20.027242: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6870800 of size 524288 next 497 2020-04-26 06:22:20.027254: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea68f0800 of size 524288 next 498 2020-04-26 06:22:20.027266: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6970800 of size 512 next 499 2020-04-26 06:22:20.027279: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6970a00 of size 512 next 500 2020-04-26 06:22:20.027289: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6970c00 of size 262144 next 501 2020-04-26 06:22:20.027301: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea69b0c00 of size 262144 next 502 2020-04-26 06:22:20.027312: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea69f0c00 of size 2048 next 503 2020-04-26 06:22:20.027324: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea69f1400 of size 2048 next 504 2020-04-26 06:22:20.027335: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea69f1c00 of size 512 next 505 2020-04-26 06:22:20.027347: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea69f1e00 of size 512 next 506 2020-04-26 06:22:20.027358: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea69f2000 of size 2048 next 507 2020-04-26 06:22:20.027371: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea69f2800 of size 2048 next 508 2020-04-26 06:22:20.027381: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea69f3000 of size 589824 next 509 2020-04-26 06:22:20.027393: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6a83000 of size 589824 next 510 2020-04-26 06:22:20.027405: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b13000 of size 2048 next 511 2020-04-26 06:22:20.027417: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b13800 of size 2048 next 512 2020-04-26 06:22:20.027428: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b14000 of size 1024 next 513 2020-04-26 06:22:20.027440: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b14400 of size 1024 next 514 2020-04-26 06:22:20.027451: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b14800 of size 262144 next 515 2020-04-26 06:22:20.027463: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b54800 of size 262144 next 516 2020-04-26 06:22:20.027474: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b94800 of size 2048 next 517 2020-04-26 06:22:20.027486: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b95000 of size 2048 next 518 2020-04-26 06:22:20.027497: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b95800 of size 256 next 519 2020-04-26 06:22:20.027509: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b95900 of size 256 next 520 2020-04-26 06:22:20.027522: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b95a00 of size 37632 next 521 2020-04-26 06:22:20.027534: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6b9ed00 of size 37632 next 522 2020-04-26 06:22:20.027546: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6ba8000 of size 256 next 523 2020-04-26 06:22:20.027558: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6ba8100 of size 256 next 524 2020-04-26 06:22:20.027571: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6ba8200 of size 1024 next 525 2020-04-26 06:22:20.027583: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6ba8600 of size 1024 next 526 2020-04-26 06:22:20.027593: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6ba8a00 of size 4096 next 527 2020-04-26 06:22:20.027605: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6ba9a00 of size 4096 next 528 2020-04-26 06:22:20.027616: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6baaa00 of size 4096 next 529 2020-04-26 06:22:20.027628: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6baba00 of size 4096 next 530 2020-04-26 06:22:20.027640: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6baca00 of size 2359296 next 531 2020-04-26 06:22:20.027652: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea6deca00 of size 2359296 next 532 2020-04-26 06:22:20.027664: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea702ca00 of size 1024 next 533 2020-04-26 06:22:20.027677: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea702ce00 of size 1024 next 534 2020-04-26 06:22:20.027689: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea702d200 of size 1048576 next 535 2020-04-26 06:22:20.027702: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea712d200 of size 1048576 next 536 2020-04-26 06:22:20.027713: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea722d200 of size 2359296 next 537 2020-04-26 06:22:20.027725: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea746d200 of size 2359296 next 538 2020-04-26 06:22:20.027737: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea76ad200 of size 1048576 next 539 2020-04-26 06:22:20.027749: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea77ad200 of size 1048576 next 540 2020-04-26 06:22:20.027759: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78ad200 of size 4096 next 541 2020-04-26 06:22:20.027770: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78ae200 of size 4096 next 542 2020-04-26 06:22:20.027782: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78af200 of size 1024 next 543 2020-04-26 06:22:20.027794: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78af600 of size 1024 next 544 2020-04-26 06:22:20.027806: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78afa00 of size 65536 next 545 2020-04-26 06:22:20.027818: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78bfa00 of size 65536 next 546 2020-04-26 06:22:20.027830: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78cfa00 of size 4096 next 547 2020-04-26 06:22:20.027842: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78d0a00 of size 4096 next 548 2020-04-26 06:22:20.027854: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78d1a00 of size 1024 next 549 2020-04-26 06:22:20.027866: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78d1e00 of size 1024 next 550 2020-04-26 06:22:20.027877: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78d2200 of size 1024 next 551 2020-04-26 06:22:20.027888: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78d2600 of size 1024 next 552 2020-04-26 06:22:20.027899: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78d2a00 of size 1024 next 553 2020-04-26 06:22:20.027912: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78d2e00 of size 1024 next 554 2020-04-26 06:22:20.027921: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78d3200 of size 1024 next 555 2020-04-26 06:22:20.027933: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78d3600 of size 1024 next 556 2020-04-26 06:22:20.027946: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea78d3a00 of size 1048576 next 557 2020-04-26 06:22:20.027958: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea79d3a00 of size 1048576 next 558 2020-04-26 06:22:20.027971: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7ad3a00 of size 1024 next 559 2020-04-26 06:22:20.027983: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7ad3e00 of size 1024 next 560 2020-04-26 06:22:20.027994: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7ad4200 of size 4096 next 561 2020-04-26 06:22:20.028005: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7ad5200 of size 4096 next 562 2020-04-26 06:22:20.028016: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7ad6200 of size 2359296 next 563 2020-04-26 06:22:20.028029: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7d16200 of size 2359296 next 564 2020-04-26 06:22:20.028039: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f56200 of size 2048 next 565 2020-04-26 06:22:20.028051: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f56a00 of size 2048 next 566 2020-04-26 06:22:20.028062: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f57200 of size 1024 next 567 2020-04-26 06:22:20.028075: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f57600 of size 1024 next 568 2020-04-26 06:22:20.028087: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f57a00 of size 8192 next 569 2020-04-26 06:22:20.028099: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f59a00 of size 8192 next 570 2020-04-26 06:22:20.028109: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f5ba00 of size 1024 next 571 2020-04-26 06:22:20.028121: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f5be00 of size 1024 next 572 2020-04-26 06:22:20.028133: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f5c200 of size 4096 next 573 2020-04-26 06:22:20.028145: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f5d200 of size 4096 next 574 2020-04-26 06:22:20.028157: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f5e200 of size 2048 next 575 2020-04-26 06:22:20.028170: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f5ea00 of size 2048 next 576 2020-04-26 06:22:20.028182: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f5f200 of size 8192 next 577 2020-04-26 06:22:20.028194: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f61200 of size 8192 next 578 2020-04-26 06:22:20.028205: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea7f63200 of size 1048576 next 579 2020-04-26 06:22:20.028217: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea8063200 of size 1048576 next 580 2020-04-26 06:22:20.028228: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea8163200 of size 8192 next 581 2020-04-26 06:22:20.028240: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea8165200 of size 8192 next 582 2020-04-26 06:22:20.028251: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea8167200 of size 4096 next 583 2020-04-26 06:22:20.028263: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea8168200 of size 4096 next 584 2020-04-26 06:22:20.028273: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea8169200 of size 2048 next 585 2020-04-26 06:22:20.028285: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea8169a00 of size 2048 next 586 2020-04-26 06:22:20.028298: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea816a200 of size 8388608 next 587 2020-04-26 06:22:20.028311: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea896a200 of size 8388608 next 588 2020-04-26 06:22:20.028322: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea916a200 of size 8192 next 589 2020-04-26 06:22:20.028335: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea916c200 of size 8192 next 590 2020-04-26 06:22:20.028345: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea916e200 of size 4194304 next 591 2020-04-26 06:22:20.028357: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea956e200 of size 4194304 next 592 2020-04-26 06:22:20.028368: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea996e200 of size 2048 next 593 2020-04-26 06:22:20.028381: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea996ea00 of size 2048 next 594 2020-04-26 06:22:20.028392: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea996f200 of size 2048 next 595 2020-04-26 06:22:20.028404: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea996fa00 of size 2048 next 596 2020-04-26 06:22:20.028414: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9970200 of size 1024 next 597 2020-04-26 06:22:20.028426: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9970600 of size 1024 next 598 2020-04-26 06:22:20.028438: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9970a00 of size 8192 next 599 2020-04-26 06:22:20.028450: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9972a00 of size 8192 next 600 2020-04-26 06:22:20.028460: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9974a00 of size 1024 next 601 2020-04-26 06:22:20.028473: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9974e00 of size 1024 next 602 2020-04-26 06:22:20.028483: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9975200 of size 1024 next 603 2020-04-26 06:22:20.028495: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9975600 of size 1024 next 604 2020-04-26 06:22:20.028507: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9975a00 of size 65536 next 605 2020-04-26 06:22:20.028519: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9985a00 of size 65536 next 606 2020-04-26 06:22:20.028530: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9995a00 of size 1024 next 607 2020-04-26 06:22:20.028543: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9995e00 of size 1024 next 608 2020-04-26 06:22:20.028553: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9996200 of size 8192 next 609 2020-04-26 06:22:20.028565: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea9998200 of size 8192 next 610 2020-04-26 06:22:20.028576: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faea999a200 of size 9437184 next 611 2020-04-26 06:22:20.028589: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaa29a200 of size 9437184 next 612 2020-04-26 06:22:20.028600: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaab9a200 of size 2048 next 613 2020-04-26 06:22:20.028612: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaab9aa00 of size 2048 next 614 2020-04-26 06:22:20.028623: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaab9b200 of size 2097152 next 615 2020-04-26 06:22:20.028635: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaad9b200 of size 2097152 next 616 2020-04-26 06:22:20.028646: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaaf9b200 of size 1024 next 617 2020-04-26 06:22:20.028659: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaaf9b600 of size 1024 next 618 2020-04-26 06:22:20.028669: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaaf9ba00 of size 1024 next 619 2020-04-26 06:22:20.028680: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaaf9be00 of size 1024 next 620 2020-04-26 06:22:20.028692: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaaf9c200 of size 1024 next 621 2020-04-26 06:22:20.028704: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaaf9c600 of size 1024 next 622 2020-04-26 06:22:20.028716: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaaf9ca00 of size 4194304 next 623 2020-04-26 06:22:20.028726: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab39ca00 of size 4194304 next 624 2020-04-26 06:22:20.028737: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab79ca00 of size 8192 next 625 2020-04-26 06:22:20.028748: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab79ea00 of size 8192 next 626 2020-04-26 06:22:20.028759: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab7a0a00 of size 1024 next 627 2020-04-26 06:22:20.028768: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab7a0e00 of size 1024 next 628 2020-04-26 06:22:20.028775: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab7a1200 of size 2048 next 629 2020-04-26 06:22:20.028782: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab7a1a00 of size 2048 next 630 2020-04-26 06:22:20.028790: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab7a2200 of size 1048576 next 631 2020-04-26 06:22:20.028797: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab8a2200 of size 1048576 next 632 2020-04-26 06:22:20.028805: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab9a2200 of size 1024 next 633 2020-04-26 06:22:20.028815: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab9a2600 of size 1024 next 634 2020-04-26 06:22:20.028827: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab9a2a00 of size 1024 next 635 2020-04-26 06:22:20.028840: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab9a2e00 of size 1024 next 636 2020-04-26 06:22:20.028852: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab9a3200 of size 256 next 637 2020-04-26 06:22:20.028864: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab9a3300 of size 256 next 638 2020-04-26 06:22:20.028874: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeab9a3400 of size 1048576 next 639 2020-04-26 06:22:20.028886: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabaa3400 of size 1048576 next 640 2020-04-26 06:22:20.028898: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabba3400 of size 1024 next 641 2020-04-26 06:22:20.028911: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabba3800 of size 1024 next 642 2020-04-26 06:22:20.028921: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabba3c00 of size 1048576 next 643 2020-04-26 06:22:20.028933: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabca3c00 of size 1048576 next 644 2020-04-26 06:22:20.028944: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabda3c00 of size 4096 next 645 2020-04-26 06:22:20.028956: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabda4c00 of size 4096 next 646 2020-04-26 06:22:20.028968: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabda5c00 of size 2048 next 647 2020-04-26 06:22:20.028980: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabda6400 of size 2048 next 648 2020-04-26 06:22:20.028990: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabda6c00 of size 8192 next 649 2020-04-26 06:22:20.029002: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabda8c00 of size 8192 next 650 2020-04-26 06:22:20.029012: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeabdaac00 of size 4194304 next 651 2020-04-26 06:22:20.029026: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac1aac00 of size 4194304 next 652 2020-04-26 06:22:20.029038: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5aac00 of size 256 next 653 2020-04-26 06:22:20.029051: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5aad00 of size 256 next 654 2020-04-26 06:22:20.029062: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5aae00 of size 1024 next 655 2020-04-26 06:22:20.029074: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5ab200 of size 1024 next 656 2020-04-26 06:22:20.029085: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5ab600 of size 4096 next 657 2020-04-26 06:22:20.029098: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5ac600 of size 4096 next 658 2020-04-26 06:22:20.029108: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5ad600 of size 4096 next 659 2020-04-26 06:22:20.029120: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5ae600 of size 4096 next 660 2020-04-26 06:22:20.029131: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5af600 of size 1024 next 661 2020-04-26 06:22:20.029143: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5afa00 of size 1024 next 662 2020-04-26 06:22:20.029154: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5afe00 of size 2048 next 663 2020-04-26 06:22:20.029166: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b0600 of size 2048 next 664 2020-04-26 06:22:20.029177: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b0e00 of size 2048 next 665 2020-04-26 06:22:20.029189: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b1600 of size 2048 next 666 2020-04-26 06:22:20.029199: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b1e00 of size 256 next 667 2020-04-26 06:22:20.029211: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b1f00 of size 256 next 668 2020-04-26 06:22:20.029221: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b2000 of size 2048 next 669 2020-04-26 06:22:20.029233: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b2800 of size 2048 next 670 2020-04-26 06:22:20.029244: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b3000 of size 2048 next 671 2020-04-26 06:22:20.029256: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b3800 of size 2048 next 672 2020-04-26 06:22:20.029268: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b4000 of size 4096 next 673 2020-04-26 06:22:20.029277: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b5000 of size 4096 next 674 2020-04-26 06:22:20.029289: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b6000 of size 1024 next 675 2020-04-26 06:22:20.029301: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b6400 of size 1024 next 676 2020-04-26 06:22:20.029313: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac5b6800 of size 2359296 next 677 2020-04-26 06:22:20.029323: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeac7f6800 of size 2359296 next 678 2020-04-26 06:22:20.029335: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaca36800 of size 1024 next 679 2020-04-26 06:22:20.029345: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaca36c00 of size 1024 next 680 2020-04-26 06:22:20.029357: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaca37000 of size 1048576 next 681 2020-04-26 06:22:20.029367: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeacb37000 of size 1048576 next 682 2020-04-26 06:22:20.029379: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeacc37000 of size 4096 next 683 2020-04-26 06:22:20.029389: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeacc38000 of size 4096 next 684 2020-04-26 06:22:20.029401: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeacc39000 of size 4194304 next 685 2020-04-26 06:22:20.029411: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead039000 of size 4194304 next 686 2020-04-26 06:22:20.029423: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead439000 of size 65536 next 687 2020-04-26 06:22:20.029433: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead449000 of size 65536 next 688 2020-04-26 06:22:20.029446: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead459000 of size 2048 next 689 2020-04-26 06:22:20.029455: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead459800 of size 2048 next 690 2020-04-26 06:22:20.029467: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead45a000 of size 2048 next 691 2020-04-26 06:22:20.029478: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead45a800 of size 2048 next 692 2020-04-26 06:22:20.029490: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead45b000 of size 1024 next 693 2020-04-26 06:22:20.029501: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead45b400 of size 1024 next 694 2020-04-26 06:22:20.029513: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead45b800 of size 1024 next 695 2020-04-26 06:22:20.029524: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead45bc00 of size 1024 next 696 2020-04-26 06:22:20.029537: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faead45c000 of size 9437184 next 697 2020-04-26 06:22:20.029548: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeadd5c000 of size 9437184 next 698 2020-04-26 06:22:20.029560: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeae65c000 of size 1024 next 699 2020-04-26 06:22:20.029571: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeae65c400 of size 1024 next 700 2020-04-26 06:22:20.029583: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeae65c800 of size 2359296 next 701 2020-04-26 06:22:20.029595: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeae89c800 of size 2359296 next 702 2020-04-26 06:22:20.029607: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaeadc800 of size 1048576 next 703 2020-04-26 06:22:20.029617: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaebdc800 of size 1048576 next 704 2020-04-26 06:22:20.029629: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaecdc800 of size 9437184 next 705 2020-04-26 06:22:20.029642: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeaf5dc800 of size 10631168 next 18446744073709551615 2020-04-26 06:22:20.029657: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 268435456 2020-04-26 06:22:20.029670: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7faeb0000000 of size 268435456 next 18446744073709551615 2020-04-26 06:22:20.029682: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 1048576 2020-04-26 06:22:20.029691: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00000 of size 256 next 68 2020-04-26 06:22:20.029703: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00100 of size 256 next 2 2020-04-26 06:22:20.029715: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00200 of size 256 next 3 2020-04-26 06:22:20.029727: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00300 of size 256 next 4 2020-04-26 06:22:20.029737: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00400 of size 256 next 5 2020-04-26 06:22:20.029748: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00500 of size 256 next 6 2020-04-26 06:22:20.029760: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00600 of size 256 next 7 2020-04-26 06:22:20.029773: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00700 of size 256 next 8 2020-04-26 06:22:20.029782: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00800 of size 256 next 9 2020-04-26 06:22:20.029794: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00900 of size 256 next 10 2020-04-26 06:22:20.029806: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00a00 of size 256 next 11 2020-04-26 06:22:20.029818: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00b00 of size 256 next 12 2020-04-26 06:22:20.029827: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00c00 of size 256 next 13 2020-04-26 06:22:20.029839: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00d00 of size 256 next 14 2020-04-26 06:22:20.029851: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00e00 of size 256 next 15 2020-04-26 06:22:20.029863: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c00f00 of size 256 next 16 2020-04-26 06:22:20.029872: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01000 of size 256 next 17 2020-04-26 06:22:20.029884: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01100 of size 256 next 18 2020-04-26 06:22:20.029896: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01200 of size 256 next 19 2020-04-26 06:22:20.029908: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01300 of size 256 next 20 2020-04-26 06:22:20.029918: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01400 of size 256 next 21 2020-04-26 06:22:20.029929: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01500 of size 256 next 22 2020-04-26 06:22:20.029941: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01600 of size 256 next 23 2020-04-26 06:22:20.029953: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01700 of size 256 next 24 2020-04-26 06:22:20.029963: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01800 of size 256 next 25 2020-04-26 06:22:20.029975: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01900 of size 256 next 26 2020-04-26 06:22:20.029986: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01a00 of size 256 next 27 2020-04-26 06:22:20.029999: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01b00 of size 256 next 28 2020-04-26 06:22:20.030008: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01c00 of size 256 next 29 2020-04-26 06:22:20.030020: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01d00 of size 256 next 30 2020-04-26 06:22:20.030032: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01e00 of size 256 next 31 2020-04-26 06:22:20.030044: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c01f00 of size 256 next 32 2020-04-26 06:22:20.030053: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02000 of size 256 next 33 2020-04-26 06:22:20.030065: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02100 of size 256 next 34 2020-04-26 06:22:20.030077: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02200 of size 256 next 35 2020-04-26 06:22:20.030089: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02300 of size 256 next 36 2020-04-26 06:22:20.030099: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02400 of size 256 next 37 2020-04-26 06:22:20.030110: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02500 of size 256 next 38 2020-04-26 06:22:20.030122: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02600 of size 256 next 39 2020-04-26 06:22:20.030134: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02700 of size 256 next 40 2020-04-26 06:22:20.030147: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02800 of size 256 next 41 2020-04-26 06:22:20.030160: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02900 of size 256 next 42 2020-04-26 06:22:20.030169: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02a00 of size 256 next 43 2020-04-26 06:22:20.030181: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02b00 of size 256 next 44 2020-04-26 06:22:20.030193: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02c00 of size 256 next 45 2020-04-26 06:22:20.030205: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02d00 of size 256 next 46 2020-04-26 06:22:20.030214: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02e00 of size 256 next 47 2020-04-26 06:22:20.030226: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c02f00 of size 256 next 48 2020-04-26 06:22:20.030238: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03000 of size 256 next 49 2020-04-26 06:22:20.030250: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03100 of size 256 next 50 2020-04-26 06:22:20.030260: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03200 of size 256 next 51 2020-04-26 06:22:20.030271: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03300 of size 256 next 52 2020-04-26 06:22:20.030284: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03400 of size 256 next 53 2020-04-26 06:22:20.030296: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03500 of size 256 next 54 2020-04-26 06:22:20.030307: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03600 of size 256 next 55 2020-04-26 06:22:20.030320: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03700 of size 256 next 56 2020-04-26 06:22:20.030329: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03800 of size 256 next 57 2020-04-26 06:22:20.030341: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03900 of size 256 next 58 2020-04-26 06:22:20.030351: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03a00 of size 256 next 59 2020-04-26 06:22:20.030363: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03b00 of size 256 next 60 2020-04-26 06:22:20.030374: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03c00 of size 256 next 61 2020-04-26 06:22:20.030386: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03d00 of size 256 next 62 2020-04-26 06:22:20.030397: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03e00 of size 256 next 63 2020-04-26 06:22:20.030417: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c03f00 of size 256 next 64 2020-04-26 06:22:20.030428: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c04000 of size 256 next 65 2020-04-26 06:22:20.030439: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c04100 of size 256 next 66 2020-04-26 06:22:20.030451: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c04200 of size 256 next 67 2020-04-26 06:22:20.030462: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c04300 of size 256 next 69 2020-04-26 06:22:20.030475: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c04400 of size 256 next 70 2020-04-26 06:22:20.030485: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c04500 of size 256 next 71 2020-04-26 06:22:20.030497: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c04600 of size 147456 next 72 2020-04-26 06:22:20.030509: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c28600 of size 256 next 73 2020-04-26 06:22:20.030521: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c28700 of size 256 next 74 2020-04-26 06:22:20.030531: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c28800 of size 256 next 75 2020-04-26 06:22:20.030543: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c28900 of size 256 next 76 2020-04-26 06:22:20.030553: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c28a00 of size 256 next 77 2020-04-26 06:22:20.030566: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c28b00 of size 65536 next 78 2020-04-26 06:22:20.030577: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c38b00 of size 1024 next 79 2020-04-26 06:22:20.030589: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c38f00 of size 1024 next 80 2020-04-26 06:22:20.030599: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c39300 of size 1024 next 81 2020-04-26 06:22:20.030611: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c39700 of size 1024 next 82 2020-04-26 06:22:20.030622: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c39b00 of size 1024 next 83 2020-04-26 06:22:20.030634: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c39f00 of size 65536 next 84 2020-04-26 06:22:20.030645: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c49f00 of size 256 next 85 2020-04-26 06:22:20.030656: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c4a000 of size 256 next 86 2020-04-26 06:22:20.030677: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c4a100 of size 256 next 87 2020-04-26 06:22:20.030689: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c4a200 of size 256 next 88 2020-04-26 06:22:20.030698: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c4a300 of size 256 next 89 2020-04-26 06:22:20.030710: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c4a400 of size 147456 next 90 2020-04-26 06:22:20.030722: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c6e400 of size 256 next 91 2020-04-26 06:22:20.030735: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c6e500 of size 256 next 92 2020-04-26 06:22:20.030744: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c6e600 of size 256 next 93 2020-04-26 06:22:20.030756: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c6e700 of size 256 next 94 2020-04-26 06:22:20.030766: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c6e800 of size 256 next 95 2020-04-26 06:22:20.030778: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c6e900 of size 65536 next 96 2020-04-26 06:22:20.030790: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c7e900 of size 1024 next 97 2020-04-26 06:22:20.030803: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c7ed00 of size 1024 next 98 2020-04-26 06:22:20.030812: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c7f100 of size 1024 next 99 2020-04-26 06:22:20.030825: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c7f500 of size 1024 next 100 2020-04-26 06:22:20.030834: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c7f900 of size 1024 next 101 2020-04-26 06:22:20.030847: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse at 7fb086c7fd00 of size 525056 next 18446744073709551615 2020-04-26 06:22:20.030856: I tensorflow/core/common_runtime/bfc_allocator.cc:1349] Summary of in-use Chunks by size: 2020-04-26 06:22:20.030875: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 162 Chunks of size 256 totalling 40.5KiB 2020-04-26 06:22:20.030890: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 104 Chunks of size 512 totalling 52.0KiB 2020-04-26 06:22:20.030903: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 182 Chunks of size 1024 totalling 182.0KiB 2020-04-26 06:22:20.030917: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 129 Chunks of size 2048 totalling 258.0KiB 2020-04-26 06:22:20.030929: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 77 Chunks of size 4096 totalling 308.0KiB 2020-04-26 06:22:20.030942: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 44 Chunks of size 8192 totalling 352.0KiB 2020-04-26 06:22:20.030954: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 16384 totalling 48.0KiB 2020-04-26 06:22:20.030967: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 37632 totalling 110.2KiB 2020-04-26 06:22:20.030979: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 18 Chunks of size 65536 totalling 1.12MiB 2020-04-26 06:22:20.030993: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 122880 totalling 360.0KiB 2020-04-26 06:22:20.031004: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 131072 totalling 384.0KiB 2020-04-26 06:22:20.031017: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 9 Chunks of size 147456 totalling 1.27MiB 2020-04-26 06:22:20.031028: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 21 Chunks of size 262144 totalling 5.25MiB 2020-04-26 06:22:20.031041: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 5 Chunks of size 524288 totalling 2.50MiB 2020-04-26 06:22:20.031053: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 525056 totalling 512.8KiB 2020-04-26 06:22:20.031065: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 12 Chunks of size 589824 totalling 6.75MiB 2020-04-26 06:22:20.031078: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 33 Chunks of size 1048576 totalling 33.00MiB 2020-04-26 06:22:20.031091: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 6 Chunks of size 2097152 totalling 12.00MiB 2020-04-26 06:22:20.031102: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 18 Chunks of size 2359296 totalling 40.50MiB 2020-04-26 06:22:20.031115: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 15 Chunks of size 4194304 totalling 60.00MiB 2020-04-26 06:22:20.031127: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 8388608 totalling 24.00MiB 2020-04-26 06:22:20.031140: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 8 Chunks of size 9437184 totalling 72.00MiB 2020-04-26 06:22:20.031151: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 10631168 totalling 10.14MiB 2020-04-26 06:22:20.031164: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 16 Chunks of size 32000000 totalling 488.28MiB 2020-04-26 06:22:20.031178: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 48288512 totalling 46.05MiB 2020-04-26 06:22:20.031191: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 11 Chunks of size 64000000 totalling 671.39MiB 2020-04-26 06:22:20.031201: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 90140928 totalling 85.96MiB 2020-04-26 06:22:20.031215: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 112428544 totalling 107.22MiB 2020-04-26 06:22:20.031227: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 5 Chunks of size 128000000 totalling 610.35MiB 2020-04-26 06:22:20.031240: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 227472384 totalling 216.93MiB 2020-04-26 06:22:20.031252: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 8 Chunks of size 256000000 totalling 1.91GiB 2020-04-26 06:22:20.031265: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 257025024 totalling 245.12MiB 2020-04-26 06:22:20.031276: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 268435456 totalling 256.00MiB 2020-04-26 06:22:20.031289: I tensorflow/core/common_runtime/bfc_allocator.cc:1356] Sum Total of in-use chunks: 4.83GiB 2020-04-26 06:22:20.031300: I tensorflow/core/common_runtime/bfc_allocator.cc:1358] total_region_allocated_bytes_: 5278566400 memory_limit_: 734003200000 available bytes: 728724633600 curr_region_allocation_bytes_: 8589934592 2020-04-26 06:22:20.031810: I tensorflow/core/common_runtime/bfc_allocator.cc:1364] Stats: Limit: 734003200000 InUse: 5192034304 MaxInUse: 5192034304 NumAllocs: 920 MaxAllocSize: 268435456 BytesInactive: 0 BytesActive: 5192034304 PeakBytesActive: 5192034304 BytesReclaimed: 0 NumSingleReclaims: 0 NumFullReclaims: 0 NumDefrags: 0 BytesDefragged: 0 2020-04-26 06:22:20.032459: W tensorflow/core/common_runtime/bfc_allocator.cc:806] *****************_****************************************x***************************************** 2020-04-26 06:22:20.032705: W tensorflow/core/common_runtime/bfc_allocator.cc:809] Enabling Large Model Support may avoid this failure. 2020-04-26 06:22:20.052779: F tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:153] Check failed: result.ok() Aborted (core dumped) (wmlce) pwrai@9ffaf92a2cb3:/app$ ```
smatzek commented 4 years ago

This behavior is due to the way TensorFlow's GPU host allocator is implemented, and this is not something that LMS added or changed.

The GPU host allocator uses allocates system memory through CUDA and makes it be pinned memory. This pinned memory is "pinned" to RAM and is ineligible for operating system swapping to a swap space. The main reason TensorFlow (and other CUDA users) allocate memory in this fashion is that it provides a large performance benefit when moving memory between system RAM and GPU memory.

Unfortunately this also means that the memory that can be used for tensor swapping is limited to physical memory and that physical memory is also needed for the operating system itself so on systems with a low system to GPU memory ratio the amount of swapping possible becomes reduced further.