search

rpautrat / SuperPoint

Efficient neural feature detector and descriptor
MIT License
1.88k stars 416 forks source link

Out Of Memory (OOM) when finetuning SuperPoint #191

Closed jack-turkey closed 3 years ago

jack-turkey commented 3 years ago

The finetuning of SP throws this error when training with my dataset of 4000 images. The batch sizes have been set to 1. The labels have been already generated using Magic-Point. How can I solve this issue? Thanks in advance

/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) experiment.py:152: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. config = yaml.load(f) [02/26/2021 12:30:49 INFO] Running command TRAIN [02/26/2021 12:30:49 INFO] Number of GPUs detected: 1 2021-02-26 12:30:53.418537: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2021-02-26 12:30:53.552840: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-02-26 12:30:53.553072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705 pciBusID: 0000:01:00.0 totalMemory: 5.94GiB freeMemory: 5.38GiB 2021-02-26 12:30:53.553085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2021-02-26 12:30:53.758206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-02-26 12:30:53.758231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2021-02-26 12:30:53.758236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2021-02-26 12:30:53.758284: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 2021-02-26 12:30:53.758310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5140 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1) [02/26/2021 12:30:53 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:53 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:54 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. [02/26/2021 12:30:55 INFO] Scale of 0 disables regularizer. 2021-02-26 12:30:55.961390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2021-02-26 12:30:55.961418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-02-26 12:30:55.961423: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2021-02-26 12:30:55.961427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2021-02-26 12:30:55.961475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5140 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1) [02/26/2021 12:30:57 INFO] Restoring parameters from /home/small_body/Data/SuperPoint-master/superpoint/EXPER_PATH/saved_model/sp_v6/model.ckpt-300000 [02/26/2021 12:30:57 INFO] Start training 2021-02-26 12:30:58.689401: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x55e1ceee5d30 2021-02-26 12:30:59.602234: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2021-02-26 12:30:59.733808: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2021-02-26 12:30:59.836894: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2021-02-26 12:31:09.878319: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.44GiB. Current allocation summary follows. 2021-02-26 12:31:09.878462: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 61, Chunks in use: 59. 15.2KiB allocated for chunks. 14.8KiB in use in bin. 5.1KiB client-requested in use in bin. 2021-02-26 12:31:09.878524: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 23, Chunks in use: 20. 11.8KiB allocated for chunks. 10.2KiB in use in bin. 10.0KiB client-requested in use in bin. 2021-02-26 12:31:09.878570: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 17, Chunks in use: 13. 18.8KiB allocated for chunks. 14.8KiB in use in bin. 14.1KiB client-requested in use in bin. 2021-02-26 12:31:09.878625: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 2, Chunks in use: 0. 5.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-02-26 12:31:09.878673: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 5, Chunks in use: 4. 31.2KiB allocated for chunks. 25.0KiB in use in bin. 25.0KiB client-requested in use in bin. 2021-02-26 12:31:09.878719: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 1, Chunks in use: 0. 9.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-02-26 12:31:09.878762: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 1, Chunks in use: 0. 18.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-02-26 12:31:09.878805: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-02-26 12:31:09.878848: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 1, Chunks in use: 0. 65.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-02-26 12:31:09.878905: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 3, Chunks in use: 3. 432.0KiB allocated for chunks. 432.0KiB in use in bin. 432.0KiB client-requested in use in bin. 2021-02-26 12:31:09.878961: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 4, Chunks in use: 3. 1.26MiB allocated for chunks. 967.0KiB in use in bin. 944.0KiB client-requested in use in bin. 2021-02-26 12:31:09.879009: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 17, Chunks in use: 15. 12.35MiB allocated for chunks. 11.01MiB in use in bin. 10.67MiB client-requested in use in bin. 2021-02-26 12:31:09.879056: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 13, Chunks in use: 11. 19.22MiB allocated for chunks. 16.97MiB in use in bin. 16.41MiB client-requested in use in bin. 2021-02-26 12:31:09.879097: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 10, Chunks in use: 10. 29.62MiB allocated for chunks. 29.62MiB in use in bin. 26.12MiB client-requested in use in bin. 2021-02-26 12:31:09.879149: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 11, Chunks in use: 10. 67.76MiB allocated for chunks. 62.30MiB in use in bin. 59.38MiB client-requested in use in bin. 2021-02-26 12:31:09.879190: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 1, Chunks in use: 1. 10.30MiB allocated for chunks. 10.30MiB in use in bin. 6.25MiB client-requested in use in bin. 2021-02-26 12:31:09.879237: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 5, Chunks in use: 5. 125.00MiB allocated for chunks. 125.00MiB in use in bin. 125.00MiB client-requested in use in bin. 2021-02-26 12:31:09.879279: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 3, Chunks in use: 3. 103.00MiB allocated for chunks. 103.00MiB in use in bin. 75.00MiB client-requested in use in bin. 2021-02-26 12:31:09.879321: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 1, Chunks in use: 0. 122.48MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-02-26 12:31:09.879355: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-02-26 12:31:09.879393: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 2, Chunks in use: 0. 2.66GiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-02-26 12:31:09.879427: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 2.44GiB was 256.00MiB, Chunk State: 2021-02-26 12:31:09.879477: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 675.41MiB | Requested Size: 139.23MiB | in_use: 0, prev: Size: 1.56MiB | Requested Size: 1.56MiB | in_use: 1 2021-02-26 12:31:09.879514: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 2.00GiB | Requested Size: 1.04GiB | in_use: 0 2021-02-26 12:31:09.879546: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8b70000000 of size 2147483648 2021-02-26 12:31:09.879613: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf0000000 of size 26214400 2021-02-26 12:31:09.879649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf1900000 of size 26214400 2021-02-26 12:31:09.879694: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf3200000 of size 26214400 2021-02-26 12:31:09.879715: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf4b00000 of size 26214400 2021-02-26 12:31:09.879732: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf6400000 of size 6553600 2021-02-26 12:31:09.879749: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf6a40000 of size 6553600 2021-02-26 12:31:09.879766: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf7080000 of size 6553600 2021-02-26 12:31:09.879781: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf76c0000 of size 6553600 2021-02-26 12:31:09.879797: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf7d00000 of size 1638400 2021-02-26 12:31:09.879814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf7e90000 of size 10797056 2021-02-26 12:31:09.879830: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf88dc000 of size 6553600 2021-02-26 12:31:09.879846: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf8f1c000 of size 6553600 2021-02-26 12:31:09.879863: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf955c000 of size 3571712 2021-02-26 12:31:09.879880: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf98c4000 of size 3276800 2021-02-26 12:31:09.879897: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bf9be4000 of size 6553600 2021-02-26 12:31:09.879914: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfa224000 of size 819712 2021-02-26 12:31:09.879930: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfa2ec200 of size 1638400 2021-02-26 12:31:09.879945: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfa47c200 of size 3276800 2021-02-26 12:31:09.879961: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfa79c200 of size 3276800 2021-02-26 12:31:09.879979: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfaabc200 of size 5733888 2021-02-26 12:31:09.879994: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfb034000 of size 6553600 2021-02-26 12:31:09.880010: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfb674000 of size 3276800 2021-02-26 12:31:09.880025: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfb994000 of size 3276800 2021-02-26 12:31:09.880042: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfbcb4000 of size 819200 2021-02-26 12:31:09.880057: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfbd7c000 of size 3276800 2021-02-26 12:31:09.880073: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8bfc09c000 of size 589824 2021-02-26 12:31:09.880089: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfc12c000 of size 819200 2021-02-26 12:31:09.880104: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfc1f4000 of size 819200 2021-02-26 12:31:09.880120: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfc2bc000 of size 1409024 2021-02-26 12:31:09.880136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfc414000 of size 819200 2021-02-26 12:31:09.880152: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfc4dc000 of size 2818048 2021-02-26 12:31:09.880168: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfc78c000 of size 2916352 2021-02-26 12:31:09.880184: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfca54000 of size 819200 2021-02-26 12:31:09.880199: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfcb1c000 of size 819200 2021-02-26 12:31:09.880215: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8bfcbe4000 of size 819200 2021-02-26 12:31:09.880230: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfccac000 of size 819200 2021-02-26 12:31:09.880246: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfcd74000 of size 1639424 2021-02-26 12:31:09.880262: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfcf04400 of size 819200 2021-02-26 12:31:09.880278: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8bfcfcc400 of size 1179648 2021-02-26 12:31:09.880293: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfd0ec400 of size 819200 2021-02-26 12:31:09.880309: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8bfd1b4400 of size 5734400 2021-02-26 12:31:09.880325: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfd72c400 of size 1639424 2021-02-26 12:31:09.880341: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfd8bc800 of size 1638400 2021-02-26 12:31:09.880356: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfda4c800 of size 1638400 2021-02-26 12:31:09.880371: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfdbdc800 of size 1638400 2021-02-26 12:31:09.880387: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfdd6c800 of size 1638400 2021-02-26 12:31:09.880403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8bfdefc800 of size 1638400 2021-02-26 12:31:09.880418: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8bfe08c800 of size 128433152 2021-02-26 12:31:09.880434: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8c05b08400 of size 1638400 2021-02-26 12:31:09.880450: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8c05c98400 of size 708213760 2021-02-26 12:31:09.880466: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8c64000000 of size 26214400 2021-02-26 12:31:09.880483: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8c65900000 of size 40894464 2021-02-26 12:31:09.880499: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8c6e000000 of size 33554432 2021-02-26 12:31:09.880517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8c86000000 of size 33554432 2021-02-26 12:31:09.880533: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600000 of size 1280 2021-02-26 12:31:09.880549: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600500 of size 256 2021-02-26 12:31:09.880565: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600600 of size 256 2021-02-26 12:31:09.880581: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600700 of size 256 2021-02-26 12:31:09.880596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600800 of size 256 2021-02-26 12:31:09.880612: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600900 of size 256 2021-02-26 12:31:09.880628: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600a00 of size 256 2021-02-26 12:31:09.880644: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600b00 of size 256 2021-02-26 12:31:09.880660: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600c00 of size 256 2021-02-26 12:31:09.880675: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600d00 of size 256 2021-02-26 12:31:09.880691: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600e00 of size 256 2021-02-26 12:31:09.880706: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde600f00 of size 256 2021-02-26 12:31:09.880722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601000 of size 256 2021-02-26 12:31:09.880738: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601100 of size 256 2021-02-26 12:31:09.880753: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601200 of size 256 2021-02-26 12:31:09.880769: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601300 of size 256 2021-02-26 12:31:09.880784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601400 of size 256 2021-02-26 12:31:09.880800: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601500 of size 256 2021-02-26 12:31:09.880815: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601600 of size 256 2021-02-26 12:31:09.880830: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601700 of size 256 2021-02-26 12:31:09.880846: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601800 of size 256 2021-02-26 12:31:09.880861: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601900 of size 256 2021-02-26 12:31:09.880877: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601a00 of size 256 2021-02-26 12:31:09.880892: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601b00 of size 256 2021-02-26 12:31:09.880908: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601c00 of size 256 2021-02-26 12:31:09.880923: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601d00 of size 256 2021-02-26 12:31:09.880939: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601e00 of size 256 2021-02-26 12:31:09.880954: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde601f00 of size 256 2021-02-26 12:31:09.880971: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602000 of size 256 2021-02-26 12:31:09.880988: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602100 of size 256 2021-02-26 12:31:09.881004: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602200 of size 256 2021-02-26 12:31:09.881020: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602300 of size 256 2021-02-26 12:31:09.881036: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602400 of size 256 2021-02-26 12:31:09.881051: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602500 of size 256 2021-02-26 12:31:09.881067: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602600 of size 256 2021-02-26 12:31:09.881082: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602700 of size 256 2021-02-26 12:31:09.881097: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602800 of size 256 2021-02-26 12:31:09.881113: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602900 of size 256 2021-02-26 12:31:09.881128: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602a00 of size 256 2021-02-26 12:31:09.881143: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602b00 of size 256 2021-02-26 12:31:09.881159: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602c00 of size 256 2021-02-26 12:31:09.881174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602d00 of size 256 2021-02-26 12:31:09.881190: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cde602e00 of size 256 2021-02-26 12:31:09.881205: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde602f00 of size 256 2021-02-26 12:31:09.881221: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde603000 of size 256 2021-02-26 12:31:09.881237: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde603100 of size 256 2021-02-26 12:31:09.881252: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cde603200 of size 256 2021-02-26 12:31:09.881268: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde603300 of size 147456 2021-02-26 12:31:09.881284: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde627300 of size 256 2021-02-26 12:31:09.881300: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde627400 of size 256 2021-02-26 12:31:09.881315: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cde627500 of size 512 2021-02-26 12:31:09.881331: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde627700 of size 256 2021-02-26 12:31:09.881347: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde627800 of size 256 2021-02-26 12:31:09.881363: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde627900 of size 147456 2021-02-26 12:31:09.881378: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde64b900 of size 256 2021-02-26 12:31:09.881395: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde64ba00 of size 294912 2021-02-26 12:31:09.881410: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde693a00 of size 147456 2021-02-26 12:31:09.881426: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b7a00 of size 512 2021-02-26 12:31:09.881442: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b7c00 of size 256 2021-02-26 12:31:09.881458: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b7d00 of size 256 2021-02-26 12:31:09.881473: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b7e00 of size 512 2021-02-26 12:31:09.881489: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cde6b8000 of size 1024 2021-02-26 12:31:09.881506: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b8400 of size 1024 2021-02-26 12:31:09.881522: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cde6b8800 of size 1024 2021-02-26 12:31:09.881537: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b8c00 of size 512 2021-02-26 12:31:09.881553: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b8e00 of size 512 2021-02-26 12:31:09.881569: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b9000 of size 512 2021-02-26 12:31:09.881585: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b9200 of size 512 2021-02-26 12:31:09.881600: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cde6b9400 of size 512 2021-02-26 12:31:09.881616: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b9600 of size 512 2021-02-26 12:31:09.881631: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6b9800 of size 1024 2021-02-26 12:31:09.881647: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cde6b9c00 of size 1024 2021-02-26 12:31:09.881662: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6ba000 of size 512 2021-02-26 12:31:09.881678: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6ba200 of size 512 2021-02-26 12:31:09.881694: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cde6ba400 of size 285696 2021-02-26 12:31:09.881711: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdea00000 of size 2097152 2021-02-26 12:31:09.881727: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cdec00000 of size 1179648 2021-02-26 12:31:09.881743: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cded20000 of size 589824 2021-02-26 12:31:09.881759: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdedb0000 of size 589824 2021-02-26 12:31:09.881775: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdee40000 of size 1024 2021-02-26 12:31:09.881791: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdee40400 of size 256 2021-02-26 12:31:09.881807: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cdee40500 of size 66560 2021-02-26 12:31:09.881822: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdee50900 of size 1024 2021-02-26 12:31:09.881838: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdee50d00 of size 512 2021-02-26 12:31:09.881854: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdee50f00 of size 589824 2021-02-26 12:31:09.881869: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cdeee0f00 of size 1024 2021-02-26 12:31:09.881885: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee1300 of size 1024 2021-02-26 12:31:09.881901: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cdeee1700 of size 2048 2021-02-26 12:31:09.881916: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee1f00 of size 256 2021-02-26 12:31:09.881933: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee2000 of size 256 2021-02-26 12:31:09.881950: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee2100 of size 256 2021-02-26 12:31:09.881966: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee2200 of size 256 2021-02-26 12:31:09.881982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee2300 of size 256 2021-02-26 12:31:09.881997: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee2400 of size 256 2021-02-26 12:31:09.882013: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee2500 of size 512 2021-02-26 12:31:09.882028: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee2700 of size 512 2021-02-26 12:31:09.882044: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee2900 of size 512 2021-02-26 12:31:09.882059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee2b00 of size 512 2021-02-26 12:31:09.882075: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cdeee2d00 of size 512 2021-02-26 12:31:09.882090: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee2f00 of size 512 2021-02-26 12:31:09.882106: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee3100 of size 512 2021-02-26 12:31:09.882121: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee3300 of size 512 2021-02-26 12:31:09.882137: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee3500 of size 768 2021-02-26 12:31:09.882153: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee3800 of size 6400 2021-02-26 12:31:09.882169: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cdeee5100 of size 19200 2021-02-26 12:31:09.882185: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeee9c00 of size 1792 2021-02-26 12:31:09.882200: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeeea300 of size 1792 2021-02-26 12:31:09.882216: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cdeeeaa00 of size 9216 2021-02-26 12:31:09.882231: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeeece00 of size 256 2021-02-26 12:31:09.882246: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeeecf00 of size 512 2021-02-26 12:31:09.882261: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeeed100 of size 512 2021-02-26 12:31:09.882277: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeeed300 of size 1024 2021-02-26 12:31:09.882292: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeeed700 of size 1024 2021-02-26 12:31:09.882307: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeeedb00 of size 1024 2021-02-26 12:31:09.882323: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeeedf00 of size 1024 2021-02-26 12:31:09.882338: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeeee300 of size 1024 2021-02-26 12:31:09.882353: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cdeeee700 of size 3072 2021-02-26 12:31:09.882368: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeeef300 of size 6400 2021-02-26 12:31:09.882383: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cdeef0c00 of size 6400 2021-02-26 12:31:09.882399: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeef2500 of size 6400 2021-02-26 12:31:09.882414: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdeef3e00 of size 6400 2021-02-26 12:31:09.882430: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7f8cdeef5700 of size 329728 2021-02-26 12:31:09.882454: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdef45f00 of size 762112 2021-02-26 12:31:09.882479: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdf000000 of size 819200 2021-02-26 12:31:09.882505: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdf0c8000 of size 409600 2021-02-26 12:31:09.882530: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f8cdf12c000 of size 7159808 2021-02-26 12:31:09.882545: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size: 2021-02-26 12:31:09.882568: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 59 Chunks of size 256 totalling 14.8KiB 2021-02-26 12:31:09.882586: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 19 Chunks of size 512 totalling 9.5KiB 2021-02-26 12:31:09.882603: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 768 totalling 768B 2021-02-26 12:31:09.882622: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 1024 totalling 10.0KiB 2021-02-26 12:31:09.882640: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB 2021-02-26 12:31:09.882659: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 1792 totalling 3.5KiB 2021-02-26 12:31:09.882677: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 6400 totalling 25.0KiB 2021-02-26 12:31:09.882695: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 147456 totalling 432.0KiB 2021-02-26 12:31:09.882713: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 285696 totalling 279.0KiB 2021-02-26 12:31:09.882731: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB 2021-02-26 12:31:09.882748: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 409600 totalling 400.0KiB 2021-02-26 12:31:09.882766: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 589824 totalling 1.69MiB 2021-02-26 12:31:09.882784: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 762112 totalling 744.2KiB 2021-02-26 12:31:09.882801: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 819200 totalling 7.81MiB 2021-02-26 12:31:09.882819: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 819712 totalling 800.5KiB 2021-02-26 12:31:09.882836: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1409024 totalling 1.34MiB 2021-02-26 12:31:09.882853: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 8 Chunks of size 1638400 totalling 12.50MiB 2021-02-26 12:31:09.882870: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 1639424 totalling 3.13MiB 2021-02-26 12:31:09.882887: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2097152 totalling 2.00MiB 2021-02-26 12:31:09.882904: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2818048 totalling 2.69MiB 2021-02-26 12:31:09.882921: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2916352 totalling 2.78MiB 2021-02-26 12:31:09.882939: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 3276800 totalling 18.75MiB 2021-02-26 12:31:09.882956: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3571712 totalling 3.41MiB 2021-02-26 12:31:09.882973: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 5733888 totalling 5.47MiB 2021-02-26 12:31:09.882991: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 8 Chunks of size 6553600 totalling 50.00MiB 2021-02-26 12:31:09.883007: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 7159808 totalling 6.83MiB 2021-02-26 12:31:09.883026: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 10797056 totalling 10.30MiB 2021-02-26 12:31:09.883043: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 26214400 totalling 125.00MiB 2021-02-26 12:31:09.883061: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 33554432 totalling 64.00MiB 2021-02-26 12:31:09.883078: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 40894464 totalling 39.00MiB 2021-02-26 12:31:09.883097: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 359.63MiB 2021-02-26 12:31:09.883120: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit: 5389811712 InUse: 377096192 MaxInUse: 2168218624 NumAllocs: 556 MaxAllocSize: 1888026624

2021-02-26 12:31:09.883167: W tensorflow/core/common_runtime/bfc_allocator.cc:271] ____*****____**** 2021-02-26 12:31:09.883211: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[1,40,40,40,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(args) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,40,40,40,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node superpoint/train_tower0/mul_3}} = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](superpoint/train_tower0/l2_normalize, superpoint/train_tower0/l2_normalize_1)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[{{node superpoint/train_tower0/gradients/AddN_30/_453}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2591_superpoint/train_tower0/gradients/AddN_30", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "experiment.py", line 159, in args.func(config, output_dir, args) File "experiment.py", line 96, in _cli_train train(config, config['train_iter'], output_dir, pretrained_dir) File "experiment.py", line 29, in train keep_checkpoints=config.get('keep_checkpoints', 1)) File "/home/small_body/Data/SuperPoint-master/superpoint/models/base_model.py", line 313, in train options=options, run_metadata=run_metadata) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,40,40,40,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node superpoint/train_tower0/mul_3 (defined at /home/small_body/Data/SuperPoint-master/superpoint/models/utils.py:109) = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](superpoint/train_tower0/l2_normalize, superpoint/train_tower0/l2_normalize_1)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[{{node superpoint/train_tower0/gradients/AddN_30/_453}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2591_superpoint/train_tower0/gradients/AddN_30", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'superpoint/train_tower0/mul_3', defined at: File "experiment.py", line 159, in args.func(config, output_dir, args) File "experiment.py", line 96, in _cli_train train(config, config['train_iter'], output_dir, pretrained_dir) File "experiment.py", line 21, in train with _init_graph(config) as net: File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/contextlib.py", line 81, in enter return next(self.gen) File "experiment.py", line 73, in _init_graph n_gpus=n_gpus, config['model']) File "/home/small_body/Data/SuperPoint-master/superpoint/models/base_model.py", line 122, in init self._build_graph() File "/home/small_body/Data/SuperPoint-master/superpoint/models/base_model.py", line 264, in _build_graph self._train_graph(data) File "/home/small_body/Data/SuperPoint-master/superpoint/models/base_model.py", line 188, in _train_graph data, Mode.TRAIN, self.config['batch_size']) File "/home/small_body/Data/SuperPoint-master/superpoint/models/base_model.py", line 159, in _gpu_tower loss = self._loss(net_outputs, shards[i], self.config) File "/home/small_body/Data/SuperPoint-master/superpoint/models/super_point.py", line 82, in _loss valid_mask=inputs['warped']['valid_mask'], config) File "/home/small_body/Data/SuperPoint-master/superpoint/models/utils.py", line 109, in descriptor_loss dot_product_desc = tf.reduce_sum(descriptors warped_descriptors, -1) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 866, in binary_op_wrapper return func(x, y, name=name) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1131, in _mul_dispatch return gen_math_ops.mul(x, y, name=name) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5042, in mul "Mul", x=x, y=y, name=name) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(args, kwargs) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/home/small_body/anaconda3/envs/condasp/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,40,40,40,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node superpoint/train_tower0/mul_3 (defined at /home/small_body/Data/SuperPoint-master/superpoint/models/utils.py:109) = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](superpoint/train_tower0/l2_normalize, superpoint/train_tower0/l2_normalize_1)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[{{node superpoint/train_tower0/gradients/AddN_30/_453}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2591_superpoint/train_tower0/gradients/AddN_30", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

rpautrat commented 3 years ago

Hi, I think that the memory of your GPU is unfortunately too small (5 Gb), even with a batch size of 1. In my case I was only able to get a batch size of 2 or 3 with a GPU memory of 11 Gb.

What you can do is to keep a batch size of 1 and to reduce the dimension of the input images. You can do it by setting the parameter data->preprocessing->resize = [h, w] in the config file. By default, the code uses images of size [240, 320], so you can reduce it a bit until it fits into your GPU memory. I recommend using multiples of 8 in the image size to avoid rounding in the downsampling layers.

jack-turkey commented 3 years ago

Thanks a lot! Now it works