search

XuyangBai / D3Feat

[TensorFlow] Official implementation of CVPR'20 oral paper - D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features https://arxiv.org/abs/2003.03164
MIT License
259 stars 38 forks source link

How to test ETH? #14

Closed Gilgamesh666666 closed 4 years ago

Gilgamesh666666 commented 4 years ago

Hi, Xuyang

Sorry for the bother! I am trying to the ETH dataset using the model which trained on 3dmatch, but the following error raise. So i wonder if that need more space to test ETH, or there are some settings i ignored?

/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)])

Dataset Preparation

(50367, 3) (54203, 3) (53067, 3) (48749, 3) (44246, 3) (42854, 3) (41144, 3) (38663, 3) (39982, 3) (38772, 3) (35781, 3) (33488, 3) (30180, 3) (28480, 3) (26907, 3) (30962, 3) (30314, 3) (30571, 3) (31442, 3) (29981, 3) (36504, 3) (38673, 3) (38908, 3) (43512, 3) (43254, 3) (43394, 3) (46688, 3) (47217, 3) (49071, 3) (55600, 3) (56516, 3) (53873, 3) (44141, 3) (45239, 3) (45198, 3) (44476, 3) (43106, 3) (32730, 3) (36993, 3) (37667, 3) (40342, 3) (39887, 3) (37221, 3) (35227, 3) (35270, 3) (39027, 3) (40245, 3) (35224, 3) (32548, 3) (38184, 3) (39914, 3) (41074, 3) (35561, 3) (37025, 3) (40676, 3) (42016, 3) (43244, 3) (45002, 3) (45876, 3) (47714, 3) (48683, 3) (49047, 3) (48733, 3) (55752, 3) (56517, 3) (55095, 3) (53169, 3) (51016, 3) (51420, 3) (52332, 3) (50786, 3) (50527, 3) (47891, 3) (51851, 3) (56069, 3) (57539, 3) (54436, 3) (52746, 3) (52892, 3) (53423, 3) (54421, 3) (53736, 3) (52290, 3) (51320, 3) (51047, 3) (50506, 3) (49948, 3) (46921, 3) (46470, 3) (48085, 3) (49550, 3) (51027, 3) (49111, 3) (49417, 3) (49074, 3) (53985, 3) (55507, 3) (53216, 3) (49678, 3) (52591, 3) (60122, 3) (58153, 3) (55480, 3) (61990, 3) (67667, 3) (62600, 3) (62394, 3) (62576, 3) (62745, 3) (61567, 3) (58630, 3) (56838, 3) (56258, 3) (55453, 3) (52196, 3) (52221, 3) (51866, 3) (49033, 3) (47441, 3) (47629, 3) (48152, 3) (49553, 3) (50461, 3) (49966, 3) (50028, 3) (49642, 3) (50830, 3) (51745, 3) (52989, 3) (52877, 3) (51941, 3) (52474, 3) Initiating test input pipelines WARNING:tensorflow:From /home/Gilgamesh/D3Feat/datasets/common.py:1308: calling reduce_min (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead 2020-08-01 18:18:22.394755: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2020-08-01 18:18:22.663084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582 pciBusID: 0000:04:00.0 totalMemory: 11.91GiB freeMemory: 11.76GiB 2020-08-01 18:18:22.916220: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582 pciBusID: 0000:08:00.0 totalMemory: 11.91GiB freeMemory: 11.76GiB 2020-08-01 18:18:22.920787: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1 2020-08-01 18:18:24.278918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-08-01 18:18:24.278990: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1 2020-08-01 18:18:24.279008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N Y 2020-08-01 18:18:24.279028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: Y N 2020-08-01 18:18:24.285207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11376 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:04:00.0, compute capability: 6.1) 2020-08-01 18:18:24.285817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 11376 MB memory) -> physical GPU (device: 1, name: TITAN Xp, pci bus id: 0000:08:00.0, compute capability: 6.1) [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131] Calib Neighbors 00000000 : timings 3178.73 385.08 Calib Neighbors 00000001 : timings 2902.80 279.60 Calib Neighbors 00000002 : timings 2865.64 658.51 Calib Neighbors 00000003 : timings 2657.15 247.62 Calib Neighbors 00000004 : timings 2587.97 143.16

self.neighborhood: [ 66 117 144 152 169] Creating Model

WARNING:tensorflow:From /home/Gilgamesh/D3Feat/models/D3Feat.py:111: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20. Instructions for updating: Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:

    sess = tf.Session()
    with sess.as_default():
        tensor = tf.range(10)
        print_op = tf.print(tensor)
        with tf.control_dependencies([print_op]):
          out = tf.add(tensor, tensor)
        sess.run(out)

Additionally, to use tf.print in python 2.7, users must make sure to import the following:

from __future__ import print_function

2020-08-01 18:18:49.254766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1 2020-08-01 18:18:49.254975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-08-01 18:18:49.255005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1 2020-08-01 18:18:49.255047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N Y 2020-08-01 18:18:49.255068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: Y N 2020-08-01 18:18:49.256962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11376 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:04:00.0, compute capability: 6.1) 2020-08-01 18:18:49.257232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 11376 MB memory) -> physical GPU (device: 1, name: TITAN Xp, pci bus id: 0000:08:00.0, compute capability: 6.1) Model restored from results_kitti/Log_11011605/snapshots/snap-61

Done in 4.8 s

Start Test

[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131] num of local max[33906] Generate cloud_bin_0 for gazebo_summer

num of local max[35646] Generate cloud_bin_1 for gazebo_summer

num of local max[30748] Generate cloud_bin_2 for gazebo_summer

num of local max[27088] Generate cloud_bin_3 for gazebo_summer

num of local max[24068] Generate cloud_bin_4 for gazebo_summer

num of local max[24182] Generate cloud_bin_5 for gazebo_summer

num of local max[24650] Generate cloud_bin_6 for gazebo_summer

num of local max[22276] Generate cloud_bin_7 for gazebo_summer

num of local max[22546] Generate cloud_bin_8 for gazebo_summer

num of local max[22662] Generate cloud_bin_9 for gazebo_summer

num of local max[20822] Generate cloud_bin_10 for gazebo_summer

num of local max[19382] Generate cloud_bin_11 for gazebo_summer

num of local max[19734] Generate cloud_bin_12 for gazebo_summer

num of local max[18154] Generate cloud_bin_13 for gazebo_summer

num of local max[19532] Generate cloud_bin_14 for gazebo_summer

num of local max[22790] Generate cloud_bin_15 for gazebo_summer

num of local max[21532] Generate cloud_bin_16 for gazebo_summer

num of local max[20368] Generate cloud_bin_17 for gazebo_summer

num of local max[24498] Generate cloud_bin_18 for gazebo_summer

num of local max[25424] Generate cloud_bin_19 for gazebo_summer

num of local max[29964] Generate cloud_bin_20 for gazebo_summer

num of local max[31310] Generate cloud_bin_21 for gazebo_summer

num of local max[35868] Generate cloud_bin_22 for gazebo_summer

num of local max[38020] Generate cloud_bin_23 for gazebo_summer

num of local max[36818] Generate cloud_bin_24 for gazebo_summer

num of local max[35306] Generate cloud_bin_25 for gazebo_summer

num of local max[36596] Generate cloud_bin_26 for gazebo_summer

num of local max[39436] Generate cloud_bin_27 for gazebo_summer

num of local max[39752] Generate cloud_bin_28 for gazebo_summer

num of local max[46806] Generate cloud_bin_29 for gazebo_summer

num of local max[49566] Generate cloud_bin_30 for gazebo_summer

num of local max[45802] Generate cloud_bin_31 for gazebo_summer

num of local max[57398] Generate cloud_bin_0 for gazebo_winter

num of local max[55134] Generate cloud_bin_1 for gazebo_winter

num of local max[50668] Generate cloud_bin_2 for gazebo_winter

num of local max[47366] Generate cloud_bin_3 for gazebo_winter

num of local max[45184] Generate cloud_bin_4 for gazebo_winter

num of local max[36462] Generate cloud_bin_5 for gazebo_winter

num of local max[39358] Generate cloud_bin_6 for gazebo_winter

num of local max[39224] Generate cloud_bin_7 for gazebo_winter

num of local max[37402] Generate cloud_bin_8 for gazebo_winter

num of local max[36522] Generate cloud_bin_9 for gazebo_winter

num of local max[35572] Generate cloud_bin_10 for gazebo_winter

num of local max[34852] Generate cloud_bin_11 for gazebo_winter

num of local max[36472] Generate cloud_bin_12 for gazebo_winter

num of local max[39158] Generate cloud_bin_13 for gazebo_winter

num of local max[38880] Generate cloud_bin_14 for gazebo_winter

num of local max[35684] Generate cloud_bin_15 for gazebo_winter

num of local max[38366] Generate cloud_bin_16 for gazebo_winter

num of local max[44426] Generate cloud_bin_17 for gazebo_winter

num of local max[50016] Generate cloud_bin_18 for gazebo_winter

num of local max[51948] Generate cloud_bin_19 for gazebo_winter

num of local max[52352] Generate cloud_bin_20 for gazebo_winter

num of local max[52042] Generate cloud_bin_21 for gazebo_winter

num of local max[51230] Generate cloud_bin_22 for gazebo_winter

num of local max[56012] Generate cloud_bin_23 for gazebo_winter

num of local max[55040] Generate cloud_bin_24 for gazebo_winter

num of local max[56960] Generate cloud_bin_25 for gazebo_winter

num of local max[59904] Generate cloud_bin_26 for gazebo_winter

num of local max[57582] Generate cloud_bin_27 for gazebo_winter

num of local max[60356] Generate cloud_bin_28 for gazebo_winter

num of local max[69758] Generate cloud_bin_29 for gazebo_winter

num of local max[67220] Generate cloud_bin_30 for gazebo_winter

num of local max[60774] Generate cloud_bin_0 for wood_autmn

num of local max[65700] Generate cloud_bin_1 for wood_autmn

num of local max[68522] Generate cloud_bin_2 for wood_autmn

num of local max[70236] Generate cloud_bin_3 for wood_autmn

num of local max[65856] Generate cloud_bin_4 for wood_autmn

num of local max[60350] Generate cloud_bin_5 for wood_autmn

num of local max[62804] Generate cloud_bin_6 for wood_autmn

num of local max[62538] Generate cloud_bin_7 for wood_autmn

num of local max[64912] Generate cloud_bin_8 for wood_autmn

num of local max[70100] Generate cloud_bin_9 for wood_autmn

num of local max[70348] Generate cloud_bin_10 for wood_autmn

num of local max[72372] Generate cloud_bin_11 for wood_autmn

num of local max[79158]

num of local max[77498] Generate cloud_bin_13 for wood_autmn

num of local max[75524] Generate cloud_bin_14 for wood_autmn

num of local max[74736] Generate cloud_bin_15 for wood_autmn

num of local max[76278] Generate cloud_bin_16 for wood_autmn

num of local max[81292] Generate cloud_bin_17 for wood_autmn

num of local max[77378] Generate cloud_bin_18 for wood_autmn

num of local max[75340] Generate cloud_bin_19 for wood_autmn

num of local max[69720] Generate cloud_bin_20 for wood_autmn

num of local max[73258] Generate cloud_bin_21 for wood_autmn

num of local max[74398] Generate cloud_bin_22 for wood_autmn

num of local max[72218] Generate cloud_bin_23 for wood_autmn

num of local max[69036] Generate cloud_bin_24 for wood_autmn

num of local max[73088] Generate cloud_bin_25 for wood_autmn

num of local max[73986] Generate cloud_bin_26 for wood_autmn

num of local max[74708] Generate cloud_bin_27 for wood_autmn

num of local max[73818] Generate cloud_bin_28 for wood_autmn

num of local max[70550] Generate cloud_bin_29 for wood_autmn

num of local max[70328] Generate cloud_bin_30 for wood_autmn

num of local max[62090] Generate cloud_bin_31 for wood_autmn

num of local max[67538] Generate cloud_bin_0 for wood_summer

num of local max[69884] Generate cloud_bin_1 for wood_summer

num of local max[73172] Generate cloud_bin_2 for wood_summer

num of local max[69554] Generate cloud_bin_3 for wood_summer

num of local max[70372] Generate cloud_bin_4 for wood_summer

num of local max[71776] Generate cloud_bin_5 for wood_summer

num of local max[69822] Generate cloud_bin_6 for wood_summer

2020-08-01 18:23:38.838700: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.10GiB. Current allocation summary follows. 2020-08-01 18:23:38.838854: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 132, Chunks in use: 127. 33.0KiB allocated for chunks. 31.8KiB in use in bin. 20.9KiB client-requested in use in bin. 2020-08-01 18:23:38.838871: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 81, Chunks in use: 75. 41.0KiB allocated for chunks. 37.5KiB in use in bin. 37.5KiB client-requested in use in bin. 2020-08-01 18:23:38.838885: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 82, Chunks in use: 78. 82.5KiB allocated for chunks. 78.2KiB in use in bin. 78.0KiB client-requested in use in bin. 2020-08-01 18:23:38.838898: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 62, Chunks in use: 62. 125.8KiB allocated for chunks. 125.8KiB in use in bin. 125.8KiB client-requested in use in bin. 2020-08-01 18:23:38.838910: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 20, Chunks in use: 20. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin. 2020-08-01 18:23:38.838923: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 17, Chunks in use: 17. 136.0KiB allocated for chunks. 136.0KiB in use in bin. 136.0KiB client-requested in use in bin. 2020-08-01 18:23:38.838935: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 3, Chunks in use: 3. 48.0KiB allocated for chunks. 48.0KiB in use in bin. 48.0KiB client-requested in use in bin. 2020-08-01 18:23:38.838947: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 5, Chunks in use: 5. 240.0KiB allocated for chunks. 240.0KiB in use in bin. 239.8KiB client-requested in use in bin. 2020-08-01 18:23:38.838959: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 4, Chunks in use: 4. 256.0KiB allocated for chunks. 256.0KiB in use in bin. 256.0KiB client-requested in use in bin. 2020-08-01 18:23:38.838971: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 9, Chunks in use: 8. 1.68MiB allocated for chunks. 1.47MiB in use in bin. 1.47MiB client-requested in use in bin. 2020-08-01 18:23:38.838982: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 7, Chunks in use: 7. 1.95MiB allocated for chunks. 1.95MiB in use in bin. 1.93MiB client-requested in use in bin. 2020-08-01 18:23:38.838993: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 3, Chunks in use: 3. 1.94MiB allocated for chunks. 1.94MiB in use in bin. 1.94MiB client-requested in use in bin. 2020-08-01 18:23:38.839005: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 7, Chunks in use: 5. 8.37MiB allocated for chunks. 5.22MiB in use in bin. 4.94MiB client-requested in use in bin. 2020-08-01 18:23:38.839023: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 5, Chunks in use: 5. 13.35MiB allocated for chunks. 13.35MiB in use in bin. 13.30MiB client-requested in use in bin. 2020-08-01 18:23:38.839043: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 3, Chunks in use: 3. 16.09MiB allocated for chunks. 16.09MiB in use in bin. 15.60MiB client-requested in use in bin. 2020-08-01 18:23:38.839066: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 4, Chunks in use: 4. 40.28MiB allocated for chunks. 40.28MiB in use in bin. 37.28MiB client-requested in use in bin. 2020-08-01 18:23:38.839082: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 4, Chunks in use: 4. 90.00MiB allocated for chunks. 90.00MiB in use in bin. 71.48MiB client-requested in use in bin. 2020-08-01 18:23:38.839096: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 5, Chunks in use: 4. 239.61MiB allocated for chunks. 207.19MiB in use in bin. 165.36MiB client-requested in use in bin. 2020-08-01 18:23:38.839109: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 4, Chunks in use: 4. 411.71MiB allocated for chunks. 411.71MiB in use in bin. 381.42MiB client-requested in use in bin. 2020-08-01 18:23:38.839121: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-08-01 18:23:38.839134: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 8, Chunks in use: 2. 10.30GiB allocated for chunks. 558.96MiB in use in bin. 549.11MiB client-requested in use in bin. 2020-08-01 18:23:38.839145: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 4.10GiB was 256.00MiB, Chunk State: 2020-08-01 18:23:38.839164: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 256.74MiB | Requested Size: 69.74MiB | in_use: 0, prev: Size: 69.74MiB | Requested Size: 69.74MiB | in_use: 1, next: Size: 61.28MiB | Requested Size: 39.42MiB | in_use: 1 2020-08-01 18:23:38.839180: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 353.09MiB | Requested Size: 261.54MiB | in_use: 0, next: Size: 117.70MiB | Requested Size: 117.70MiB | in_use: 1 2020-08-01 18:23:38.839192: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.00GiB | Requested Size: 203.17MiB | in_use: 0 2020-08-01 18:23:38.839205: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.24GiB | Requested Size: 575.39MiB | in_use: 0, prev: Size: 302.96MiB | Requested Size: 302.96MiB | in_use: 1 2020-08-01 18:23:38.839217: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 2.92GiB | Requested Size: 1.99GiB | in_use: 0, next: Size: 43.83MiB | Requested Size: 43.83MiB | in_use: 1 2020-08-01 18:23:38.839227: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 4.00GiB | Requested Size: 3.67GiB | in_use: 0 2020-08-01 18:23:38.839241: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac21600000 of size 1024 2020-08-01 18:23:38.839250: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac21600400 of size 1024 2020-08-01 18:23:38.839257: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac21600800 of size 1024 2020-08-01 18:23:38.839286: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac21600c00 of size 1024 2020-08-01 18:23:38.839295: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac21601000 of size 1024 2020-08-01 18:23:38.839303: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac21601400 of size 1024 2020-08-01 18:23:38.839310: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac21601800 of size 1024 2020-08-01 18:23:38.839318: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac21601c00 of size 1024 2020-08-01 18:23:38.839326: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac21602000 of size 256 2020-08-01 18:23:38.839334: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac21602100 of size 3932160 2020-08-01 18:23:38.839349: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac219c2100 of size 4448000 2020-08-01 18:23:38.839357: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29000000 of size 256 2020-08-01 18:23:38.839365: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29000100 of size 256 2020-08-01 18:23:38.839372: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29000200 of size 256 2020-08-01 18:23:38.839379: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29000300 of size 256 2020-08-01 18:23:38.839387: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29000400 of size 256 2020-08-01 18:23:38.839394: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29000500 of size 256 2020-08-01 18:23:38.849209: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29010300 of size 1024 2020-08-01 18:23:38.849217: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29010700 of size 1024 2020-08-01 18:23:38.849224: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29010b00 of size 1024 2020-08-01 18:23:38.849231: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29010f00 of size 1024 2020-08-01 18:23:38.849239: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29011300 of size 256 2020-08-01 18:23:38.849246: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29011400 of size 1024 2020-08-01 18:23:38.849253: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29011800 of size 1024 2020-08-01 18:23:38.849260: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29011c00 of size 1024 2020-08-01 18:23:38.849268: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29012000 of size 1024 2020-08-01 18:23:38.849275: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29012400 of size 1024 2020-08-01 18:23:38.849282: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29012800 of size 1024 2020-08-01 18:23:38.849291: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29012c00 of size 1024 2020-08-01 18:23:38.849298: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29013000 of size 1024 2020-08-01 18:23:38.849306: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29013400 of size 256 2020-08-01 18:23:38.849313: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29013500 of size 4096 2020-08-01 18:23:38.849321: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29014500 of size 4096 2020-08-01 18:23:38.849328: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29015500 of size 256 2020-08-01 18:23:38.849336: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29015600 of size 2048 2020-08-01 18:23:38.849343: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29015e00 of size 2048 2020-08-01 18:23:38.849350: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29016600 of size 2048 2020-08-01 18:23:38.849368: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29016e00 of size 2048 2020-08-01 18:23:38.849386: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29017600 of size 2048 2020-08-01 18:23:38.849393: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29017e00 of size 2048 2020-08-01 18:23:38.849401: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29018600 of size 2048 2020-08-01 18:23:38.849408: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29018e00 of size 2048 2020-08-01 18:23:38.849415: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29019600 of size 256 2020-08-01 18:23:38.849424: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29019700 of size 8192 2020-08-01 18:23:38.849431: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901b700 of size 8192 2020-08-01 18:23:38.849439: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901d700 of size 256 2020-08-01 18:23:38.849460: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901d800 of size 256 2020-08-01 18:23:38.849468: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901d900 of size 256 2020-08-01 18:23:38.849476: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901da00 of size 256 2020-08-01 18:23:38.849483: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901db00 of size 256 2020-08-01 18:23:38.849490: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901dc00 of size 256 2020-08-01 18:23:38.849497: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901dd00 of size 256 2020-08-01 18:23:38.849505: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901de00 of size 512 2020-08-01 18:23:38.849512: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901e000 of size 512 2020-08-01 18:23:38.849519: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901e200 of size 512 2020-08-01 18:23:38.849526: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901e400 of size 512 2020-08-01 18:23:38.849534: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901e600 of size 256 2020-08-01 18:23:38.849541: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901e700 of size 1024 2020-08-01 18:23:38.849548: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901eb00 of size 1024 2020-08-01 18:23:38.849556: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901ef00 of size 1024 2020-08-01 18:23:38.849563: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901f300 of size 1024 2020-08-01 18:23:38.849570: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901f700 of size 256 2020-08-01 18:23:38.849578: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2901f800 of size 2048 2020-08-01 18:23:38.849585: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29020000 of size 2048 2020-08-01 18:23:38.849592: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29020800 of size 2048 2020-08-01 18:23:38.849600: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29021000 of size 2048 2020-08-01 18:23:38.849607: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29021800 of size 256 2020-08-01 18:23:38.849614: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29021900 of size 256 2020-08-01 18:23:38.849622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29021a00 of size 256 2020-08-01 18:23:38.849629: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29021b00 of size 1280 2020-08-01 18:23:38.849637: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022000 of size 256 2020-08-01 18:23:38.849644: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022100 of size 256 2020-08-01 18:23:38.849651: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022200 of size 256 2020-08-01 18:23:38.849659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022300 of size 256 2020-08-01 18:23:38.849666: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022400 of size 256 2020-08-01 18:23:38.849673: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022500 of size 256 2020-08-01 18:23:38.849681: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022600 of size 256 2020-08-01 18:23:38.849688: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022700 of size 256 2020-08-01 18:23:38.849695: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022800 of size 256 2020-08-01 18:23:38.849702: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022900 of size 512 2020-08-01 18:23:38.853037: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022b00 of size 512 2020-08-01 18:23:38.853062: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022d00 of size 512 2020-08-01 18:23:38.853071: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29022f00 of size 512 2020-08-01 18:23:38.853078: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29023100 of size 512 2020-08-01 18:23:38.853086: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29023300 of size 512 2020-08-01 18:23:38.853094: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29023500 of size 512 2020-08-01 18:23:38.853101: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29023700 of size 512 2020-08-01 18:23:38.853110: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29023900 of size 32768 2020-08-01 18:23:38.853118: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2902b900 of size 256 2020-08-01 18:23:38.853125: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2902ba00 of size 256 2020-08-01 18:23:38.853133: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2902bb00 of size 256 2020-08-01 18:23:38.853141: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2902bc00 of size 256 2020-08-01 18:23:38.853149: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2902bd00 of size 16384 2020-08-01 18:23:38.853156: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2902fd00 of size 256 2020-08-01 18:23:38.853164: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2902fe00 of size 256 2020-08-01 18:23:38.853172: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2902ff00 of size 256 2020-08-01 18:23:38.853179: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29030000 of size 256 2020-08-01 18:23:38.853186: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29030100 of size 256 2020-08-01 18:23:38.853194: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29030200 of size 61440 2020-08-01 18:23:38.853202: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2903f200 of size 61440 2020-08-01 18:23:38.853209: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2904e200 of size 512 2020-08-01 18:23:38.853217: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2904e400 of size 512 2020-08-01 18:23:38.853224: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2904e600 of size 512 2020-08-01 18:23:38.853231: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2904e800 of size 512 2020-08-01 18:23:38.853239: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac2904ea00 of size 16384 2020-08-01 18:23:38.853247: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29052a00 of size 16384 2020-08-01 18:23:38.853254: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29056a00 of size 256 2020-08-01 18:23:38.853262: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29056b00 of size 256 2020-08-01 18:23:38.853269: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29056c00 of size 256 2020-08-01 18:23:38.853277: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29056d00 of size 256 2020-08-01 18:23:38.853284: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29056e00 of size 256 2020-08-01 18:23:38.853292: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29056f00 of size 3840 2020-08-01 18:23:38.853300: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29057e00 of size 256 2020-08-01 18:23:38.853307: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29057f00 of size 256 2020-08-01 18:23:38.853315: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29058000 of size 256 2020-08-01 18:23:38.853322: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29058100 of size 256 2020-08-01 18:23:38.853330: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29058200 of size 32768 2020-08-01 18:23:38.853337: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29060200 of size 256 2020-08-01 18:23:38.853351: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29060300 of size 256 2020-08-01 18:23:38.853366: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29060400 of size 256 2020-08-01 18:23:38.853374: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29060500 of size 256 2020-08-01 18:23:38.853381: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29060600 of size 256 2020-08-01 18:23:38.853389: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29060700 of size 1024 2020-08-01 18:23:38.853397: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29060b00 of size 1024 2020-08-01 18:23:38.853404: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29060f00 of size 1024 2020-08-01 18:23:38.853411: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29061300 of size 1024 2020-08-01 18:23:38.853419: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29061700 of size 1024 2020-08-01 18:23:38.853426: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29061b00 of size 1024 2020-08-01 18:23:38.853433: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29061f00 of size 1024 2020-08-01 18:23:38.853441: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29062300 of size 1024 2020-08-01 18:23:38.853449: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29062700 of size 131072 2020-08-01 18:23:38.853456: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29082700 of size 256 2020-08-01 18:23:38.853464: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29082800 of size 256 2020-08-01 18:23:38.853471: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29082900 of size 256 2020-08-01 18:23:38.853479: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29082a00 of size 256 2020-08-01 18:23:38.853486: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29082b00 of size 256 2020-08-01 18:23:38.853493: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29082c00 of size 256 2020-08-01 18:23:38.853501: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29082d00 of size 256 2020-08-01 18:23:38.853508: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29082e00 of size 256 2020-08-01 18:23:38.853526: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29082f00 of size 256 2020-08-01 18:23:38.853536: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac29083000 of size 245760 2020-08-01 18:23:38.853545: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aac290bf000 of size 266240 2020-08-01 18:23:38.861812: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4844700 of size 512 2020-08-01 18:23:38.861835: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4844900 of size 512 2020-08-01 18:23:38.861844: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4844b00 of size 512 2020-08-01 18:23:38.861852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4844d00 of size 262144 2020-08-01 18:23:38.861860: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4884d00 of size 262144 2020-08-01 18:23:38.861868: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd48c4d00 of size 1024 2020-08-01 18:23:38.861876: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd48c5100 of size 1024 2020-08-01 18:23:38.861884: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd48c5500 of size 1024 2020-08-01 18:23:38.861892: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd48c5900 of size 1024 2020-08-01 18:23:38.861900: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd48c5d00 of size 1048576 2020-08-01 18:23:38.861908: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd49c5d00 of size 1048576 2020-08-01 18:23:38.861916: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ac5d00 of size 2048 2020-08-01 18:23:38.861925: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ac6500 of size 2048 2020-08-01 18:23:38.861933: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ac6d00 of size 2048 2020-08-01 18:23:38.861941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ac7500 of size 2048 2020-08-01 18:23:38.861949: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ac7d00 of size 256 2020-08-01 18:23:38.861957: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ac7e00 of size 256 2020-08-01 18:23:38.861965: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ac7f00 of size 8192 2020-08-01 18:23:38.861973: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ac9f00 of size 256 2020-08-01 18:23:38.861981: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4aca000 of size 2048 2020-08-01 18:23:38.861989: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4aca800 of size 256 2020-08-01 18:23:38.861997: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4aca900 of size 1024 2020-08-01 18:23:38.862005: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acad00 of size 256 2020-08-01 18:23:38.862014: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acae00 of size 512 2020-08-01 18:23:38.862036: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acb000 of size 256 2020-08-01 18:23:38.862044: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acb100 of size 256 2020-08-01 18:23:38.862061: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acb200 of size 256 2020-08-01 18:23:38.862069: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acb300 of size 256 2020-08-01 18:23:38.862077: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acb400 of size 256 2020-08-01 18:23:38.862085: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acb500 of size 256 2020-08-01 18:23:38.862093: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acb600 of size 256 2020-08-01 18:23:38.862101: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acb700 of size 256 2020-08-01 18:23:38.862109: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acb800 of size 256 2020-08-01 18:23:38.862117: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acb900 of size 512 2020-08-01 18:23:38.862125: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acbb00 of size 1024 2020-08-01 18:23:38.865767: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acbf00 of size 512 2020-08-01 18:23:38.865805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acc100 of size 2048 2020-08-01 18:23:38.865822: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acc900 of size 4096 2020-08-01 18:23:38.865835: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acd900 of size 2048 2020-08-01 18:23:38.865848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ace100 of size 2048 2020-08-01 18:23:38.865862: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ace900 of size 512 2020-08-01 18:23:38.865875: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4aceb00 of size 512 2020-08-01 18:23:38.865888: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4aced00 of size 512 2020-08-01 18:23:38.865901: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acef00 of size 1024 2020-08-01 18:23:38.865914: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acf300 of size 512 2020-08-01 18:23:38.865927: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4acf500 of size 512 2020-08-01 18:23:38.865940: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4acf700 of size 8192 2020-08-01 18:23:38.865953: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad1700 of size 8192 2020-08-01 18:23:38.865966: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad3700 of size 4096 2020-08-01 18:23:38.865979: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad4700 of size 4096 2020-08-01 18:23:38.865992: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad5700 of size 2048 2020-08-01 18:23:38.866004: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad5f00 of size 1024 2020-08-01 18:23:38.866025: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad6300 of size 1024 2020-08-01 18:23:38.866049: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad6700 of size 2048 2020-08-01 18:23:38.866062: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad6f00 of size 1024 2020-08-01 18:23:38.866075: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad7300 of size 2048 2020-08-01 18:23:38.866088: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad7b00 of size 256 2020-08-01 18:23:38.866101: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad7c00 of size 2048 2020-08-01 18:23:38.866114: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ad8400 of size 256 2020-08-01 18:23:38.866127: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad8500 of size 1024 2020-08-01 18:23:38.866140: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad8900 of size 256 2020-08-01 18:23:38.866153: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ad8a00 of size 256 2020-08-01 18:23:38.866166: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad8b00 of size 512 2020-08-01 18:23:38.866178: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ad8d00 of size 8192 2020-08-01 18:23:38.866192: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4adad00 of size 4096 2020-08-01 18:23:38.866204: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4adbd00 of size 2048 2020-08-01 18:23:38.866217: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4adc500 of size 1024 2020-08-01 18:23:38.866230: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4adc900 of size 2048 2020-08-01 18:23:38.866243: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4add100 of size 1024 2020-08-01 18:23:38.866256: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4add500 of size 512 2020-08-01 18:23:38.866268: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4add700 of size 1024 2020-08-01 18:23:38.866280: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4addb00 of size 8192 2020-08-01 18:23:38.866292: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4adfb00 of size 256 2020-08-01 18:23:38.866310: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4adfc00 of size 512 2020-08-01 18:23:38.866322: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4adfe00 of size 4096 2020-08-01 18:23:38.866334: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ae0e00 of size 1024 2020-08-01 18:23:38.866347: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae1200 of size 2048 2020-08-01 18:23:38.866363: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ae1a00 of size 256 2020-08-01 18:23:38.866376: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae1b00 of size 1024 2020-08-01 18:23:38.866392: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae1f00 of size 2048 2020-08-01 18:23:38.866404: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ae2700 of size 768 2020-08-01 18:23:38.866417: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae2a00 of size 4096 2020-08-01 18:23:38.866429: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae3a00 of size 1024 2020-08-01 18:23:38.866442: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ae3e00 of size 256 2020-08-01 18:23:38.866454: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae3f00 of size 2048 2020-08-01 18:23:38.866466: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ae4700 of size 768 2020-08-01 18:23:38.866478: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae4a00 of size 512 2020-08-01 18:23:38.866490: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae4c00 of size 512 2020-08-01 18:23:38.866502: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae4e00 of size 512 2020-08-01 18:23:38.866514: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ae5000 of size 1024 2020-08-01 18:23:38.866526: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae5400 of size 1024 2020-08-01 18:23:38.866539: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae5800 of size 256 2020-08-01 18:23:38.866551: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ae5900 of size 256 2020-08-01 18:23:38.866564: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae5a00 of size 512 2020-08-01 18:23:38.866576: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ae5c00 of size 1280 2020-08-01 18:23:38.866588: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ae6100 of size 1024 2020-08-01 18:23:38.866600: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4ae6500 of size 1526272 2020-08-01 18:23:38.866614: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4c5af00 of size 221952 2020-08-01 18:23:38.866627: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4c91200 of size 221952 2020-08-01 18:23:38.866639: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4cc7500 of size 221952 2020-08-01 18:23:38.866651: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4cfd800 of size 1775616 2020-08-01 18:23:38.866664: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4eaf000 of size 285696 2020-08-01 18:23:38.866677: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4ef4c00 of size 147200 2020-08-01 18:23:38.866690: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4f18b00 of size 443904 2020-08-01 18:23:38.866702: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4f85100 of size 57344 2020-08-01 18:23:38.866715: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd4f93100 of size 221952 2020-08-01 18:23:38.866727: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aacd4fc9400 of size 224256 2020-08-01 18:23:38.866741: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd6000000 of size 15728640 2020-08-01 18:23:38.866773: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd6f00000 of size 8388608 2020-08-01 18:23:38.866788: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacd7700000 of size 9437184 2020-08-01 18:23:38.866801: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacdc000000 of size 29293568 2020-08-01 18:23:38.866814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacddbefc00 of size 8227584 2020-08-01 18:23:38.866826: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacde3c8700 of size 29587712 2020-08-01 18:23:38.866839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aace8000000 of size 18283008 2020-08-01 18:23:38.866851: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aace916fa00 of size 33995264 2020-08-01 18:23:38.866864: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aaceb1db400 of size 8684800 2020-08-01 18:23:38.866877: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aaceba23900 of size 2755584 2020-08-01 18:23:38.866889: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacebcc4500 of size 3063808 2020-08-01 18:23:38.866902: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacebfb0500 of size 17207552 2020-08-01 18:23:38.866915: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aaced019600 of size 50227712 2020-08-01 18:23:38.866928: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aacf8000000 of size 268435456 2020-08-01 18:23:38.866941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aad08000000 of size 130270208 2020-08-01 18:23:38.866953: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aad0fc3c400 of size 73132032 2020-08-01 18:23:38.866966: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x2aad141fac00 of size 269209600 2020-08-01 18:23:38.866978: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x2aad242b7c00 of size 64259072 2020-08-01 18:23:38.872938: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3932160 totalling 3.75MiB 2020-08-01 18:23:38.872951: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4194304 totalling 4.00MiB 2020-08-01 18:23:38.872964: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4448000 totalling 4.24MiB 2020-08-01 18:23:38.872977: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 8227584 totalling 7.85MiB 2020-08-01 18:23:38.872990: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 8388608 totalling 8.00MiB 2020-08-01 18:23:38.873003: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 8684800 totalling 8.28MiB 2020-08-01 18:23:38.873034: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 9437184 totalling 9.00MiB 2020-08-01 18:23:38.873049: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 15728640 totalling 15.00MiB 2020-08-01 18:23:38.873063: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 17207552 totalling 16.41MiB 2020-08-01 18:23:38.873077: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 18283008 totalling 17.44MiB 2020-08-01 18:23:38.873090: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 29293568 totalling 27.94MiB 2020-08-01 18:23:38.873104: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 29587712 totalling 28.22MiB 2020-08-01 18:23:38.873117: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 45954560 totalling 43.83MiB 2020-08-01 18:23:38.873131: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 50227712 totalling 47.90MiB 2020-08-01 18:23:38.873145: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 56811520 totalling 54.18MiB 2020-08-01 18:23:38.873158: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 64259072 totalling 61.28MiB 2020-08-01 18:23:38.873172: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 73132032 totalling 69.74MiB 2020-08-01 18:23:38.873185: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 104898048 totalling 100.04MiB 2020-08-01 18:23:38.873199: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 123413760 totalling 117.70MiB 2020-08-01 18:23:38.873213: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 130270208 totalling 124.24MiB 2020-08-01 18:23:38.873227: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 268435456 totalling 256.00MiB 2020-08-01 18:23:38.873241: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 317675520 totalling 302.96MiB 2020-08-01 18:23:38.873254: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 1.32GiB 2020-08-01 18:23:38.873271: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit: 11928924980 InUse: 1414706432 MaxInUse: 8091007488 NumAllocs: 62502 MaxAllocSize: 4294967296

2020-08-01 18:23:38.873351: W tensorflow/core/common_runtime/bfcallocator.cc:271] ****____**_____ 2020-08-01 18:23:38.873422: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at gather_op.cc:103 : Resource exhausted: OOM when allocating tensor with shape[36768,117,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[36768,117,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node KernelPointNetwork/layer_1/resnetb_strided_1/shortcut/GatherV2}} = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](KernelPointNetwork/layer_1/resnetb_strided_1/shortcut/concat, IteratorGetNext/_435, KernelPointNetwork/GatherV2/axis)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[{{node KernelPointNetwork/l2_normalize/_467}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2765_KernelPointNetwork/l2_normalize", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "test_eth.py", line 135, in test_caller(chosen_log, chosen_snapshot, on_val) File "test_eth.py", line 101, in test_caller tester.generate_descriptor(model, dataset) File "/home/Gilgamesh/D3Feat/utils/tester.py", line 199, in generate_descriptor [inputs, features, scores, anc_id] = self.sess.run(ops, {model.dropout_prob: 1.0}) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[36768,117,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node KernelPointNetwork/layer_1/resnetb_strided_1/shortcut/GatherV2 (defined at /home/Gilgamesh/D3Feat/models/network_blocks.py:63) = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](KernelPointNetwork/layer_1/resnetb_strided_1/shortcut/concat, IteratorGetNext/_435, KernelPointNetwork/GatherV2/axis)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[{{node KernelPointNetwork/l2_normalize/_467}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2765_KernelPointNetwork/l2_normalize", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'KernelPointNetwork/layer_1/resnetb_strided_1/shortcut/GatherV2', defined at: File "test_eth.py", line 135, in test_caller(chosen_log, chosen_snapshot, on_val) File "test_eth.py", line 75, in test_caller model = KernelPointFCNN(dataset.flat_inputs, config) File "/home/Gilgamesh/D3Feat/models/KPFCNN_model.py", line 130, in init self.out_features, self.out_scores = assemble_FCNN_blocks(self.anchor_inputs, self.config, self.dropout_prob) File "/home/Gilgamesh/D3Feat/models/D3Feat.py", line 15, in assemble_FCNN_blocks F = assemble_CNN_blocks(inputs, config, dropout_prob) File "/home/Gilgamesh/D3Feat/models/network_blocks.py", line 1099, in assemble_CNN_blocks training) File "/home/Gilgamesh/D3Feat/models/network_blocks.py", line 600, in resnetb_strided_block shortcut = ind_max_pool(features, inputs['pools'][layer_ind]) File "/home/Gilgamesh/D3Feat/models/network_blocks.py", line 63, in ind_max_pool pool_features = tf.gather(x, inds, axis=0) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 2675, in gather return gen_array_ops.gather_v2(params, indices, axis, name=name) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3332, in gather_v2 "GatherV2", params=params, indices=indices, axis=axis, name=name) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/home/Gilgamesh/anaconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[36768,117,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node KernelPointNetwork/layer_1/resnetb_strided_1/shortcut/GatherV2 (defined at /home/Gilgamesh/D3Feat/models/network_blocks.py:63) = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](KernelPointNetwork/layer_1/resnetb_strided_1/shortcut/concat, IteratorGetNext/_435, KernelPointNetwork/GatherV2/axis)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[{{node KernelPointNetwork/l2_normalize/_467}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2765_KernelPointNetwork/l2_normalize", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

2020-08-01 18:23:39.186317: W tensorflow/core/kernels/data/generator_dataset_op.cc:78] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated. [[{{node PyFunc}} = PyFuncTin=[DT_INT64], Tout=[DT_INT64], token="pyfunc_5"]]

XuyangBai commented 4 years ago

Hi @Gilgamesh666666 The situation is similar with https://github.com/XuyangBai/D3Feat/issues/10 where you want to generalize a model trained on indoor settings to outdoor settings. ETH dataset has a far larger scale than the 3DMatch dataset, so when you try to evaluate the pre-trained model to ETH, you should take care of the voxel size and receptive field. Basically you need

  1. set the config.first_subsampling_dl (the voxel size) to a proper value (I use 6.25cm for ETH)
  2. change the scale of the kernel points during testing in here which I explained in the demo file. https://github.com/XuyangBai/D3Feat/blob/3577482efbc5154affcd734e16b0d10a73560e37/demo_registration.py#L139-L142 (I scale up the range of kernel points by 0.10/0.03 or 0.11/0.03 if remember correctly, you can tune with this value but too large value will result in OOM)
Gilgamesh666666 commented 4 years ago

Thank you for the reply! But i still get the same OOM error after i change the config.first_subsampling_dl = 0.0625, and add the following code to the tester.py :( `class ModelTester:

# Initiation methods
# ------------------------------------------------------------------------------------------------------------------

def __init__(self, model, restore_snap=None):

    # Tensorflow Saver definition
    my_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='KernelPointNetwork')
    self.saver = tf.train.Saver(my_vars, max_to_keep=100)

    # Create a session for running Ops on the Graph.
    on_CPU = False
    if on_CPU:
        cProto = tf.ConfigProto(device_count={'GPU': 0})
    else:
        cProto = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True)
        cProto.gpu_options.allow_growth = True
    self.sess = tf.Session(config=cProto)

    # Init variables
    self.sess.run(tf.global_variables_initializer())

    # Name of the snapshot to restore to (None if you want to start from beginning)
    # restore_snap = join(self.saving_path, 'snapshots/snap-40000')
    if (restore_snap is not None):
        self.saver.restore(self.sess, restore_snap)
        print("Model restored from " + restore_snap)
        self.experiment_str = restore_snap.split("_")[-1][:8] + "-" + restore_snap.split("-")[-1]
    # for i, var in enumerate(my_vars):
    # print(i, var.name)
    for v in my_vars:
        if 'kernel_points' in v.name:
            rescale_op = v.assign(tf.multiply(v, 0.11 / 0.03))
            self.sess.run(rescale_op)`
XuyangBai commented 4 years ago

@Gilgamesh666666

I scale up the range of kernel points by 0.10/0.03 or 0.11/0.03 if remember correctly, you can tune with this value but too large value will result in OOM.

Could you use a smaller scale? and BTW, what GPU are you using?

Gilgamesh666666 commented 4 years ago

I change the scale to 0.05/0.03 but it still raise the OOM error.My GPU is TITAN Xp which has 12G memory.BTW, I find that there is some randomness in where the OOM error appear.Sometime the program is bocked at cloud_bin_9.ply of wood_summer, sometime the cloud_bin_10.ply, sometime the other point cloud of gazebo_winter.I just test the wood_summer dataset and remove the cloud_bin_9.ply and cloud_bin_10.ply of the wood_summer and sometime the program can work, other time it is blocked at other point cloud :( .

XuyangBai commented 4 years ago

Hi @Gilgamesh666666 Could your print the self.neighborhood here and what voxel size are you using for ETH? It is unnormal to have OOM even you use the 0.05/0.03 scale. Besides, I think the randomness also comes from the neighborhood_limits, which is calculated at the beginning of the program and it has some randomness.

Gilgamesh666666 commented 4 years ago

Thank you for you patient answer! Here the self.neighborhood self.neighborhood: [ 96 118 189 258 268].I use 0.0625 as the voxel_size.

XuyangBai commented 4 years ago

Hi @Gilgamesh666666 The self.neighborhood is much larger than usual so I checked the code and found that you need to comment the https://github.com/XuyangBai/D3Feat/blob/3577482efbc5154affcd734e16b0d10a73560e37/test_eth.py#L39 The hyperparameter controls the radius of each convolution layer and results in the OOM if set to 2. And please use the same value of config.first_subsampling_dl for scaling (e.g if you set config.first_subsampling_dl to 0.10, then you should use 0.10 / 0.03 for scaling).

Gilgamesh666666 commented 4 years ago

Thank you for your patient reply! It works well!