Caffe2 can't make full use of multi GPU

[Scofieldtangsiwei]

Hi, thanks for the code! I tried to run the code with 8 V100 GPUs. However, all of the GPUs were just working alternatively during the training phase. The estimate time was more than 6 days. I checked your training log, and the estimate time should be less than 2 days.

I am new to caffe2. Could you please give me some suggestion about what could be wrong? Any help would be greatly appreciated! The following is my training log.

W0830 19:07:06.240650 52234 init.py:55] /mnt/lustre/tangsiwei/code/video-long-term-feature-banks/lib/core/config.py:427: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. yaml_config = AttrDict(yaml.load(fopen)) I0830 19:07:06.257894 52234 config.py:369] Config: I0830 19:07:06.264337 52234 config.py:370] {'AVA': {'ANNOTATION_DIR': 'data/ava/annotations', 'DETECTION_SCORE_THRESH_EVAL': [0.85], 'DETECTION_SCORE_THRESH_TRAIN': 0.9, 'FORCE_TEST_FLIP': False, 'FPS': 30, 'FRAME_LIST_DIR': 'data/ava/frame_lists', 'FULL_EVAL_DURING_TRAINING': False, 'LFB_DETECTION_SCORE_THRESH': 0.9, 'LFB_MAX_NUM_FEAT_PER_STEP': 5, 'TEST_BOX_LISTS': ['ava_val_predicted_boxes.csv'], 'TEST_LFB_BOX_LISTS': ['ava_val_predicted_boxes.csv'], 'TEST_LISTS': ['val.csv'], 'TEST_MULTI_CROP': False, 'TEST_MULTI_CROP_SCALES': [224, 256, 320], 'TEST_ON_TEST_SET': False, 'TRAIN_BOX_LISTS': ['ava_train_v2.1.csv', 'ava_train_predicted_boxes.csv'], 'TRAIN_LFB_BOX_LISTS': ['ava_train_predicted_boxes.csv'], 'TRAIN_LISTS': ['train.csv'], 'TRAIN_ON_TRAIN_VAL': False}, 'CHARADES': {'FPS': 24, 'FRAME_LIST_DIR': 'data/charades/frame_lists', 'LFB_CLIPS_PER_SECOND': 2, 'NUM_TEST_CLIPS_DURING_TRAINING': 9, 'NUM_TEST_CLIPS_FINAL_EVAL': 30, 'TEST_LISTS': ['val.csv'], 'TRAIN_LISTS': ['train.csv']}, 'CHECKPOINT': {'CHECKPOINT_MODEL': True, 'CHECKPOINT_PERIOD': 4000, 'CONVERT_MODEL': True, 'DIR': './tmp/testing', 'RESUME': True}, 'CUDNN_WORKSPACE_LIMIT': 256, 'DATADIR': 'data/ava/frames', 'DATALOADER': {'MAX_BAD_IMAGES': 100}, 'DATASET': 'ava', 'DATA_MEAN': [0.45, 0.45, 0.45], 'DATA_STD': [0.225, 0.225, 0.225], 'DEBUG': False, 'EPIC': {'ANNOTATIONS': 'EPIC_train_action_labels.csv', 'ANNOTATION_DIR': 'data/epic/annotations', 'CLASS_TYPE': '', 'FPS': 30, 'FRAME_LIST_DIR': 'data/epic/frame_lists', 'MAX_NUM_FEATS_PER_NOUN_LFB_FRAME': 10, 'NOUN_LFB_FRAMES_PER_SECOND': 1, 'TEST_LISTS': ['val.csv'], 'TRAIN_LISTS': ['train.csv'], 'VERB_LFB_CLIPS_PER_SECOND': 1}, 'FBO_NL': {'DROPOUT_RATE': 0.2, 'INPUT_DROPOUT_ON': True, 'INPUT_REDUCE_DIM': True, 'LATENT_DIM': 512, 'LFB_DROPOUT_ON': True, 'NL_DROPOUT_ON': True, 'NUM_LAYERS': 2, 'PRE_ACT': True, 'PRE_ACT_LN': True, 'SCALE': True}, 'GET_TRAIN_LFB': False, 'IMG_LOAD_RETRY': 10, 'INTERPOLATION': 'INTER_LINEAR', 'LFB': {'ENABLED': False, 'FBO_TYPE': 'nl', 'LFB_DIM': 2048, 'LOAD_LFB': False, 'LOAD_LFB_PATH': '', 'MODEL_PARAMS_FILE': '', 'NUM_LFB_FEAT': 500, 'WINDOW_SIZE': 100, 'WRITE_LFB': False}, 'LOG_PERIOD': 10, 'MINIBATCH_QUEUE_SIZE': 64, 'MODEL': {'ALLOW_INPLACE_RELU': True, 'ALLOW_INPLACE_RESHAPE': True, 'ALLOW_INPLACE_SUM': True, 'BN_EPSILON': 1.0000001e-05, 'BN_INIT_GAMMA': 0.0, 'BN_MOMENTUM': 0.9, 'DEPTH': 50, 'DILATIONS_AFTER_CONV5': True, 'FC_INIT_STD': 0.01, 'FREEZE_BACKBONE': False, 'MEAN': 114.75, 'MEMONGER': True, 'MODEL_NAME': 'resnet_video', 'MULTI_LABEL': True, 'NUM_CLASSES': 80, 'SAMPLE_THREADS': 8, 'STD': 57.375, 'USE_AFFINE': True, 'USE_BGR': False, 'VIDEO_ARC_CHOICE': 2}, 'NONLOCAL': {'BN_EPSILON': 1.0000001e-05, 'BN_INIT_GAMMA': 0.0, 'BN_MOMENTUM': 0.9, 'CONV3_NONLOCAL': True, 'CONV4_NONLOCAL': True, 'CONV_INIT_STD': 0.01, 'LAYER_MOD': 2, 'NO_BIAS': 0, 'USE_AFFINE': True, 'USE_BN': False, 'USE_MAXPOOL': True, 'USE_SCALE': True, 'USE_SOFTMAX': True, 'USE_ZERO_INIT_CONV': True}, 'NUM_GPUS': 8, 'PROF_DAG': False, 'RESNETS': {'NUM_GROUPS': 1, 'STRIDE_1X1': False, 'TRANS_FUNC': 'bottleneck_transformation_3d', 'WIDTH_PER_GROUP': 64}, 'RNG_SEED': 2, 'ROI': {'SCALE_FACTOR': 16, 'XFORM_RESOLUTION': 7}, 'ROOT_GPU_ID': 0, 'SOLVER': {'BASE_LR': 0.04, 'GAMMA': 0.1, 'LRS': [1, 0.1, 0.01, 0.001], 'LR_POLICY': 'steps_with_relative_lrs', 'MAX_ITER': 140000, 'MOMENTUM': 0.9, 'NESTEROV': True, 'SCALE_MOMENTUM': True, 'SCALE_MOMENTUM_THRESHOLD': 1.1, 'STEPS': [0, 100000, 120000, 140000], 'STEP_SIZES': [100000, 20000, 20000], 'WARMUP': {'WARMUP_END_ITER': 2000, 'WARMUP_ON': True, 'WARMUP_START_LR': 0.01}, 'WEIGHT_DECAY': 1e-06, 'WEIGHT_DECAY_BN': 0.0}, 'TEST': {'BATCH_SIZE': 16, 'CROP_SHIFT': 1, 'CROP_SIZE': 256, 'DATASET_SIZE': 64, 'DATA_TYPE': 'val', 'PARAMS_FILE': '', 'SAMPLE_RATE': 2, 'SCALE': 256, 'VIDEO_LENGTH': 32}, 'TRAIN': {'BATCH_SIZE': 16, 'COMPUTE_PRECISE_BN': False, 'CROP_SIZE': 224, 'DATASET_SIZE': 235, 'DATA_TYPE': 'train', 'DROPOUT_RATE': 0.3, 'EVAL_PERIOD': 8000, 'ITER_COMPUTE_PRECISE_BN': 200, 'JITTER_SCALES': [256, 320], 'PARAMS_FILE': 'pretrained_weights/r50_k400_pretrained.pkl', 'PCA_EIGVAL': [0.225, 0.224, 0.229], 'PCA_EIGVEC': [[-0.5675, 0.7192, 0.4009], [-0.5808, -0.0045, -0.814], [-0.5836, -0.6948, 0.4203]], 'RESET_START_ITER': True, 'RESUME_FROM_BATCH_SIZE': -1, 'SAMPLE_RATE': 2, 'TEST_AFTER_TRAIN': True, 'USE_COLOR_AUGMENTATION': False, 'VIDEO_LENGTH': 32}, 'USE_CYTHON': False} Ignoring @/caffe2/caffe2/contrib/gloo:gloo_ops as it is not a valid file. Ignoring @/caffe2/caffe2/contrib/nccl:nccl_ops as it is not a valid file. Ignoring @/caffe2/caffe2/contrib/gloo:gloo_ops_gpu as it is not a valid file. Found Detectron ops lib: /mnt/lustre/tangsiwei/anaconda3/envs/video-lfb/lib/python2.7/site-packages/torch/lib/libcaffe2_detectron_ops_gpu.so [E init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. [E init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. [E init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. I0830 19:07:06.314635 52234 train_net.py:104] Checkpoint directory created: /mnt/lustre/tangsiwei/code/video-long-term-feature-banks/tmp/testing/checkpoints W0830 19:07:06.314785 52234 cnn.py:25] [====DEPRECATE WARNING====]: you are creating an object from CNNModelHelper class which will be deprecated soon. Please use ModelHelper object with brew module. For more information, please refer to caffe2.ai and python/brew.py, python/brew_test.py for more information. I0830 19:07:13.510235 52234 ava.py:98] Finished loading annotations from I0830 19:07:13.510567 52234 ava.py:100] data/ava/annotations/ava_val_predicted_boxes.csv I0830 19:07:13.510648 52234 ava.py:101] Number of unique boxes: 23011 I0830 19:07:13.510713 52234 ava.py:102] Number of annotations: 0 I0830 19:07:14.123112 52234 ava.py:123] 12880 keyframes used. I0830 19:07:14.124880 52234 ava.py:285] === AVA dataset summary === I0830 19:07:14.124999 52234 ava.py:286] Split: val I0830 19:07:14.125072 52234 ava.py:287] Use LFB? False I0830 19:07:14.125166 52234 ava.py:288] Detection threshold: 0.9 I0830 19:07:14.125235 52234 ava.py:290] Full evaluation? False I0830 19:07:14.125298 52234 ava.py:291] Spatial shift position: 1 I0830 19:07:14.125355 52234 ava.py:292] Number of videos: 64 I0830 19:07:14.125439 52234 ava.py:295] Number of frames: 1729931 I0830 19:07:14.125504 52234 ava.py:296] Number of key frames: 12880 I0830 19:07:14.125559 52234 ava.py:297] Number of boxes: 23011. I0830 19:07:14.167545 52234 ava_data_input.py:51] Creating the execution context for worker_ids: [100, 101, 102, 103], batch size: 2 I0830 19:07:14.167649 52234 data_input_helper.py:166] CREATING EXECUTION CONTEXT I0830 19:07:14.167721 52234 data_input_helper.py:173] POOLS: {} I0830 19:07:14.167783 52234 data_input_helper.py:174] SHARED DATA LISTS: 0 I0830 19:07:14.167855 52234 data_input_helper.py:186] worker_id: 100 list: 1 I0830 19:07:14.167917 52234 data_input_helper.py:188] worker_id: 100 list keys: [100] I0830 19:07:14.765388 52234 data_input_helper.py:186] worker_id: 101 list: 2 I0830 19:07:14.765767 52234 data_input_helper.py:188] worker_id: 101 list keys: [100, 101] I0830 19:07:15.353636 52234 data_input_helper.py:186] worker_id: 102 list: 3 I0830 19:07:15.354064 52234 data_input_helper.py:188] worker_id: 102 list keys: [100, 101, 102] I0830 19:07:15.939766 52234 data_input_helper.py:186] worker_id: 103 list: 4 I0830 19:07:15.940150 52234 data_input_helper.py:188] worker_id: 103 list keys: [100, 101, 102, 103] I0830 19:07:16.529685 52234 data_input_helper.py:233] CREATED POOL: {100: <multiprocessing.pool.Pool object at 0x7fec739d9b10>, 101: <multiprocessing.pool.Pool object at 0x7fec6c155590>, 102: <multiprocessing.pool.Pool object at 0x7fec6b876850>, 103: <multiprocessing.pool.Pool object at 0x7fec6f7c7b50>} I0830 19:07:16.530008 52234 data_input_helper.py:234] CREATED LISTS: 4 I0830 19:07:16.530093 52234 data_input_helper.py:235] POOL keys: [100, 101, 102, 103] I0830 19:07:16.530158 52234 data_input_helper.py:236] LIST keys: [100, 101, 102, 103] I0830 19:07:16.530597 52234 data_parallel_model.py:179] Parallelizing model for devices: [0, 1, 2, 3, 4, 5, 6, 7] I0830 19:07:16.530780 52234 data_parallel_model.py:202] Create input and model training operators W0830 19:07:16.530849 52234 data_parallel_model.py:219] W0830 19:07:16.530901 52234 data_parallel_model.py:220] ############# WARNING ############# W0830 19:07:16.530961 52234 data_parallel_model.py:222] Model CNN/<models.model_builder_video.ModelBuilder object at 0x7fed279cc310> is used for testing/validation but W0830 19:07:16.531017 52234 data_parallel_model.py:223] has init_params=True! W0830 19:07:16.531064 52234 data_parallel_model.py:224] This can conflict with model training. W0830 19:07:16.531111 52234 data_parallel_model.py:225] Please ensure model = ModelHelper(init_params=False) W0830 19:07:16.531156 52234 data_parallel_model.py:226] #################################### W0830 19:07:16.531198 52234 data_parallel_model.py:227] I0830 19:07:16.531383 52234 data_parallel_model.py:235] Model for GPU : 0 I0830 19:07:16.532376 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, val, infer LFB? False, suffix: "_test" --------------- I0830 19:07:16.532517 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:07:16.532596 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:07:16.538197 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:07:16.542280 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:07:16.545941 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:07:16.549863 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:07:16.554495 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:07:16.556226 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.558917 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.559146 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.559367 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.560030 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.561593 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.564148 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:07:16.568073 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:07:16.569758 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.572395 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.572626 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.572838 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.573496 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.575174 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.577769 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:07:16.582813 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:07:16.587093 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.587363 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.587609 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.588323 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.592242 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:07:16.596340 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:07:16.600120 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.600358 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.600565 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.601265 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.605360 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:07:16.609683 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:07:16.613583 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.613812 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.614017 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.614656 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.619314 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:07:16.624944 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:07:16.629695 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:07:16.632124 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:07:16.632683 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.633805 52234 data_parallel_model.py:235] Model for GPU : 1 I0830 19:07:16.634202 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, val, infer LFB? False, suffix: "_test" --------------- I0830 19:07:16.634341 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:07:16.634438 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:07:16.639658 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:07:16.645735 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:07:16.650482 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:07:16.655728 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:07:16.662046 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:07:16.664256 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.666789 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.667027 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.667237 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.667980 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.670360 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.674360 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:07:16.680109 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:07:16.682604 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.685549 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.685810 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.686036 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.686685 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.688900 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.693095 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:07:16.700845 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:07:16.705326 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.705559 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.705765 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.706406 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.711894 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:07:16.717701 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:07:16.722105 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.722346 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.722554 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.723192 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.728904 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:07:16.735032 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:07:16.739373 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.739603 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.739819 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.740566 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.747328 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:07:16.755331 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:07:16.762231 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:07:16.764925 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:07:16.765393 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.766457 52234 data_parallel_model.py:235] Model for GPU : 2 I0830 19:07:16.766812 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, val, infer LFB? False, suffix: "_test" --------------- I0830 19:07:16.766932 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:07:16.767009 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:07:16.773197 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:07:16.781512 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:07:16.788110 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:07:16.795691 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:07:16.805157 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:07:16.808296 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.811170 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.811429 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.811670 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.812407 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.815433 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.820075 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:07:16.826854 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:07:16.829720 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.832274 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.832515 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.832721 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.833368 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.836142 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.840914 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:07:16.850320 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:07:16.855798 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.856064 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.856308 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.857040 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.864788 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:07:16.872922 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:07:16.878706 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.878974 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.879223 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.879868 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.887182 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:07:16.894690 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:07:16.899629 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.899859 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.900073 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.900839 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.908355 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:07:16.919411 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:07:16.928198 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:07:16.931871 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:07:16.932555 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.933684 52234 data_parallel_model.py:235] Model for GPU : 3 I0830 19:07:16.934117 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, val, infer LFB? False, suffix: "_test" --------------- I0830 19:07:16.934256 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:07:16.934344 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:07:16.942167 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:07:16.952580 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:07:16.960613 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:07:16.969947 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:07:16.981986 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:07:16.985594 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.988522 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.988792 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.989056 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.989713 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:16.993065 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:16.998684 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:07:17.007006 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:07:17.010421 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.012913 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.013143 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.013365 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.014005 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.017393 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.023313 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:07:17.034627 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:07:17.040735 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.041017 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.041279 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.042010 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.051167 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:07:17.059959 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:07:17.065463 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.065690 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.065896 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.066526 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.075443 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:07:17.084569 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:07:17.090187 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.090610 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.090873 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.091677 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.101994 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:07:17.115643 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:07:17.124939 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:07:17.128842 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:07:17.129312 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.130256 52234 data_parallel_model.py:235] Model for GPU : 4 I0830 19:07:17.130621 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, val, infer LFB? False, suffix: "_test" --------------- I0830 19:07:17.130733 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:07:17.130810 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:07:17.140160 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:07:17.154155 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:07:17.164874 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:07:17.176101 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:07:17.190340 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:07:17.194624 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.197227 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.197462 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.197670 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.198307 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.202248 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.208916 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:07:17.219376 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:07:17.223649 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.226494 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.226761 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.226970 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.227623 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.231674 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.238514 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:07:17.252203 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:07:17.258006 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.258234 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.258450 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.259093 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.269506 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:07:17.280340 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:07:17.287172 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.287461 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.287684 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.288347 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.299495 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:07:17.311429 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:07:17.318329 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.318603 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.318842 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.319477 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.330691 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:07:17.346623 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:07:17.357547 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:07:17.361977 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:07:17.362447 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.363399 52234 data_parallel_model.py:235] Model for GPU : 5 I0830 19:07:17.363749 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, val, infer LFB? False, suffix: "_test" --------------- I0830 19:07:17.363868 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:07:17.363945 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:07:17.374900 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:07:17.390019 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:07:17.402614 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:07:17.415337 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:07:17.431907 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:07:17.436716 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.439537 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.439804 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.440182 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.440932 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.445830 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.454572 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:07:17.467727 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:07:17.472603 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.475419 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.475687 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.475912 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.476552 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.481340 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.489276 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:07:17.504934 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:07:17.511311 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.511542 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.511749 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.512388 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.525268 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:07:17.538783 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:07:17.545888 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.546156 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.546417 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.547152 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.560823 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:07:17.574482 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:07:17.581939 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.582206 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.582477 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.583225 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.597234 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:07:17.613816 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:07:17.626466 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:07:17.631302 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:07:17.631764 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.632704 52234 data_parallel_model.py:235] Model for GPU : 6 I0830 19:07:17.633054 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, val, infer LFB? False, suffix: "_test" --------------- I0830 19:07:17.633173 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:07:17.633249 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:07:17.646836 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:07:17.665616 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:07:17.680110 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:07:17.694772 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:07:17.713988 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:07:17.719443 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.722382 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.722659 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.722902 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.723552 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.728450 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.737257 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:07:17.750847 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:07:17.755753 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.758280 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.758516 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.758730 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.759489 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.764461 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.773648 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:07:17.791815 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:07:17.799212 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.799484 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.799735 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.800647 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.814413 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:07:17.829649 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:07:17.837615 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.837873 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.838133 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.838933 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.854860 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:07:17.869170 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:07:17.876590 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.876820 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.877027 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:17.877671 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.892055 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:07:17.912065 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:07:17.928257 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:07:17.934355 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:07:17.934880 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:17.935945 52234 data_parallel_model.py:235] Model for GPU : 7 I0830 19:07:17.936340 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, val, infer LFB? False, suffix: "_test" --------------- I0830 19:07:17.936476 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:07:17.936556 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:07:17.952896 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:07:17.973505 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:07:17.988411 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:07:18.003474 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:07:18.023256 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:07:18.028675 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.031632 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.031862 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.032073 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.032718 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.038476 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:18.049645 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:07:18.066459 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:07:18.072689 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.075720 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.075995 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.076277 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.077018 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.083389 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:18.094871 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:07:18.115284 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:07:18.122823 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.123054 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.123461 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.124134 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:18.139659 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:07:18.155309 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:07:18.162944 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.163172 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.163388 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.164010 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:18.181622 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:07:18.199265 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:07:18.208132 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.208528 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.208816 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:07:18.209470 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:18.225647 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:07:18.246584 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:07:18.262943 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:07:18.268963 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:07:18.269433 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:07:18.281382 52234 data_parallel_model.py:280] Parameter update function not defined --> only forward [I net_dag_utils.cc:102] Operator graph pruning prior to chain compute took: 0.00124239 secs I0830 19:08:14.466706 52234 model_builder_video.py:160] Starting data loader... I0830 19:08:14.474042 52234 dataloader.py:366] Pre-filling val minibatch queue I0830 19:08:24.484534 52234 dataloader.py:369] val minibatch queue pre-filled. I0830 19:08:24.484950 52234 metrics.py:136] Resetting val metrics... I0830 19:08:24.501225 52234 metrics.py:112] CATEGORIES (60): [ { u'id': 1, u'name': u'bend/bow (at the waist)'}, { u'id': 3, u'name': u'crouch/kneel'}, { u'id': 4, u'name': u'dance'}, { u'id': 5, u'name': u'fall down'}, { u'id': 6, u'name': u'get up'}, { u'id': 7, u'name': u'jump/leap'}, { u'id': 8, u'name': u'lie/sleep'}, { u'id': 9, u'name': u'martial art'}, { u'id': 10, u'name': u'run/jog'}, { u'id': 11, u'name': u'sit'}, { u'id': 12, u'name': u'stand'}, { u'id': 13, u'name': u'swim'}, { u'id': 14, u'name': u'walk'}, { u'id': 15, u'name': u'answer phone'}, { u'id': 17, u'name': u'carry/hold (an object)'}, { u'id': 20, u'name': u'climb (e.g., a mountain)'}, { u'id': 22, u'name': u'close (e.g., a door, a box)'}, { u'id': 24, u'name': u'cut'}, { u'id': 26, u'name': u'dress/put on clothing'}, { u'id': 27, u'name': u'drink'}, { u'id': 28, u'name': u'drive (e.g., a car, a truck)'}, { u'id': 29, u'name': u'eat'}, { u'id': 30, u'name': u'enter'}, { u'id': 34, u'name': u'hit (an object)'}, { u'id': 36, u'name': u'lift/pick up'}, { u'id': 37, u'name': u'listen (e.g., to music)'}, { u'id': 38, u'name': u'open (e.g., a window, a car door)'}, { u'id': 41, u'name': u'play musical instrument'}, { u'id': 43, u'name': u'point to (an object)'}, { u'id': 45, u'name': u'pull (an object)'}, { u'id': 46, u'name': u'push (an object)'}, { u'id': 47, u'name': u'put down'}, { u'id': 48, u'name': u'read'}, { u'id': 49, u'name': u'ride (e.g., a bike, a car, a horse)'}, { u'id': 51, u'name': u'sail boat'}, { u'id': 52, u'name': u'shoot'}, { u'id': 54, u'name': u'smoke'}, { u'id': 56, u'name': u'take a photo'}, { u'id': 57, u'name': u'text on/look at a cellphone'}, { u'id': 58, u'name': u'throw'}, { u'id': 59, u'name': u'touch (an object)'}, { u'id': 60, u'name': u'turn (e.g., a screwdriver)'}, { u'id': 61, u'name': u'watch (e.g., TV)'}, { u'id': 62, u'name': u'work on a computer'}, { u'id': 63, u'name': u'write'}, { u'id': 64, u'name': u'fight/hit (a person)'}, { u'id': 65, u'name': u'give/serve (an object) to (a person)'}, { u'id': 66, u'name': u'grab (a person)'}, { u'id': 67, u'name': u'hand clap'}, { u'id': 68, u'name': u'hand shake'}, { u'id': 69, u'name': u'hand wave'}, { u'id': 70, u'name': u'hug (a person)'}, { u'id': 72, u'name': u'kiss (a person)'}, { u'id': 73, u'name': u'lift (a person)'}, { u'id': 74, u'name': u'listen to (a person)'}, { u'id': 76, u'name': u'push (another person)'}, { u'id': 77, u'name': u'sing to (e.g., self, a person, a group)'}, { u'id': 78, u'name': u'take (an object) from (a person)'}, { u'id': 79, u'name': u'talk to (e.g., self, a person, a group)'}, { u'id': 80, u'name': u'watch (a person)'}] I0830 19:08:26.846582 52234 metrics.py:125] 50250 (mini: 12521) GT boxes loaded from data/ava/annotations/ava_val_v2.1.csv. I0830 19:08:26.846802 52234 metrics.py:129] 50250 (mini: 12521) GT labels loaded from data/ava/annotations/ava_val_v2.1.csv. I0830 19:08:26.888446 52234 misc.py:94] CNN: Net proto saved to: /mnt/lustre/tangsiwei/code/video-long-term-feature-banks/tmp/testing/CNN.pbtxt I0830 19:08:26.899565 52234 misc.py:94] CNN_init: Net proto saved to: /mnt/lustre/tangsiwei/code/video-long-term-feature-banks/tmp/testing/CNN_init.pbtxt I0830 19:08:26.899830 52234 train_net.py:124] Test iters: 805 W0830 19:08:26.899945 52234 cnn.py:25] [====DEPRECATE WARNING====]: you are creating an object from CNNModelHelper class which will be deprecated soon. Please use ModelHelper object with brew module. For more information, please refer to caffe2.ai and python/brew.py, python/brew_test.py for more information. I0830 19:09:13.773592 52234 ava.py:98] Finished loading annotations from I0830 19:09:13.773950 52234 ava.py:100] data/ava/annotations/ava_train_v2.1.csv I0830 19:09:13.774058 52234 ava.py:100] data/ava/annotations/ava_train_predicted_boxes.csv I0830 19:09:13.774177 52234 ava.py:101] Number of unique boxes: 664550 I0830 19:09:13.774302 52234 ava.py:102] Number of annotations: 1603805 I0830 19:09:15.853971 52234 ava.py:123] 193296 keyframes used. I0830 19:09:15.879163 52234 ava.py:285] === AVA dataset summary === I0830 19:09:15.879404 52234 ava.py:286] Split: train I0830 19:09:15.879477 52234 ava.py:287] Use LFB? False I0830 19:09:15.879570 52234 ava.py:288] Detection threshold: 0.9 I0830 19:09:15.879656 52234 ava.py:291] Spatial shift position: 1 I0830 19:09:15.879712 52234 ava.py:292] Number of videos: 235 I0830 19:09:15.879863 52234 ava.py:295] Number of frames: 6352104 I0830 19:09:15.879925 52234 ava.py:296] Number of key frames: 193296 I0830 19:09:15.879982 52234 ava.py:297] Number of boxes: 664550. I0830 19:09:15.969002 52234 ava_data_input.py:51] Creating the execution context for worker_ids: [0, 1, 2, 3], batch size: 2 I0830 19:09:15.969388 52234 data_input_helper.py:166] CREATING EXECUTION CONTEXT I0830 19:09:15.969487 52234 data_input_helper.py:173] POOLS: {100: <multiprocessing.pool.Pool object at 0x7fec739d9b10>, 101: <multiprocessing.pool.Pool object at 0x7fec6c155590>, 102: <multiprocessing.pool.Pool object at 0x7fec6b876850>, 103: <multiprocessing.pool.Pool object at 0x7fec6f7c7b50>} I0830 19:09:15.969563 52234 data_input_helper.py:174] SHARED DATA LISTS: 4 I0830 19:09:15.969644 52234 data_input_helper.py:186] worker_id: 0 list: 5 I0830 19:09:15.969727 52234 data_input_helper.py:188] worker_id: 0 list keys: [0, 100, 101, 102, 103] I0830 19:09:19.118391 52234 data_input_helper.py:186] worker_id: 1 list: 6 I0830 19:09:19.118803 52234 data_input_helper.py:188] worker_id: 1 list keys: [0, 1, 100, 101, 102, 103] I0830 19:09:22.278009 52234 data_input_helper.py:186] worker_id: 2 list: 7 I0830 19:09:22.278753 52234 data_input_helper.py:188] worker_id: 2 list keys: [0, 1, 2, 100, 101, 102, 103] I0830 19:09:25.459793 52234 data_input_helper.py:186] worker_id: 3 list: 8 I0830 19:09:25.460221 52234 data_input_helper.py:188] worker_id: 3 list keys: [0, 1, 2, 3, 100, 101, 102, 103] I0830 19:09:28.649919 52234 data_input_helper.py:233] CREATED POOL: {0: <multiprocessing.pool.Pool object at 0x7fec6b917b90>, 1: <multiprocessing.pool.Pool object at 0x7fe908882c90>, 2: <multiprocessing.pool.Pool object at 0x7fe8165a8fd0>, 3: <multiprocessing.pool.Pool object at 0x7fece93a5150>, 100: <multiprocessing.pool.Pool object at 0x7fec739d9b10>, 101: <multiprocessing.pool.Pool object at 0x7fec6c155590>, 102: <multiprocessing.pool.Pool object at 0x7fec6b876850>, 103: <multiprocessing.pool.Pool object at 0x7fec6f7c7b50>} I0830 19:09:28.650397 52234 data_input_helper.py:234] CREATED LISTS: 8 I0830 19:09:28.650516 52234 data_input_helper.py:235] POOL keys: [0, 1, 2, 3, 100, 101, 102, 103] I0830 19:09:28.650593 52234 data_input_helper.py:236] LIST keys: [0, 1, 2, 3, 100, 101, 102, 103] I0830 19:09:28.651276 52234 data_parallel_model.py:179] Parallelizing model for devices: [0, 1, 2, 3, 4, 5, 6, 7] I0830 19:09:28.652611 52234 data_parallel_model.py:202] Create input and model training operators I0830 19:09:28.653100 52234 data_parallel_model.py:235] Model for GPU : 0 I0830 19:09:28.654417 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, train, infer LFB? False, suffix: "_train" --------------- I0830 19:09:28.654571 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:09:28.654660 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:09:28.659837 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:09:28.665136 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:09:28.670670 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:09:28.675144 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:09:28.680469 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:09:28.682694 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.686098 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.686414 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.686690 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.687441 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.689328 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.692172 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:09:28.696769 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:09:28.698951 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.702254 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.702563 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.702820 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.703737 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.705739 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.708966 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:09:28.714900 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:09:28.719639 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.719944 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.720273 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.721085 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.725974 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:09:28.731035 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:09:28.735909 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.736192 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.736509 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.737338 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.742502 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:09:28.747786 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:09:28.752815 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.753190 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.753823 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.754784 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.760376 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:09:28.767510 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:09:28.773307 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:09:28.776067 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:09:28.776712 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.778121 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: SigmoidCrossEntropyLoss. I0830 19:09:28.778541 52234 data_parallel_model.py:235] Model for GPU : 1 I0830 19:09:28.778987 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, train, infer LFB? False, suffix: "_train" --------------- I0830 19:09:28.779185 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:09:28.779274 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:09:28.785080 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:09:28.792534 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:09:28.798434 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:09:28.804635 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:09:28.812396 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:09:28.815089 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.818437 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.818775 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.819111 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.819920 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.822500 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.826706 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:09:28.833091 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:09:28.835910 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.839248 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.839582 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.839833 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.840537 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.843158 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.847609 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:09:28.856164 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:09:28.861639 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.861960 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.862406 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.863260 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.870143 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:09:28.877443 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:09:28.882994 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.883315 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.883594 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.884321 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.891436 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:09:28.898601 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:09:28.904164 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.904453 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.904762 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.905508 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.912806 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:09:28.922363 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:09:28.930206 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:09:28.933595 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:09:28.934222 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.935692 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: SigmoidCrossEntropyLoss. I0830 19:09:28.935981 52234 data_parallel_model.py:235] Model for GPU : 2 I0830 19:09:28.936393 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, train, infer LFB? False, suffix: "_train" --------------- I0830 19:09:28.936532 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:09:28.936619 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:09:28.944143 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:09:28.954183 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:09:28.962094 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:09:28.970392 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:09:28.980633 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:09:28.984044 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.987533 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.987996 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.988326 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.989151 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:28.992436 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:28.998114 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:09:29.006261 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:09:29.009684 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.013032 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.013335 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.013659 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.014503 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.017807 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.023565 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:09:29.034796 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:09:29.040769 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.041074 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.041349 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.042176 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.050826 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:09:29.059725 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:09:29.065632 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.065911 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.066219 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.067122 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.075906 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:09:29.085078 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:09:29.091207 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.091742 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.092174 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.093014 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.102336 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:09:29.114418 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:09:29.123857 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:09:29.127736 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:09:29.128413 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.129875 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: SigmoidCrossEntropyLoss. I0830 19:09:29.130304 52234 data_parallel_model.py:235] Model for GPU : 3 I0830 19:09:29.130752 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, train, infer LFB? False, suffix: "_train" --------------- I0830 19:09:29.130873 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:09:29.130959 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:09:29.140233 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:09:29.152796 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:09:29.162410 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:09:29.172506 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:09:29.185235 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:09:29.189249 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.192491 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.192843 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.193183 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.194016 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.197947 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.204898 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:09:29.215068 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:09:29.219309 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.222692 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.223056 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.223376 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.224191 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.228065 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.235281 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:09:29.248838 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:09:29.255561 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.255887 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.256161 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.256999 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.267656 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:09:29.278332 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:09:29.285098 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.285542 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.285818 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.286576 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.296924 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:09:29.308238 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:09:29.314995 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.315324 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.315606 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.316515 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.327600 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:09:29.342370 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:09:29.353537 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:09:29.358077 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:09:29.358680 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.360349 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: SigmoidCrossEntropyLoss. I0830 19:09:29.360707 52234 data_parallel_model.py:235] Model for GPU : 4 I0830 19:09:29.361282 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, train, infer LFB? False, suffix: "_train" --------------- I0830 19:09:29.361423 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:09:29.361586 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:09:29.372606 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:09:29.387558 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:09:29.399009 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:09:29.411051 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:09:29.426318 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:09:29.431092 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.434360 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.434662 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.434896 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.435616 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.440191 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.448579 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:09:29.460676 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:09:29.465436 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.468662 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.469017 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.469331 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.470292 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.474948 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.483335 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:09:29.499169 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:09:29.506524 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.506856 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.507122 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.507977 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.520477 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:09:29.532798 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:09:29.540523 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.540839 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.541186 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.542043 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.554579 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:09:29.567687 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:09:29.575059 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.575427 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.575684 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.576683 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.589602 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:09:29.606734 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:09:29.619787 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:09:29.625047 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:09:29.625895 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.627419 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: SigmoidCrossEntropyLoss. I0830 19:09:29.627741 52234 data_parallel_model.py:235] Model for GPU : 5 I0830 19:09:29.628225 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, train, infer LFB? False, suffix: "_train" --------------- I0830 19:09:29.628386 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:09:29.628499 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:09:29.641808 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:09:29.659559 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:09:29.672844 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:09:29.686876 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:09:29.704896 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:09:29.710073 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.713407 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.713746 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.714025 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.714844 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.720221 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.729626 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:09:29.743360 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:09:29.748924 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.752302 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.752635 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.752968 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.753777 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.759031 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.768712 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:09:29.787394 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:09:29.795617 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.795939 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.796221 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.797029 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.811767 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:09:29.826169 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:09:29.834435 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.834741 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.835149 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.835978 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.850524 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:09:29.865463 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:09:29.873516 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.874002 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.874324 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.875173 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:29.890012 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:09:29.909723 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:09:29.924559 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:09:29.930460 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:09:29.931093 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:29.932595 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: SigmoidCrossEntropyLoss. I0830 19:09:29.932925 52234 data_parallel_model.py:235] Model for GPU : 6 I0830 19:09:29.933429 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, train, infer LFB? False, suffix: "_train" --------------- I0830 19:09:29.933629 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:09:29.933734 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:09:29.948755 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:09:29.968950 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:09:29.984209 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:09:29.999731 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:09:30.019891 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:09:30.025645 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.028886 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.029208 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.029541 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.030596 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.036437 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:30.047653 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:09:30.063982 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:09:30.070091 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.073535 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.073950 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.074239 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.075124 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.081247 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:30.092174 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:09:30.114026 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:09:30.122634 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.122931 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.123239 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.124078 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:30.140419 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:09:30.156651 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:09:30.165060 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.165451 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.165808 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.166655 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:30.183330 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:09:30.200120 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:09:30.209012 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.209328 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.209616 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.210438 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:30.227205 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:09:30.249531 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:09:30.266454 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:09:30.273011 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:09:30.273639 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.275023 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: SigmoidCrossEntropyLoss. I0830 19:09:30.275368 52234 data_parallel_model.py:235] Model for GPU : 7 I0830 19:09:30.275844 52234 resnet_video.py:150] --------------- ResNet-50 1x64d-bottleneck_transformation_3d, ava, train, infer LFB? False, suffix: "_train" --------------- I0830 19:09:30.275989 52234 resnet_video.py:166] use_temp_convs_set: [[2], [1, 1, 1], [1, 0, 1, 0], [1, 0, 1, 0, 1, 0], [0, 1, 0]] I0830 19:09:30.276107 52234 resnet_video.py:167] temp_strides_set: [[1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1]] I0830 19:09:30.293078 52234 resnet_helper.py:62] res2_0 using dilation 1 I0830 19:09:30.315736 52234 resnet_helper.py:62] res2_1 using dilation 1 I0830 19:09:30.333303 52234 resnet_helper.py:62] res2_2 using dilation 1 I0830 19:09:30.351192 52234 resnet_helper.py:62] res3_0 using dilation 1 I0830 19:09:30.374474 52234 resnet_helper.py:62] res3_1 using dilation 1 W0830 19:09:30.381048 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.384134 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.384649 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.385051 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.385761 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.392219 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:30.403996 52234 resnet_helper.py:62] res3_2 using dilation 1 I0830 19:09:30.421693 52234 resnet_helper.py:62] res3_3 using dilation 1 W0830 19:09:30.428195 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.431152 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.431482 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.431767 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.433079 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.439639 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:30.451611 52234 resnet_helper.py:62] res4_0 using dilation 1 I0830 19:09:30.475369 52234 resnet_helper.py:62] res4_1 using dilation 1 W0830 19:09:30.484308 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.484662 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.484976 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.485940 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:30.504401 52234 resnet_helper.py:62] res4_2 using dilation 1 I0830 19:09:30.522706 52234 resnet_helper.py:62] res4_3 using dilation 1 W0830 19:09:30.531685 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.532015 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.532294 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.533013 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:30.551585 52234 resnet_helper.py:62] res4_4 using dilation 1 I0830 19:09:30.569916 52234 resnet_helper.py:62] res4_5 using dilation 1 W0830 19:09:30.579063 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.579376 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.579664 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.580425 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. I0830 19:09:30.598889 52234 resnet_helper.py:62] res5_0 using dilation 2 I0830 19:09:30.623996 52234 resnet_helper.py:62] res5_1 using dilation 2 I0830 19:09:30.642997 52234 resnet_helper.py:62] res5_2 using dilation 2 W0830 19:09:30.650089 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: RoIAlign. W0830 19:09:30.650825 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: Reshape. W0830 19:09:30.652317 52234 model_helper.py:442] You are creating an op that the ModelHelper does not recognize: SigmoidCrossEntropyLoss. I0830 19:09:30.666845 52234 data_parallel_model.py:269] Adding gradient operators I0830 19:09:31.329250 52234 data_parallel_model.py:313] Add gradient all-reduces for SyncSGD I0830 19:09:31.357079 52234 data_parallel_model.py:333] Post-iteration operators for updating params I0830 19:09:31.939603 52234 data_parallel_model.py:372] Add initial parameter sync W0830 19:09:32.140983 52234 data_parallel_model.py:1737] ------- DEPRECATED API, please use data_parallel_model.OptimizeGradientMemory() ----- W0830 19:09:32.141791 52234 memonger.py:55] NOTE: Executing memonger to optimize gradient memory [I memonger.cc:236] Remapping 155 using 22 shared blobs. I0830 19:09:32.421159 52234 memonger.py:97] Memonger memory optimization took 0.103475809097 secs W0830 19:09:32.608856 52234 memonger.py:55] NOTE: Executing memonger to optimize gradient memory [I memonger.cc:236] Remapping 155 using 22 shared blobs. I0830 19:09:32.863713 52234 memonger.py:97] Memonger memory optimization took 0.0943260192871 secs W0830 19:09:34.418710 52234 memonger.py:55] NOTE: Executing memonger to optimize gradient memory [I memonger.cc:236] Remapping 155 using 22 shared blobs. I0830 19:09:34.668531 52234 memonger.py:97] Memonger memory optimization took 0.0895349979401 secs W0830 19:09:34.858258 52234 memonger.py:55] NOTE: Executing memonger to optimize gradient memory [I memonger.cc:236] Remapping 155 using 22 shared blobs. I0830 19:09:35.108897 52234 memonger.py:97] Memonger memory optimization took 0.0889720916748 secs W0830 19:09:35.298327 52234 memonger.py:55] NOTE: Executing memonger to optimize gradient memory [I memonger.cc:236] Remapping 155 using 22 shared blobs. I0830 19:09:35.545830 52234 memonger.py:97] Memonger memory optimization took 0.0890431404114 secs W0830 19:09:35.736470 52234 memonger.py:55] NOTE: Executing memonger to optimize gradient memory [I memonger.cc:236] Remapping 155 using 22 shared blobs. I0830 19:09:35.985302 52234 memonger.py:97] Memonger memory optimization took 0.0887980461121 secs W0830 19:09:36.175476 52234 memonger.py:55] NOTE: Executing memonger to optimize gradient memory [I memonger.cc:236] Remapping 155 using 22 shared blobs. I0830 19:09:36.419379 52234 memonger.py:97] Memonger memory optimization took 0.0890748500824 secs W0830 19:09:36.610584 52234 memonger.py:55] NOTE: Executing memonger to optimize gradient memory [I memonger.cc:236] Remapping 155 using 22 shared blobs. I0830 19:09:36.857012 52234 memonger.py:97] Memonger memory optimization took 0.089213848114 secs [I net_dag_utils.cc:102] Operator graph pruning prior to chain compute took: 0.00454839 secs I0830 19:09:50.474113 52234 model_builder_video.py:160] Starting data loader... I0830 19:09:50.609410 52234 dataloader.py:366] Pre-filling train minibatch queue I0830 19:10:23.974463 52234 dataloader.py:369] train minibatch queue pre-filled. I0830 19:10:23.974859 52234 metrics.py:136] Resetting train metrics... I0830 19:10:24.002840 52234 metrics.py:112] CATEGORIES (60): [ { u'id': 1, u'name': u'bend/bow (at the waist)'}, { u'id': 3, u'name': u'crouch/kneel'}, { u'id': 4, u'name': u'dance'}, { u'id': 5, u'name': u'fall down'}, { u'id': 6, u'name': u'get up'}, { u'id': 7, u'name': u'jump/leap'}, { u'id': 8, u'name': u'lie/sleep'}, { u'id': 9, u'name': u'martial art'}, { u'id': 10, u'name': u'run/jog'}, { u'id': 11, u'name': u'sit'}, { u'id': 12, u'name': u'stand'}, { u'id': 13, u'name': u'swim'}, { u'id': 14, u'name': u'walk'}, { u'id': 15, u'name': u'answer phone'}, { u'id': 17, u'name': u'carry/hold (an object)'}, { u'id': 20, u'name': u'climb (e.g., a mountain)'}, { u'id': 22, u'name': u'close (e.g., a door, a box)'}, { u'id': 24, u'name': u'cut'}, { u'id': 26, u'name': u'dress/put on clothing'}, { u'id': 27, u'name': u'drink'}, { u'id': 28, u'name': u'drive (e.g., a car, a truck)'}, { u'id': 29, u'name': u'eat'}, { u'id': 30, u'name': u'enter'}, { u'id': 34, u'name': u'hit (an object)'}, { u'id': 36, u'name': u'lift/pick up'}, { u'id': 37, u'name': u'listen (e.g., to music)'}, { u'id': 38, u'name': u'open (e.g., a window, a car door)'}, { u'id': 41, u'name': u'play musical instrument'}, { u'id': 43, u'name': u'point to (an object)'}, { u'id': 45, u'name': u'pull (an object)'}, { u'id': 46, u'name': u'push (an object)'}, { u'id': 47, u'name': u'put down'}, { u'id': 48, u'name': u'read'}, { u'id': 49, u'name': u'ride (e.g., a bike, a car, a horse)'}, { u'id': 51, u'name': u'sail boat'}, { u'id': 52, u'name': u'shoot'}, { u'id': 54, u'name': u'smoke'}, { u'id': 56, u'name': u'take a photo'}, { u'id': 57, u'name': u'text on/look at a cellphone'}, { u'id': 58, u'name': u'throw'}, { u'id': 59, u'name': u'touch (an object)'}, { u'id': 60, u'name': u'turn (e.g., a screwdriver)'}, { u'id': 61, u'name': u'watch (e.g., TV)'}, { u'id': 62, u'name': u'work on a computer'}, { u'id': 63, u'name': u'write'}, { u'id': 64, u'name': u'fight/hit (a person)'}, { u'id': 65, u'name': u'give/serve (an object) to (a person)'}, { u'id': 66, u'name': u'grab (a person)'}, { u'id': 67, u'name': u'hand clap'}, { u'id': 68, u'name': u'hand shake'}, { u'id': 69, u'name': u'hand wave'}, { u'id': 70, u'name': u'hug (a person)'}, { u'id': 72, u'name': u'kiss (a person)'}, { u'id': 73, u'name': u'lift (a person)'}, { u'id': 74, u'name': u'listen to (a person)'}, { u'id': 76, u'name': u'push (another person)'}, { u'id': 77, u'name': u'sing to (e.g., self, a person, a group)'}, { u'id': 78, u'name': u'take (an object) from (a person)'}, { u'id': 79, u'name': u'talk to (e.g., self, a person, a group)'}, { u'id': 80, u'name': u'watch (a person)'}] I0830 19:10:25.898386 52234 metrics.py:125] 50250 (mini: 12521) GT boxes loaded from data/ava/annotations/ava_val_v2.1.csv. I0830 19:10:25.898581 52234 metrics.py:129] 50250 (mini: 12521) GT labels loaded from data/ava/annotations/ava_val_v2.1.csv. I0830 19:10:25.965214 52234 misc.py:94] CNN_1: Net proto saved to: /mnt/lustre/tangsiwei/code/video-long-term-feature-banks/tmp/testing/CNN_1.pbtxt I0830 19:10:25.990777 52234 misc.py:94] CNN_init_1: Net proto saved to: /mnt/lustre/tangsiwei/code/video-long-term-feature-banks/tmp/testing/CNN_init_1.pbtxt I0830 19:10:25.992065 52234 checkpoints.py:201] Resume training: True I0830 19:10:25.992189 52234 checkpoints.py:151] Coverting model : pretrained_weights/r50_k400_pretrained.pkl I0830 19:10:35.829881 52234 checkpoints.py:207] Checkpoint model converted I0830 19:10:35.830411 52234 checkpoints.py:230] Initializing from checkpoints... I0830 19:10:35.831393 52234 checkpoints.py:422] Initializing model params from file: /mnt/lustre/tangsiwei/code/video-long-term-feature-banks/tmp/testing/checkpoints/c2_model_iter8000.pkl I0830 19:10:35.837887 52234 checkpoints.py:277] Initializing model params from file: /mnt/lustre/tangsiwei/code/video-long-term-feature-banks/tmp/testing/checkpoints/c2_model_iter8000.pkl I0830 19:10:39.085561 52234 checkpoints.py:377] conv1_w_momentum loaded from weights file into: gpu_0/conv1_w_momentum (64, 3, 5, 7, 7) I0830 19:10:39.087021 52234 checkpoints.py:377] res2_0_branch2a_w_momentum loaded from weights file into: gpu_0/res2_0_branch2a_w_momentum (64, 64, 3, 1, 1) I0830 19:10:39.087332 52234 checkpoints.py:377] res2_0_branch2b_w_momentum loaded from weights file into: gpu_0/res2_0_branch2b_w_momentum (64, 64, 1, 3, 3) I0830 19:10:39.087625 52234 checkpoints.py:377] res2_0_branch2c_w_momentum loaded from weights file into: gpu_0/res2_0_branch2c_w_momentum (256, 64, 1, 1, 1) I0830 19:10:39.087891 52234 checkpoints.py:377] res2_0_branch1_w_momentum loaded from weights file into: gpu_0/res2_0_branch1_w_momentum (256, 64, 1, 1, 1) I0830 19:10:39.088236 52234 checkpoints.py:377] res2_1_branch2a_w_momentum loaded from weights file into: gpu_0/res2_1_branch2a_w_momentum (64, 256, 3, 1, 1) I0830 19:10:39.088551 52234 checkpoints.py:377] res2_1_branch2b_w_momentum loaded from weights file into: gpu_0/res2_1_branch2b_w_momentum (64, 64, 1, 3, 3) I0830 19:10:39.088828 52234 checkpoints.py:377] res2_1_branch2c_w_momentum loaded from weights file into: gpu_0/res2_1_branch2c_w_momentum (256, 64, 1, 1, 1) I0830 19:10:39.089107 52234 checkpoints.py:377] res2_2_branch2a_w_momentum loaded from weights file into: gpu_0/res2_2_branch2a_w_momentum (64, 256, 3, 1, 1) I0830 19:10:39.089410 52234 checkpoints.py:377] res2_2_branch2b_w_momentum loaded from weights file into: gpu_0/res2_2_branch2b_w_momentum (64, 64, 1, 3, 3) I0830 19:10:39.089683 52234 checkpoints.py:377] res2_2_branch2c_w_momentum loaded from weights file into: gpu_0/res2_2_branch2c_w_momentum (256, 64, 1, 1, 1) I0830 19:10:39.089998 52234 checkpoints.py:377] res3_0_branch2a_w_momentum loaded from weights file into: gpu_0/res3_0_branch2a_w_momentum (128, 256, 3, 1, 1) I0830 19:10:39.090430 52234 checkpoints.py:377] res3_0_branch2b_w_momentum loaded from weights file into: gpu_0/res3_0_branch2b_w_momentum (128, 128, 1, 3, 3) I0830 19:10:39.090852 52234 checkpoints.py:377] res3_0_branch2c_w_momentum loaded from weights file into: gpu_0/res3_0_branch2c_w_momentum (512, 128, 1, 1, 1) I0830 19:10:39.091228 52234 checkpoints.py:377] res3_0_branch1_w_momentum loaded from weights file into: gpu_0/res3_0_branch1_w_momentum (512, 256, 1, 1, 1) I0830 19:10:39.091640 52234 checkpoints.py:377] res3_1_branch2a_w_momentum loaded from weights file into: gpu_0/res3_1_branch2a_w_momentum (128, 512, 1, 1, 1) I0830 19:10:39.092039 52234 checkpoints.py:377] res3_1_branch2b_w_momentum loaded from weights file into: gpu_0/res3_1_branch2b_w_momentum (128, 128, 1, 3, 3) I0830 19:10:39.092482 52234 checkpoints.py:377] res3_1_branch2c_w_momentum loaded from weights file into: gpu_0/res3_1_branch2c_w_momentum (512, 128, 1, 1, 1) I0830 19:10:39.092915 52234 checkpoints.py:377] nonlocal_conv3_1_theta_w_momentum loaded from weights file into: gpu_0/nonlocal_conv3_1_theta_w_momentum (256, 512, 1, 1, 1) I0830 19:10:39.093269 52234 checkpoints.py:377] nonlocal_conv3_1_theta_b_momentum loaded from weights file into: gpu_0/nonlocal_conv3_1_theta_b_momentum (256,) I0830 19:10:39.093595 52234 checkpoints.py:377] nonlocal_conv3_1_phi_w_momentum loaded from weights file into: gpu_0/nonlocal_conv3_1_phi_w_momentum (256, 512, 1, 1, 1) I0830 19:10:39.094115 52234 checkpoints.py:377] nonlocal_conv3_1_phi_b_momentum loaded from weights file into: gpu_0/nonlocal_conv3_1_phi_b_momentum (256,) I0830 19:10:39.094526 52234 checkpoints.py:377] nonlocal_conv3_1_g_w_momentum loaded from weights file into: gpu_0/nonlocal_conv3_1_g_w_momentum (256, 512, 1, 1, 1) I0830 19:10:39.094930 52234 checkpoints.py:377] nonlocal_conv3_1_g_b_momentum loaded from weights file into: gpu_0/nonlocal_conv3_1_g_b_momentum (256,) I0830 19:10:39.095343 52234 checkpoints.py:377] nonlocal_conv3_1_out_w_momentum loaded from weights file into: gpu_0/nonlocal_conv3_1_out_w_momentum (512, 256, 1, 1, 1) I0830 19:10:39.095715 52234 checkpoints.py:377] nonlocal_conv3_1_out_b_momentum loaded from weights file into: gpu_0/nonlocal_conv3_1_out_b_momentum (512,) I0830 19:10:39.096070 52234 checkpoints.py:377] res3_2_branch2a_w_momentum loaded from weights file into: gpu_0/res3_2_branch2a_w_momentum (128, 512, 3, 1, 1) I0830 19:10:39.096647 52234 checkpoints.py:377] res3_2_branch2b_w_momentum loaded from weights file into: gpu_0/res3_2_branch2b_w_momentum (128, 128, 1, 3, 3) I0830 19:10:39.097110 52234 checkpoints.py:377] res3_2_branch2c_w_momentum loaded from weights file into: gpu_0/res3_2_branch2c_w_momentum (512, 128, 1, 1, 1) I0830 19:10:39.097487 52234 checkpoints.py:377] res3_3_branch2a_w_momentum loaded from weights file into: gpu_0/res3_3_branch2a_w_momentum (128, 512, 1, 1, 1) I0830 19:10:39.097901 52234 checkpoints.py:377] res3_3_branch2b_w_momentum loaded from weights file into: gpu_0/res3_3_branch2b_w_momentum (128, 128, 1, 3, 3) I0830 19:10:39.098350 52234 checkpoints.py:377] res3_3_branch2c_w_momentum loaded from weights file into: gpu_0/res3_3_branch2c_w_momentum (512, 128, 1, 1, 1) I0830 19:10:39.098733 52234 checkpoints.py:377] nonlocal_conv3_3_theta_w_momentum loaded from weights file into: gpu_0/nonlocal_conv3_3_theta_w_momentum (256, 512, 1, 1, 1) I0830 19:10:39.099153 52234 checkpoints.py:377] nonlocal_conv3_3_theta_b_momentum loaded from weights file into: gpu_0/nonlocal_conv3_3_theta_b_momentum (256,) I0830 19:10:39.099503 52234 checkpoints.py:377] nonlocal_conv3_3_phi_w_momentum loaded from weights file into: gpu_0/nonlocal_conv3_3_phi_w_momentum (256, 512, 1, 1, 1) I0830 19:10:39.099910 52234 checkpoints.py:377] nonlocal_conv3_3_phi_b_momentum loaded from weights file into: gpu_0/nonlocal_conv3_3_phi_b_momentum (256,) I0830 19:10:39.100230 52234 checkpoints.py:377] nonlocal_conv3_3_g_w_momentum loaded from weights file into: gpu_0/nonlocal_conv3_3_g_w_momentum (256, 512, 1, 1, 1) I0830 19:10:39.100608 52234 checkpoints.py:377] nonlocal_conv3_3_g_b_momentum loaded from weights file into: gpu_0/nonlocal_conv3_3_g_b_momentum (256,) I0830 19:10:39.100903 52234 checkpoints.py:377] nonlocal_conv3_3_out_w_momentum loaded from weights file into: gpu_0/nonlocal_conv3_3_out_w_momentum (512, 256, 1, 1, 1) I0830 19:10:39.101253 52234 checkpoints.py:377] nonlocal_conv3_3_out_b_momentum loaded from weights file into: gpu_0/nonlocal_conv3_3_out_b_momentum (512,) I0830 19:10:39.101741 52234 checkpoints.py:377] res4_0_branch2a_w_momentum loaded from weights file into: gpu_0/res4_0_branch2a_w_momentum (256, 512, 3, 1, 1) I0830 19:10:39.102605 52234 checkpoints.py:377] res4_0_branch2b_w_momentum loaded from weights file into: gpu_0/res4_0_branch2b_w_momentum (256, 256, 1, 3, 3) I0830 19:10:39.103498 52234 checkpoints.py:377] res4_0_branch2c_w_momentum loaded from weights file into: gpu_0/res4_0_branch2c_w_momentum (1024, 256, 1, 1, 1) I0830 19:10:39.104295 52234 checkpoints.py:377] res4_0_branch1_w_momentum loaded from weights file into: gpu_0/res4_0_branch1_w_momentum (1024, 512, 1, 1, 1) I0830 19:10:39.105215 52234 checkpoints.py:377] res4_1_branch2a_w_momentum loaded from weights file into: gpu_0/res4_1_branch2a_w_momentum (256, 1024, 1, 1, 1) I0830 19:10:39.106192 52234 checkpoints.py:377] res4_1_branch2b_w_momentum loaded from weights file into: gpu_0/res4_1_branch2b_w_momentum (256, 256, 1, 3, 3) I0830 19:10:39.107131 52234 checkpoints.py:377] res4_1_branch2c_w_momentum loaded from weights file into: gpu_0/res4_1_branch2c_w_momentum (1024, 256, 1, 1, 1) I0830 19:10:39.107920 52234 checkpoints.py:377] nonlocal_conv4_1_theta_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_1_theta_w_momentum (512, 1024, 1, 1, 1) I0830 19:10:39.108627 52234 checkpoints.py:377] nonlocal_conv4_1_theta_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_1_theta_b_momentum (512,) I0830 19:10:39.109138 52234 checkpoints.py:377] nonlocal_conv4_1_phi_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_1_phi_w_momentum (512, 1024, 1, 1, 1) I0830 19:10:39.109839 52234 checkpoints.py:377] nonlocal_conv4_1_phi_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_1_phi_b_momentum (512,) I0830 19:10:39.110347 52234 checkpoints.py:377] nonlocal_conv4_1_g_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_1_g_w_momentum (512, 1024, 1, 1, 1) I0830 19:10:39.111028 52234 checkpoints.py:377] nonlocal_conv4_1_g_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_1_g_b_momentum (512,) I0830 19:10:39.111515 52234 checkpoints.py:377] nonlocal_conv4_1_out_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_1_out_w_momentum (1024, 512, 1, 1, 1) I0830 19:10:39.112183 52234 checkpoints.py:377] nonlocal_conv4_1_out_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_1_out_b_momentum (1024,) I0830 19:10:39.112803 52234 checkpoints.py:377] res4_2_branch2a_w_momentum loaded from weights file into: gpu_0/res4_2_branch2a_w_momentum (256, 1024, 3, 1, 1) I0830 19:10:39.113956 52234 checkpoints.py:377] res4_2_branch2b_w_momentum loaded from weights file into: gpu_0/res4_2_branch2b_w_momentum (256, 256, 1, 3, 3) I0830 19:10:39.114877 52234 checkpoints.py:377] res4_2_branch2c_w_momentum loaded from weights file into: gpu_0/res4_2_branch2c_w_momentum (1024, 256, 1, 1, 1) I0830 19:10:39.115578 52234 checkpoints.py:377] res4_3_branch2a_w_momentum loaded from weights file into: gpu_0/res4_3_branch2a_w_momentum (256, 1024, 1, 1, 1) I0830 19:10:39.116447 52234 checkpoints.py:377] res4_3_branch2b_w_momentum loaded from weights file into: gpu_0/res4_3_branch2b_w_momentum (256, 256, 1, 3, 3) I0830 19:10:39.117409 52234 checkpoints.py:377] res4_3_branch2c_w_momentum loaded from weights file into: gpu_0/res4_3_branch2c_w_momentum (1024, 256, 1, 1, 1) I0830 19:10:39.118231 52234 checkpoints.py:377] nonlocal_conv4_3_theta_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_3_theta_w_momentum (512, 1024, 1, 1, 1) I0830 19:10:39.118999 52234 checkpoints.py:377] nonlocal_conv4_3_theta_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_3_theta_b_momentum (512,) I0830 19:10:39.119513 52234 checkpoints.py:377] nonlocal_conv4_3_phi_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_3_phi_w_momentum (512, 1024, 1, 1, 1) I0830 19:10:39.120213 52234 checkpoints.py:377] nonlocal_conv4_3_phi_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_3_phi_b_momentum (512,) I0830 19:10:39.120776 52234 checkpoints.py:377] nonlocal_conv4_3_g_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_3_g_w_momentum (512, 1024, 1, 1, 1) I0830 19:10:39.121476 52234 checkpoints.py:377] nonlocal_conv4_3_g_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_3_g_b_momentum (512,) I0830 19:10:39.121979 52234 checkpoints.py:377] nonlocal_conv4_3_out_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_3_out_w_momentum (1024, 512, 1, 1, 1) I0830 19:10:39.122685 52234 checkpoints.py:377] nonlocal_conv4_3_out_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_3_out_b_momentum (1024,) I0830 19:10:39.123297 52234 checkpoints.py:377] res4_4_branch2a_w_momentum loaded from weights file into: gpu_0/res4_4_branch2a_w_momentum (256, 1024, 3, 1, 1) I0830 19:10:39.124453 52234 checkpoints.py:377] res4_4_branch2b_w_momentum loaded from weights file into: gpu_0/res4_4_branch2b_w_momentum (256, 256, 1, 3, 3) I0830 19:10:39.125355 52234 checkpoints.py:377] res4_4_branch2c_w_momentum loaded from weights file into: gpu_0/res4_4_branch2c_w_momentum (1024, 256, 1, 1, 1) I0830 19:10:39.126040 52234 checkpoints.py:377] res4_5_branch2a_w_momentum loaded from weights file into: gpu_0/res4_5_branch2a_w_momentum (256, 1024, 1, 1, 1) I0830 19:10:39.126856 52234 checkpoints.py:377] res4_5_branch2b_w_momentum loaded from weights file into: gpu_0/res4_5_branch2b_w_momentum (256, 256, 1, 3, 3) I0830 19:10:39.127815 52234 checkpoints.py:377] res4_5_branch2c_w_momentum loaded from weights file into: gpu_0/res4_5_branch2c_w_momentum (1024, 256, 1, 1, 1) I0830 19:10:39.128628 52234 checkpoints.py:377] nonlocal_conv4_5_theta_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_5_theta_w_momentum (512, 1024, 1, 1, 1) I0830 19:10:39.129412 52234 checkpoints.py:377] nonlocal_conv4_5_theta_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_5_theta_b_momentum (512,) I0830 19:10:39.129945 52234 checkpoints.py:377] nonlocal_conv4_5_phi_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_5_phi_w_momentum (512, 1024, 1, 1, 1) I0830 19:10:39.130665 52234 checkpoints.py:377] nonlocal_conv4_5_phi_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_5_phi_b_momentum (512,) I0830 19:10:39.131175 52234 checkpoints.py:377] nonlocal_conv4_5_g_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_5_g_w_momentum (512, 1024, 1, 1, 1) I0830 19:10:39.131908 52234 checkpoints.py:377] nonlocal_conv4_5_g_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_5_g_b_momentum (512,) I0830 19:10:39.132467 52234 checkpoints.py:377] nonlocal_conv4_5_out_w_momentum loaded from weights file into: gpu_0/nonlocal_conv4_5_out_w_momentum (1024, 512, 1, 1, 1) I0830 19:10:39.133141 52234 checkpoints.py:377] nonlocal_conv4_5_out_b_momentum loaded from weights file into: gpu_0/nonlocal_conv4_5_out_b_momentum (1024,) I0830 19:10:39.133673 52234 checkpoints.py:377] res5_0_branch2a_w_momentum loaded from weights file into: gpu_0/res5_0_branch2a_w_momentum (512, 1024, 1, 1, 1) I0830 19:10:39.135302 52234 checkpoints.py:377] res5_0_branch2b_w_momentum loaded from weights file into: gpu_0/res5_0_branch2b_w_momentum (512, 512, 1, 3, 3) I0830 19:10:39.137653 52234 checkpoints.py:377] res5_0_branch2c_w_momentum loaded from weights file into: gpu_0/res5_0_branch2c_w_momentum (2048, 512, 1, 1, 1) I0830 19:10:39.139499 52234 checkpoints.py:377] res5_0_branch1_w_momentum loaded from weights file into: gpu_0/res5_0_branch1_w_momentum (2048, 1024, 1, 1, 1) I0830 19:10:39.147516 52234 checkpoints.py:377] res5_1_branch2a_w_momentum loaded from weights file into: gpu_0/res5_1_branch2a_w_momentum (512, 2048, 3, 1, 1) I0830 19:10:39.150810 52234 checkpoints.py:377] res5_1_branch2b_w_momentum loaded from weights file into: gpu_0/res5_1_branch2b_w_momentum (512, 512, 1, 3, 3) I0830 19:10:39.153089 52234 checkpoints.py:377] res5_1_branch2c_w_momentum loaded from weights file into: gpu_0/res5_1_branch2c_w_momentum (2048, 512, 1, 1, 1) I0830 19:10:39.154673 52234 checkpoints.py:377] res5_2_branch2a_w_momentum loaded from weights file into: gpu_0/res5_2_branch2a_w_momentum (512, 2048, 1, 1, 1) I0830 19:10:39.156618 52234 checkpoints.py:377] res5_2_branch2b_w_momentum loaded from weights file into: gpu_0/res5_2_branch2b_w_momentum (512, 512, 1, 3, 3) I0830 19:10:39.159029 52234 checkpoints.py:377] res5_2_branch2c_w_momentum loaded from weights file into: gpu_0/res5_2_branch2c_w_momentum (2048, 512, 1, 1, 1) I0830 19:10:39.160852 52234 checkpoints.py:377] pred_w_momentum loaded from weights file into: gpu_0/pred_w_momentum (80, 2048) I0830 19:10:39.161456 52234 checkpoints.py:377] pred_b_momentum loaded from weights file into: gpu_0/pred_b_momentum (80,) I0830 19:10:39.161772 52234 checkpoints.py:377] conv1_w loaded from weights file into: gpu_0/conv1_w (64, 3, 5, 7, 7) I0830 19:10:39.162143 52234 checkpoints.py:377] res_conv1_bn_s loaded from weights file into: gpu_0/res_conv1_bn_s (64,) I0830 19:10:39.162415 52234 checkpoints.py:377] res_conv1_bn_b loaded from weights file into: gpu_0/res_conv1_bn_b (64,) I0830 19:10:39.162650 52234 checkpoints.py:377] res2_0_branch2a_w loaded from weights file into: gpu_0/res2_0_branch2a_w (64, 64, 3, 1, 1) I0830 19:10:39.162910 52234 checkpoints.py:377] res2_0_branch2a_bn_s loaded from weights file into: gpu_0/res2_0_branch2a_bn_s (64,) I0830 19:10:39.163137 52234 checkpoints.py:377] res2_0_branch2a_bn_b loaded from weights file into: gpu_0/res2_0_branch2a_bn_b (64,) I0830 19:10:39.163391 52234 checkpoints.py:377] res2_0_branch2b_w loaded from weights file into: gpu_0/res2_0_branch2b_w (64, 64, 1, 3, 3) I0830 19:10:39.163729 52234 checkpoints.py:377] res2_0_branch2b_bn_s loaded from weights file into: gpu_0/res2_0_branch2b_bn_s (64,) I0830 19:10:39.163996 52234 checkpoints.py:377] res2_0_branch2b_bn_b loaded from weights file into: gpu_0/res2_0_branch2b_bn_b (64,) I0830 19:10:39.164252 52234 checkpoints.py:377] res2_0_branch2c_w loaded from weights file into: gpu_0/res2_0_branch2c_w (256, 64, 1, 1, 1) I0830 19:10:39.164511 52234 checkpoints.py:377] res2_0_branch2c_bn_s loaded from weights file into: gpu_0/res2_0_branch2c_bn_s (256,) I0830 19:10:39.164741 52234 checkpoints.py:377] res2_0_branch2c_bn_b loaded from weights file into: gpu_0/res2_0_branch2c_bn_b (256,) I0830 19:10:39.164973 52234 checkpoints.py:377] res2_0_branch1_w loaded from weights file into: gpu_0/res2_0_branch1_w (256, 64, 1, 1, 1) I0830 19:10:39.165240 52234 checkpoints.py:377] res2_0_branch1_bn_s loaded from weights file into: gpu_0/res2_0_branch1_bn_s (256,) I0830 19:10:39.165472 52234 checkpoints.py:377] res2_0_branch1_bn_b loaded from weights file into: gpu_0/res2_0_branch1_bn_b (256,) I0830 19:10:39.165745 52234 checkpoints.py:377] res2_1_branch2a_w loaded from weights file into: gpu_0/res2_1_branch2a_w (64, 256, 3, 1, 1) I0830 19:10:39.166029 52234 checkpoints.py:377] res2_1_branch2a_bn_s loaded from weights file into: gpu_0/res2_1_branch2a_bn_s (64,) I0830 19:10:39.166237 52234 checkpoints.py:377] res2_1_branch2a_bn_b loaded from weights file into: gpu_0/res2_1_branch2a_bn_b (64,) I0830 19:10:39.166477 52234 checkpoints.py:377] res2_1_branch2b_w loaded from weights file into: gpu_0/res2_1_branch2b_w (64, 64, 1, 3, 3) I0830 19:10:39.166733 52234 checkpoints.py:377] res2_1_branch2b_bn_s loaded from weights file into: gpu_0/res2_1_branch2b_bn_s (64,) I0830 19:10:39.166941 52234 checkpoints.py:377] res2_1_branch2b_bn_b loaded from weights file into: gpu_0/res2_1_branch2b_bn_b (64,) I0830 19:10:39.167186 52234 checkpoints.py:377] res2_1_branch2c_w loaded from weights file into: gpu_0/res2_1_branch2c_w (256, 64, 1, 1, 1) I0830 19:10:39.167475 52234 checkpoints.py:377] res2_1_branch2c_bn_s loaded from weights file into: gpu_0/res2_1_branch2c_bn_s (256,) I0830 19:10:39.167692 52234 checkpoints.py:377] res2_1_branch2c_bn_b loaded from weights file into: gpu_0/res2_1_branch2c_bn_b (256,) I0830 19:10:39.167965 52234 checkpoints.py:377] res2_2_branch2a_w loaded from weights file into: gpu_0/res2_2_branch2a_w (64, 256, 3, 1, 1) I0830 19:10:39.168253 52234 checkpoints.py:377] res2_2_branch2a_bn_s loaded from weights file into: gpu_0/res2_2_branch2a_bn_s (64,) I0830 19:10:39.168486 52234 checkpoints.py:377] res2_2_branch2a_bn_b loaded from weights file into: gpu_0/res2_2_branch2a_bn_b (64,) I0830 19:10:39.168723 52234 checkpoints.py:377] res2_2_branch2b_w loaded from weights file into: gpu_0/res2_2_branch2b_w (64, 64, 1, 3, 3) I0830 19:10:39.168999 52234 checkpoints.py:377] res2_2_branch2b_bn_s loaded from weights file into: gpu_0/res2_2_branch2b_bn_s (64,) I0830 19:10:39.169217 52234 checkpoints.py:377] res2_2_branch2b_bn_b loaded from weights file into: gpu_0/res2_2_branch2b_bn_b (64,) I0830 19:10:39.169473 52234 checkpoints.py:377] res2_2_branch2c_w loaded from weights file into: gpu_0/res2_2_branch2c_w (256, 64, 1, 1, 1) I0830 19:10:39.169780 52234 checkpoints.py:377] res2_2_branch2c_bn_s loaded from weights file into: gpu_0/res2_2_branch2c_bn_s (256,) I0830 19:10:39.170013 52234 checkpoints.py:377] res2_2_branch2c_bn_b loaded from weights file into: gpu_0/res2_2_branch2c_bn_b (256,) I0830 19:10:39.170296 52234 checkpoints.py:377] res3_0_branch2a_w loaded from weights file into: gpu_0/res3_0_branch2a_w (128, 256, 3, 1, 1) I0830 19:10:39.170618 52234 checkpoints.py:377] res3_0_branch2a_bn_s loaded from weights file into: gpu_0/res3_0_branch2a_bn_s (128,) I0830 19:10:39.170826 52234 checkpoints.py:377] res3_0_branch2a_bn_b loaded from weights file into: gpu_0/res3_0_branch2a_bn_b (128,) I0830 19:10:39.171154 52234 checkpoints.py:377] res3_0_branch2b_w loaded from weights file into: gpu_0/res3_0_branch2b_w (128, 128, 1, 3, 3) I0830 19:10:39.171585 52234 checkpoints.py:377] res3_0_branch2b_bn_s loaded from weights file into: gpu_0/res3_0_branch2b_bn_s (128,) I0830 19:10:39.171850 52234 checkpoints.py:377] res3_0_branch2b_bn_b loaded from weights file into: gpu_0/res3_0_branch2b_bn_b (128,) I0830 19:10:39.172122 52234 checkpoints.py:377] res3_0_branch2c_w loaded from weights file into: gpu_0/res3_0_branch2c_w (512, 128, 1, 1, 1) I0830 19:10:39.172441 52234 checkpoints.py:377] res3_0_branch2c_bn_s loaded from weights file into: gpu_0/res3_0_branch2c_bn_s (512,) I0830 19:10:39.172688 52234 checkpoints.py:377] res3_0_branch2c_bn_b loaded from weights file into: gpu_0/res3_0_branch2c_bn_b (512,) I0830 19:10:39.172991 52234 checkpoints.py:377] res3_0_branch1_w loaded from weights file into: gpu_0/res3_0_branch1_w (512, 256, 1, 1, 1) I0830 19:10:39.173377 52234 checkpoints.py:377] res3_0_branch1_bn_s loaded from weights file into: gpu_0/res3_0_branch1_bn_s (512,) I0830 19:10:39.173610 52234 checkpoints.py:377] res3_0_branch1_bn_b loaded from weights file into: gpu_0/res3_0_branch1_bn_b (512,) I0830 19:10:39.173871 52234 checkpoints.py:377] res3_1_branch2a_w loaded from weights file into: gpu_0/res3_1_branch2a_w (128, 512, 1, 1, 1) I0830 19:10:39.174232 52234 checkpoints.py:377] res3_1_branch2a_bn_s loaded from weights file into: gpu_0/res3_1_branch2a_bn_s (128,) I0830 19:10:39.174503 52234 checkpoints.py:377] res3_1_branch2a_bn_b loaded from weights file into: gpu_0/res3_1_branch2a_bn_b (128,) I0830 19:10:39.174818 52234 checkpoints.py:377] res3_1_branch2b_w loaded from weights file into: gpu_0/res3_1_branch2b_w (128, 128, 1, 3, 3) I0830 19:10:39.175205 52234 checkpoints.py:377] res3_1_branch2b_bn_s loaded from weights file into: gpu_0/res3_1_branch2b_bn_s (128,) I0830 19:10:39.175460 52234 checkpoints.py:377] res3_1_branch2b_bn_b loaded from weights file into: gpu_0/res3_1_branch2b_bn_b (128,) I0830 19:10:39.175738 52234 checkpoints.py:377] res3_1_branch2c_w loaded from weights file into: gpu_0/res3_1_branch2c_w (512, 128, 1, 1, 1) I0830 19:10:39.176060 52234 checkpoints.py:377] res3_1_branch2c_bn_s loaded from weights file into: gpu_0/res3_1_branch2c_bn_s (512,) I0830 19:10:39.176279 52234 checkpoints.py:377] res3_1_branch2c_bn_b loaded from weights file into: gpu_0/res3_1_branch2c_bn_b (512,) I0830 19:10:39.176582 52234 checkpoints.py:377] nonlocal_conv3_1_theta_w loaded from weights file into: gpu_0/nonlocal_conv3_1_theta_w (256, 512, 1, 1, 1) I0830 19:10:39.176929 52234 checkpoints.py:377] nonlocal_conv3_1_theta_b loaded from weights file into: gpu_0/nonlocal_conv3_1_theta_b (256,) I0830 19:10:39.177225 52234 checkpoints.py:377] nonlocal_conv3_1_phi_w loaded from weights file into: gpu_0/nonlocal_conv3_1_phi_w (256, 512, 1, 1, 1) I0830 19:10:39.177603 52234 checkpoints.py:377] nonlocal_conv3_1_phi_b loaded from weights file into: gpu_0/nonlocal_conv3_1_phi_b (256,) I0830 19:10:39.177911 52234 checkpoints.py:377] nonlocal_conv3_1_g_w loaded from weights file into: gpu_0/nonlocal_conv3_1_g_w (256, 512, 1, 1, 1) I0830 19:10:39.178311 52234 checkpoints.py:377] nonlocal_conv3_1_g_b loaded from weights file into: gpu_0/nonlocal_conv3_1_g_b (256,) I0830 19:10:39.178605 52234 checkpoints.py:377] nonlocal_conv3_1_out_w loaded from weights file into: gpu_0/nonlocal_conv3_1_out_w (512, 256, 1, 1, 1) I0830 19:10:39.178946 52234 checkpoints.py:377] nonlocal_conv3_1_out_b loaded from weights file into: gpu_0/nonlocal_conv3_1_out_b (512,) I0830 19:10:39.179157 52234 checkpoints.py:377] nonlocal_conv3_1_bn_s loaded from weights file into: gpu_0/nonlocal_conv3_1_bn_s (512,) I0830 19:10:39.179387 52234 checkpoints.py:377] nonlocal_conv3_1_bn_b loaded from weights file into: gpu_0/nonlocal_conv3_1_bn_b (512,) I0830 19:10:39.179755 52234 checkpoints.py:377] res3_2_branch2a_w loaded from weights file into: gpu_0/res3_2_branch2a_w (128, 512, 3, 1, 1) I0830 19:10:39.180193 52234 checkpoints.py:377] res3_2_branch2a_bn_s loaded from weights file into: gpu_0/res3_2_branch2a_bn_s (128,) I0830 19:10:39.180496 52234 checkpoints.py:377] res3_2_branch2a_bn_b loaded from weights file into: gpu_0/res3_2_branch2a_bn_b (128,) I0830 19:10:39.180843 52234 checkpoints.py:377] res3_2_branch2b_w loaded from weights file into: gpu_0/res3_2_branch2b_w (128, 128, 1, 3, 3) I0830 19:10:39.181235 52234 checkpoints.py:377] res3_2_branch2b_bn_s loaded from weights file into: gpu_0/res3_2_branch2b_bn_s (128,) I0830 19:10:39.181479 52234 checkpoints.py:377] res3_2_branch2b_bn_b loaded from weights file into: gpu_0/res3_2_branch2b_bn_b (128,) I0830 19:10:39.181766 52234 checkpoints.py:377] res3_2_branch2c_w loaded from weights file into: gpu_0/res3_2_branch2c_w (512, 128, 1, 1, 1) I0830 19:10:39.182148 52234 checkpoints.py:377] res3_2_branch2c_bn_s loaded from weights file into: gpu_0/res3_2_branch2c_bn_s (512,) I0830 19:10:39.182421 52234 checkpoints.py:377] res3_2_branch2c_bn_b loaded from weights file into: gpu_0/res3_2_branch2c_bn_b (512,) I0830 19:10:39.182701 52234 checkpoints.py:377] res3_3_branch2a_w loaded from weights file into: gpu_0/res3_3_branch2a_w (128, 512, 1, 1, 1) I0830 19:10:39.182996 52234 checkpoints.py:377] res3_3_branch2a_bn_s loaded from weights file into: gpu_0/res3_3_branch2a_bn_s (128,) I0830 19:10:39.183231 52234 checkpoints.py:377] res3_3_branch2a_bn_b loaded from weights file into: gpu_0/res3_3_branch2a_bn_b (128,) I0830 19:10:39.183569 52234 checkpoints.py:377] res3_3_branch2b_w loaded from weights file into: gpu_0/res3_3_branch2b_w (128, 128, 1, 3, 3) I0830 19:10:39.184128 52234 checkpoints.py:377] res3_3_branch2b_bn_s loaded from weights file into: gpu_0/res3_3_branch2b_bn_s (128,) I0830 19:10:39.184417 52234 checkpoints.py:377] res3_3_branch2b_bn_b loaded from weights file into: gpu_0/res3_3_branch2b_bn_b (128,) I0830 19:10:39.184735 52234 checkpoints.py:377] res3_3_branch2c_w loaded from weights file into: gpu_0/res3_3_branch2c_w (512, 128, 1, 1, 1) I0830 19:10:39.185086 52234 checkpoints.py:377] res3_3_branch2c_bn_s loaded from weights file into: gpu_0/res3_3_branch2c_bn_s (512,) I0830 19:10:39.185508 52234 checkpoints.py:377] res3_3_branch2c_bn_b loaded from weights file into: gpu_0/res3_3_branch2c_bn_b (512,) I0830 19:10:39.185931 52234 checkpoints.py:377] nonlocal_conv3_3_theta_w loaded from weights file into: gpu_0/nonlocal_conv3_3_theta_w (256, 512, 1, 1, 1) I0830 19:10:39.186331 52234 checkpoints.py:377] nonlocal_conv3_3_theta_b loaded from weights file into: gpu_0/nonlocal_conv3_3_theta_b (256,) I0830 19:10:39.186650 52234 checkpoints.py:377] nonlocal_conv3_3_phi_w loaded from weights file into: gpu_0/nonlocal_conv3_3_phi_w (256, 512, 1, 1, 1) I0830 19:10:39.187021 52234 checkpoints.py:377] nonlocal_conv3_3_phi_b loaded from weights file into: gpu_0/nonlocal_conv3_3_phi_b (256,) I0830 19:10:39.187325 52234 checkpoints.py:377] nonlocal_conv3_3_g_w loaded from weights file into: gpu_0/nonlocal_conv3_3_g_w (256, 512, 1, 1, 1) I0830 19:10:39.187669 52234 checkpoints.py:377] nonlocal_conv3_3_g_b loaded from weights file into: gpu_0/nonlocal_conv3_3_g_b (256,) I0830 19:10:39.187968 52234 checkpoints.py:377] nonlocal_conv3_3_out_w loaded from weights file into: gpu_0/nonlocal_conv3_3_out_w (512, 256, 1, 1, 1) I0830 19:10:39.188325 52234 checkpoints.py:377] nonlocal_conv3_3_out_b loaded from weights file into: gpu_0/nonlocal_conv3_3_out_b (512,) I0830 19:10:39.188555 52234 checkpoints.py:377] nonlocal_conv3_3_bn_s loaded from weights file into: gpu_0/nonlocal_conv3_3_bn_s (512,) I0830 19:10:39.188817 52234 checkpoints.py:377] nonlocal_conv3_3_bn_b loaded from weights file into: gpu_0/nonlocal_conv3_3_bn_b (512,) I0830 19:10:39.189270 52234 checkpoints.py:377] res4_0_branch2a_w loaded from weights file into: gpu_0/res4_0_branch2a_w (256, 512, 3, 1, 1) I0830 19:10:39.189873 52234 checkpoints.py:377] res4_0_branch2a_bn_s loaded from weights file into: gpu_0/res4_0_branch2a_bn_s (256,) I0830 19:10:39.190100 52234 checkpoints.py:377] res4_0_branch2a_bn_b loaded from weights file into: gpu_0/res4_0_branch2a_bn_b (256,) I0830 19:10:39.190675 52234 checkpoints.py:377] res4_0_branch2b_w loaded from weights file into: gpu_0/res4_0_branch2b_w (256, 256, 1, 3, 3) I0830 19:10:39.191417 52234 checkpoints.py:377] res4_0_branch2b_bn_s loaded from weights file into: gpu_0/res4_0_branch2b_bn_s (256,) I0830 19:10:39.191668 52234 checkpoints.py:377] res4_0_branch2b_bn_b loaded from weights file into: gpu_0/res4_0_branch2b_bn_b (256,) I0830 19:10:39.192061 52234 checkpoints.py:377] res4_0_branch2c_w loaded from weights file into: gpu_0/res4_0_branch2c_w (1024, 256, 1, 1, 1) I0830 19:10:39.192622 52234 checkpoints.py:377] res4_0_branch2c_bn_s loaded from weights file into: gpu_0/res4_0_branch2c_bn_s (1024,) I0830 19:10:39.192869 52234 checkpoints.py:377] res4_0_branch2c_bn_b loaded from weights file into: gpu_0/res4_0_branch2c_bn_b (1024,) I0830 19:10:39.193377 52234 checkpoints.py:377] res4_0_branch1_w loaded from weights file into: gpu_0/res4_0_branch1_w (1024, 512, 1, 1, 1) I0830 19:10:39.194119 52234 checkpoints.py:377] res4_0_branch1_bn_s loaded from weights file into: gpu_0/res4_0_branch1_bn_s (1024,) I0830 19:10:39.194360 52234 checkpoints.py:377] res4_0_branch1_bn_b loaded from weights file into: gpu_0/res4_0_branch1_bn_b (1024,) I0830 19:10:39.194776 52234 checkpoints.py:377] res4_1_branch2a_w loaded from weights file into: gpu_0/res4_1_branch2a_w (256, 1024, 1, 1, 1) I0830 19:10:39.195293 52234 checkpoints.py:377] res4_1_branch2a_bn_s loaded from weights file into: gpu_0/res4_1_branch2a_bn_s (256,) I0830 19:10:39.195565 52234 checkpoints.py:377] res4_1_branch2a_bn_b loaded from weights file into: gpu_0/res4_1_branch2a_bn_b (256,) I0830 19:10:39.196114 52234 checkpoints.py:377] res4_1_branch2b_w loaded from weights file into: gpu_0/res4_1_branch2b_w (256, 256, 1, 3, 3) I0830 19:10:39.196873 52234 checkpoints.py:377] res4_1_branch2b_bn_s loaded from weights file into: gpu_0/res4_1_branch2b_bn_s (256,) I0830 19:10:39.197120 52234 checkpoints.py:377] res4_1_branch2b_bn_b loaded from weights file into: gpu_0/res4_1_branch2b_bn_b (256,) I0830 19:10:39.197559 52234 checkpoints.py:377] res4_1_branch2c_w loaded from weights file into: gpu_0/res4_1_branch2c_w (1024, 256, 1, 1, 1) I0830 19:10:39.198067 52234 checkpoints.py:377] res4_1_branch2c_bn_s loaded from weights file into: gpu_0/res4_1_branch2c_bn_s (1024,) I0830 19:10:39.198302 52234 checkpoints.py:377] res4_1_branch2c_bn_b loaded from weights file into: gpu_0/res4_1_branch2c_bn_b (1024,) I0830 19:10:39.198806 52234 checkpoints.py:377] nonlocal_conv4_1_theta_w loaded from weights file into: gpu_0/nonlocal_conv4_1_theta_w (512, 1024, 1, 1, 1) I0830 19:10:39.199491 52234 checkpoints.py:377] nonlocal_conv4_1_theta_b loaded from weights file into: gpu_0/nonlocal_conv4_1_theta_b (512,) I0830 19:10:39.200032 52234 checkpoints.py:377] nonlocal_conv4_1_phi_w loaded from weights file into: gpu_0/nonlocal_conv4_1_phi_w (512, 1024, 1, 1, 1) I0830 19:10:39.200715 52234 checkpoints.py:377] nonlocal_conv4_1_phi_b loaded from weights file into: gpu_0/nonlocal_conv4_1_phi_b (512,) I0830 19:10:39.201255 52234 checkpoints.py:377] nonlocal_conv4_1_g_w loaded from weights file into: gpu_0/nonlocal_conv4_1_g_w (512, 1024, 1, 1, 1) I0830 19:10:39.201940 52234 checkpoints.py:377] nonlocal_conv4_1_g_b loaded from weights file into: gpu_0/nonlocal_conv4_1_g_b (512,) I0830 19:10:39.202502 52234 checkpoints.py:377] nonlocal_conv4_1_out_w loaded from weights file into: gpu_0/nonlocal_conv4_1_out_w (1024, 512, 1, 1, 1) I0830 19:10:39.203252 52234 checkpoints.py:377] nonlocal_conv4_1_out_b loaded from weights file into: gpu_0/nonlocal_conv4_1_out_b (1024,) I0830 19:10:39.203501 52234 checkpoints.py:377] nonlocal_conv4_1_bn_s loaded from weights file into: gpu_0/nonlocal_conv4_1_bn_s (1024,) I0830 19:10:39.203794 52234 checkpoints.py:377] nonlocal_conv4_1_bn_b loaded from weights file into: gpu_0/nonlocal_conv4_1_bn_b (1024,) I0830 19:10:39.204401 52234 checkpoints.py:377] res4_2_branch2a_w loaded from weights file into: gpu_0/res4_2_branch2a_w (256, 1024, 3, 1, 1) I0830 19:10:39.205292 52234 checkpoints.py:377] res4_2_branch2a_bn_s loaded from weights file into: gpu_0/res4_2_branch2a_bn_s (256,) I0830 19:10:39.205545 52234 checkpoints.py:377] res4_2_branch2a_bn_b loaded from weights file into: gpu_0/res4_2_branch2a_bn_b (256,) I0830 19:10:39.206096 52234 checkpoints.py:377] res4_2_branch2b_w loaded from weights file into: gpu_0/res4_2_branch2b_w (256, 256, 1, 3, 3) I0830 19:10:39.206825 52234 checkpoints.py:377] res4_2_branch2b_bn_s loaded from weights file into: gpu_0/res4_2_branch2b_bn_s (256,) I0830 19:10:39.207075 52234 checkpoints.py:377] res4_2_branch2b_bn_b loaded from weights file into: gpu_0/res4_2_branch2b_bn_b (256,) I0830 19:10:39.207596 52234 checkpoints.py:377] res4_2_branch2c_w loaded from weights file into: gpu_0/res4_2_branch2c_w (1024, 256, 1, 1, 1) I0830 19:10:39.208129 52234 checkpoints.py:377] res4_2_branch2c_bn_s loaded from weights file into: gpu_0/res4_2_branch2c_bn_s (1024,) I0830 19:10:39.208384 52234 checkpoints.py:377] res4_2_branch2c_bn_b loaded from weights file into: gpu_0/res4_2_branch2c_bn_b (1024,) I0830 19:10:39.208798 52234 checkpoints.py:377] res4_3_branch2a_w loaded from weights file into: gpu_0/res4_3_branch2a_w (256, 1024, 1, 1, 1) I0830 19:10:39.209316 52234 checkpoints.py:377] res4_3_branch2a_bn_s loaded from weights file into: gpu_0/res4_3_branch2a_bn_s (256,) I0830 19:10:39.209563 52234 checkpoints.py:377] res4_3_branch2a_bn_b loaded from weights file into: gpu_0/res4_3_branch2a_bn_b (256,) I0830 19:10:39.210078 52234 checkpoints.py:377] res4_3_branch2b_w loaded from weights file into: gpu_0/res4_3_branch2b_w (256, 256, 1, 3, 3) I0830 19:10:39.210829 52234 checkpoints.py:377] res4_3_branch2b_bn_s loaded from weights file into: gpu_0/res4_3_branch2b_bn_s (256,) I0830 19:10:39.211072 52234 checkpoints.py:377] res4_3_branch2b_bn_b loaded from weights file into: gpu_0/res4_3_branch2b_bn_b (256,) I0830 19:10:39.211489 52234 checkpoints.py:377] res4_3_branch2c_w loaded from weights file into: gpu_0/res4_3_branch2c_w (1024, 256, 1, 1, 1) I0830 19:10:39.212006 52234 checkpoints.py:377] res4_3_branch2c_bn_s loaded from weights file into: gpu_0/res4_3_branch2c_bn_s (1024,) I0830 19:10:39.212253 52234 checkpoints.py:377] res4_3_branch2c_bn_b loaded from weights file into: gpu_0/res4_3_branch2c_bn_b (1024,) I0830 19:10:39.212840 52234 checkpoints.py:377] nonlocal_conv4_3_theta_w loaded from weights file into: gpu_0/nonlocal_conv4_3_theta_w (512, 1024, 1, 1, 1) I0830 19:10:39.213581 52234 checkpoints.py:377] nonlocal_conv4_3_theta_b loaded from weights file into: gpu_0/nonlocal_conv4_3_theta_b (512,) I0830 19:10:39.214083 52234 checkpoints.py:377] nonlocal_conv4_3_phi_w loaded from weights file into: gpu_0/nonlocal_conv4_3_phi_w (512, 1024, 1, 1, 1) I0830 19:10:39.214767 52234 checkpoints.py:377] nonlocal_conv4_3_phi_b loaded from weights file into: gpu_0/nonlocal_conv4_3_phi_b (512,) I0830 19:10:39.215312 52234 checkpoints.py:377] nonlocal_conv4_3_g_w loaded from weights file into: gpu_0/nonlocal_conv4_3_g_w (512, 1024, 1, 1, 1) I0830 19:10:39.216016 52234 checkpoints.py:377] nonlocal_conv4_3_g_b loaded from weights file into: gpu_0/nonlocal_conv4_3_g_b (512,) I0830 19:10:39.216520 52234 checkpoints.py:377] nonlocal_conv4_3_out_w loaded from weights file into: gpu_0/nonlocal_conv4_3_out_w (1024, 512, 1, 1, 1) I0830 19:10:39.217204 52234 checkpoints.py:377] nonlocal_conv4_3_out_b loaded from weights file into: gpu_0/nonlocal_conv4_3_out_b (1024,) I0830 19:10:39.217475 52234 checkpoints.py:377] nonlocal_conv4_3_bn_s loaded from weights file into: gpu_0/nonlocal_conv4_3_bn_s (1024,) I0830 19:10:39.217708 52234 checkpoints.py:377] nonlocal_conv4_3_bn_b loaded from weights file into: gpu_0/nonlocal_conv4_3_bn_b (1024,) I0830 19:10:39.218369 52234 checkpoints.py:377] res4_4_branch2a_w loaded from weights file into: gpu_0/res4_4_branch2a_w (256, 1024, 3, 1, 1) I0830 19:10:39.219244 52234 checkpoints.py:377] res4_4_branch2a_bn_s loaded from weights file into: gpu_0/res4_4_branch2a_bn_s (256,) I0830 19:10:39.219501 52234 checkpoints.py:377] res4_4_branch2a_bn_b loaded from weights file into: gpu_0/res4_4_branch2a_bn_b (256,) I0830 19:10:39.220058 52234 checkpoints.py:377] res4_4_branch2b_w loaded from weights file into: gpu_0/res4_4_branch2b_w (256, 256, 1, 3, 3) I0830 19:10:39.220762 52234 checkpoints.py:377] res4_4_branch2b_bn_s loaded from weights file into: gpu_0/res4_4_branch2b_bn_s (256,) I0830 19:10:39.221014 52234 checkpoints.py:377] res4_4_branch2b_bn_b loaded from weights file into: gpu_0/res4_4_branch2b_bn_b (256,) I0830 19:10:39.221446 52234 checkpoints.py:377] res4_4_branch2c_w loaded from weights file into: gpu_0/res4_4_branch2c_w (1024, 256, 1, 1, 1) I0830 19:10:39.221995 52234 checkpoints.py:377] res4_4_branch2c_bn_s loaded from weights file into: gpu_0/res4_4_branch2c_bn_s (1024,) I0830 19:10:39.222244 52234 checkpoints.py:377] res4_4_branch2c_bn_b loaded from weights file into: gpu_0/res4_4_branch2c_bn_b (1024,) I0830 19:10:39.222692 52234 checkpoints.py:377] res4_5_branch2a_w loaded from weights file into: gpu_0/res4_5_branch2a_w (256, 1024, 1, 1, 1) I0830 19:10:39.223246 52234 checkpoints.py:377] res4_5_branch2a_bn_s loaded from weights file into: gpu_0/res4_5_branch2a_bn_s (256,) I0830 19:10:39.223500 52234 checkpoints.py:377] res4_5_branch2a_bn_b loaded from weights file into: gpu_0/res4_5_branch2a_bn_b (256,) I0830 19:10:39.224065 52234 checkpoints.py:377] res4_5_branch2b_w loaded from weights file into: gpu_0/res4_5_branch2b_w (256, 256, 1, 3, 3) I0830 19:10:39.224813 52234 checkpoints.py:377] res4_5_branch2b_bn_s loaded from weights file into: gpu_0/res4_5_branch2b_bn_s (256,) I0830 19:10:39.225064 52234 checkpoints.py:377] res4_5_branch2b_bn_b loaded from weights file into: gpu_0/res4_5_branch2b_bn_b (256,) I0830 19:10:39.225519 52234 checkpoints.py:377] res4_5_branch2c_w loaded from weights file into: gpu_0/res4_5_branch2c_w (1024, 256, 1, 1, 1) I0830 19:10:39.226049 52234 checkpoints.py:377] res4_5_branch2c_bn_s loaded from weights file into: gpu_0/res4_5_branch2c_bn_s (1024,) I0830 19:10:39.226305 52234 checkpoints.py:377] res4_5_branch2c_bn_b loaded from weights file into: gpu_0/res4_5_branch2c_bn_b (1024,) I0830 19:10:39.226824 52234 checkpoints.py:377] nonlocal_conv4_5_theta_w loaded from weights file into: gpu_0/nonlocal_conv4_5_theta_w (512, 1024, 1, 1, 1) I0830 19:10:39.227519 52234 checkpoints.py:377] nonlocal_conv4_5_theta_b loaded from weights file into: gpu_0/nonlocal_conv4_5_theta_b (512,) I0830 19:10:39.228041 52234 checkpoints.py:377] nonlocal_conv4_5_phi_w loaded from weights file into: gpu_0/nonlocal_conv4_5_phi_w (512, 1024, 1, 1, 1) I0830 19:10:39.228796 52234 checkpoints.py:377] nonlocal_conv4_5_phi_b loaded from weights file into: gpu_0/nonlocal_conv4_5_phi_b (512,) I0830 19:10:39.229347 52234 checkpoints.py:377] nonlocal_conv4_5_g_w loaded from weights file into: gpu_0/nonlocal_conv4_5_g_w (512, 1024, 1, 1, 1) I0830 19:10:39.230037 52234 checkpoints.py:377] nonlocal_conv4_5_g_b loaded from weights file into: gpu_0/nonlocal_conv4_5_g_b (512,) I0830 19:10:39.230545 52234 checkpoints.py:377] nonlocal_conv4_5_out_w loaded from weights file into: gpu_0/nonlocal_conv4_5_out_w (1024, 512, 1, 1, 1) I0830 19:10:39.231220 52234 checkpoints.py:377] nonlocal_conv4_5_out_b loaded from weights file into: gpu_0/nonlocal_conv4_5_out_b (1024,) I0830 19:10:39.231455 52234 checkpoints.py:377] nonlocal_conv4_5_bn_s loaded from weights file into: gpu_0/nonlocal_conv4_5_bn_s (1024,) I0830 19:10:39.231709 52234 checkpoints.py:377] nonlocal_conv4_5_bn_b loaded from weights file into: gpu_0/nonlocal_conv4_5_bn_b (1024,) I0830 19:10:39.232249 52234 checkpoints.py:377] res5_0_branch2a_w loaded from weights file into: gpu_0/res5_0_branch2a_w (512, 1024, 1, 1, 1) I0830 19:10:39.232940 52234 checkpoints.py:377] res5_0_branch2a_bn_s loaded from weights file into: gpu_0/res5_0_branch2a_bn_s (512,) I0830 19:10:39.233192 52234 checkpoints.py:377] res5_0_branch2a_bn_b loaded from weights file into: gpu_0/res5_0_branch2a_bn_b (512,) I0830 19:10:39.234406 52234 checkpoints.py:377] res5_0_branch2b_w loaded from weights file into: gpu_0/res5_0_branch2b_w (512, 512, 1, 3, 3) I0830 19:10:39.236259 52234 checkpoints.py:377] res5_0_branch2b_bn_s loaded from weights file into: gpu_0/res5_0_branch2b_bn_s (512,) I0830 19:10:39.236522 52234 checkpoints.py:377] res5_0_branch2b_bn_b loaded from weights file into: gpu_0/res5_0_branch2b_bn_b (512,) I0830 19:10:39.237299 52234 checkpoints.py:377] res5_0_branch2c_w loaded from weights file into: gpu_0/res5_0_branch2c_w (2048, 512, 1, 1, 1) I0830 19:10:39.238352 52234 checkpoints.py:377] res5_0_branch2c_bn_s loaded from weights file into: gpu_0/res5_0_branch2c_bn_s (2048,) I0830 19:10:39.238630 52234 checkpoints.py:377] res5_0_branch2c_bn_b loaded from weights file into: gpu_0/res5_0_branch2c_bn_b (2048,) I0830 19:10:39.239759 52234 checkpoints.py:377] res5_0_branch1_w loaded from weights file into: gpu_0/res5_0_branch1_w (2048, 1024, 1, 1, 1) I0830 19:10:39.241494 52234 checkpoints.py:377] res5_0_branch1_bn_s loaded from weights file into: gpu_0/res5_0_branch1_bn_s (2048,) I0830 19:10:39.241750 52234 checkpoints.py:377] res5_0_branch1_bn_b loaded from weights file into: gpu_0/res5_0_branch1_bn_b (2048,) I0830 19:10:39.243195 52234 checkpoints.py:377] res5_1_branch2a_w loaded from weights file into: gpu_0/res5_1_branch2a_w (512, 2048, 3, 1, 1) I0830 19:10:39.245608 52234 checkpoints.py:377] res5_1_branch2a_bn_s loaded from weights file into: gpu_0/res5_1_branch2a_bn_s (512,) I0830 19:10:39.245913 52234 checkpoints.py:377] res5_1_branch2a_bn_b loaded from weights file into: gpu_0/res5_1_branch2a_bn_b (512,) I0830 19:10:39.247059 52234 checkpoints.py:377] res5_1_branch2b_w loaded from weights file into: gpu_0/res5_1_branch2b_w (512, 512, 1, 3, 3) I0830 19:10:39.248929 52234 checkpoints.py:377] res5_1_branch2b_bn_s loaded from weights file into: gpu_0/res5_1_branch2b_bn_s (512,) I0830 19:10:39.249190 52234 checkpoints.py:377] res5_1_branch2b_bn_b loaded from weights file into: gpu_0/res5_1_branch2b_bn_b (512,) I0830 19:10:39.249892 52234 checkpoints.py:377] res5_1_branch2c_w loaded from weights file into: gpu_0/res5_1_branch2c_w (2048, 512, 1, 1, 1) I0830 19:10:39.250961 52234 checkpoints.py:377] res5_1_branch2c_bn_s loaded from weights file into: gpu_0/res5_1_branch2c_bn_s (2048,) I0830 19:10:39.251276 52234 checkpoints.py:377] res5_1_branch2c_bn_b loaded from weights file into: gpu_0/res5_1_branch2c_bn_b (2048,) I0830 19:10:39.252002 52234 checkpoints.py:377] res5_2_branch2a_w loaded from weights file into: gpu_0/res5_2_branch2a_w (512, 2048, 1, 1, 1) I0830 19:10:39.253053 52234 checkpoints.py:377] res5_2_branch2a_bn_s loaded from weights file into: gpu_0/res5_2_branch2a_bn_s (512,) I0830 19:10:39.253338 52234 checkpoints.py:377] res5_2_branch2a_bn_b loaded from weights file into: gpu_0/res5_2_branch2a_bn_b (512,) I0830 19:10:39.254529 52234 checkpoints.py:377] res5_2_branch2b_w loaded from weights file into: gpu_0/res5_2_branch2b_w (512, 512, 1, 3, 3) I0830 19:10:39.256372 52234 checkpoints.py:377] res5_2_branch2b_bn_s loaded from weights file into: gpu_0/res5_2_branch2b_bn_s (512,) I0830 19:10:39.256669 52234 checkpoints.py:377] res5_2_branch2b_bn_b loaded from weights file into: gpu_0/res5_2_branch2b_bn_b (512,) I0830 19:10:39.257407 52234 checkpoints.py:377] res5_2_branch2c_w loaded from weights file into: gpu_0/res5_2_branch2c_w (2048, 512, 1, 1, 1) I0830 19:10:39.258481 52234 checkpoints.py:377] res5_2_branch2c_bn_s loaded from weights file into: gpu_0/res5_2_branch2c_bn_s (2048,) I0830 19:10:39.258759 52234 checkpoints.py:377] res5_2_branch2c_bn_b loaded from weights file into: gpu_0/res5_2_branch2c_bn_b (2048,) I0830 19:10:39.259224 52234 checkpoints.py:377] pred_w loaded from weights file into: gpu_0/pred_w (80, 2048) I0830 19:10:39.259680 52234 checkpoints.py:377] pred_b loaded from weights file into: gpu_0/pred_b (80,) I0830 19:10:39.592730 52234 checkpoints.py:410] Broadcasting gpu_0/conv1_w to I0830 19:10:39.593101 52234 checkpoints.py:415] |-> gpu_1/conv1_w I0830 19:10:39.593489 52234 checkpoints.py:415] |-> gpu_2/conv1_w I0830 19:10:39.593792 52234 checkpoints.py:415] |-> gpu_3/conv1_w I0830 19:10:39.594176 52234 checkpoints.py:415] |-> gpu_4/conv1_w I0830 19:10:39.594506 52234 checkpoints.py:415] |-> gpu_5/conv1_w I0830 19:10:39.594774 52234 checkpoints.py:415] |-> gpu_6/conv1_w I0830 19:10:39.595016 52234 checkpoints.py:415] |-> gpu_7/conv1_w I0830 19:10:39.595339 52234 checkpoints.py:410] Broadcasting gpu_0/res_conv1_bn_s to I0830 19:10:39.595480 52234 checkpoints.py:415] |-> gpu_1/res_conv1_bn_s I0830 19:10:39.595663 52234 checkpoints.py:415] |-> gpu_2/res_conv1_bn_s I0830 19:10:39.595860 52234 checkpoints.py:415] |-> gpu_3/res_conv1_bn_s I0830 19:10:39.596035 52234 checkpoints.py:415] |-> gpu_4/res_conv1_bn_s I0830 19:10:39.596210 52234 checkpoints.py:415] |-> gpu_5/res_conv1_bn_s I0830 19:10:39.596395 52234 checkpoints.py:415] |-> gpu_6/res_conv1_bn_s I0830 19:10:39.596597 52234 checkpoints.py:415] |-> gpu_7/res_conv1_bn_s I0830 19:10:39.596806 52234 checkpoints.py:410] Broadcasting gpu_0/res_conv1_bn_b to I0830 19:10:39.596906 52234 checkpoints.py:415] |-> gpu_1/res_conv1_bn_b I0830 19:10:39.597073 52234 checkpoints.py:415] |-> gpu_2/res_conv1_bn_b I0830 19:10:39.597265 52234 checkpoints.py:415] |-> gpu_3/res_conv1_bn_b I0830 19:10:39.597465 52234 checkpoints.py:415] |-> gpu_4/res_conv1_bn_b I0830 19:10:39.597635 52234 checkpoints.py:415] |-> gpu_5/res_conv1_bn_b I0830 19:10:39.597807 52234 checkpoints.py:415] |-> gpu_6/res_conv1_bn_b I0830 19:10:39.598005 52234 checkpoints.py:415] |-> gpu_7/res_conv1_bn_b I0830 19:10:39.598241 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2a_w to I0830 19:10:39.598357 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2a_w I0830 19:10:39.598548 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2a_w I0830 19:10:39.598754 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2a_w I0830 19:10:39.598947 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2a_w I0830 19:10:39.599138 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2a_w I0830 19:10:39.599335 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2a_w I0830 19:10:39.599526 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2a_w I0830 19:10:39.599750 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2a_bn_s to I0830 19:10:39.599853 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2a_bn_s I0830 19:10:39.600019 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2a_bn_s I0830 19:10:39.600193 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2a_bn_s I0830 19:10:39.600370 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2a_bn_s I0830 19:10:39.600564 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2a_bn_s I0830 19:10:39.600739 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2a_bn_s I0830 19:10:39.600908 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2a_bn_s I0830 19:10:39.601114 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2a_bn_b to I0830 19:10:39.601217 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2a_bn_b I0830 19:10:39.601409 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2a_bn_b I0830 19:10:39.601614 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2a_bn_b I0830 19:10:39.601788 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2a_bn_b I0830 19:10:39.601963 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2a_bn_b I0830 19:10:39.602155 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2a_bn_b I0830 19:10:39.602351 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2a_bn_b I0830 19:10:39.602591 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2b_w to I0830 19:10:39.602694 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2b_w I0830 19:10:39.602910 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2b_w I0830 19:10:39.603168 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2b_w I0830 19:10:39.603410 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2b_w I0830 19:10:39.603679 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2b_w I0830 19:10:39.603915 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2b_w I0830 19:10:39.604187 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2b_w I0830 19:10:39.604480 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2b_bn_s to I0830 19:10:39.604645 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2b_bn_s I0830 19:10:39.604897 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2b_bn_s I0830 19:10:39.605098 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2b_bn_s I0830 19:10:39.605272 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2b_bn_s I0830 19:10:39.605457 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2b_bn_s I0830 19:10:39.605628 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2b_bn_s I0830 19:10:39.605802 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2b_bn_s I0830 19:10:39.606043 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2b_bn_b to I0830 19:10:39.606143 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2b_bn_b I0830 19:10:39.606314 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2b_bn_b I0830 19:10:39.606506 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2b_bn_b I0830 19:10:39.606678 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2b_bn_b I0830 19:10:39.606848 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2b_bn_b I0830 19:10:39.607021 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2b_bn_b I0830 19:10:39.607209 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2b_bn_b I0830 19:10:39.607441 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2c_w to I0830 19:10:39.607551 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2c_w I0830 19:10:39.607745 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2c_w I0830 19:10:39.607945 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2c_w I0830 19:10:39.608177 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2c_w I0830 19:10:39.608402 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2c_w I0830 19:10:39.608613 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2c_w I0830 19:10:39.608812 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2c_w I0830 19:10:39.609042 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2c_bn_s to I0830 19:10:39.609146 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2c_bn_s I0830 19:10:39.609319 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2c_bn_s I0830 19:10:39.609493 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2c_bn_s I0830 19:10:39.609664 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2c_bn_s I0830 19:10:39.609837 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2c_bn_s I0830 19:10:39.610008 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2c_bn_s I0830 19:10:39.610179 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2c_bn_s I0830 19:10:39.610392 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2c_bn_b to I0830 19:10:39.610507 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2c_bn_b I0830 19:10:39.610676 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2c_bn_b I0830 19:10:39.610846 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2c_bn_b I0830 19:10:39.611016 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2c_bn_b I0830 19:10:39.611191 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2c_bn_b I0830 19:10:39.611390 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2c_bn_b I0830 19:10:39.611562 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2c_bn_b I0830 19:10:39.611802 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch1_w to I0830 19:10:39.611905 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch1_w I0830 19:10:39.612103 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch1_w I0830 19:10:39.612333 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch1_w I0830 19:10:39.612555 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch1_w I0830 19:10:39.612750 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch1_w I0830 19:10:39.612941 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch1_w I0830 19:10:39.613138 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch1_w I0830 19:10:39.613421 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch1_bn_s to I0830 19:10:39.613524 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch1_bn_s I0830 19:10:39.613692 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch1_bn_s I0830 19:10:39.613869 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch1_bn_s I0830 19:10:39.614058 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch1_bn_s I0830 19:10:39.614276 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch1_bn_s I0830 19:10:39.614491 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch1_bn_s I0830 19:10:39.614717 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch1_bn_s I0830 19:10:39.615041 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch1_bn_b to I0830 19:10:39.615210 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch1_bn_b I0830 19:10:39.615415 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch1_bn_b I0830 19:10:39.615603 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch1_bn_b I0830 19:10:39.615775 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch1_bn_b I0830 19:10:39.615950 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch1_bn_b I0830 19:10:39.616153 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch1_bn_b I0830 19:10:39.616354 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch1_bn_b I0830 19:10:39.616631 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2a_w to I0830 19:10:39.616731 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2a_w I0830 19:10:39.616959 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2a_w I0830 19:10:39.617175 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2a_w I0830 19:10:39.617398 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2a_w I0830 19:10:39.617619 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2a_w I0830 19:10:39.617851 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2a_w I0830 19:10:39.618067 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2a_w I0830 19:10:39.618343 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2a_bn_s to I0830 19:10:39.618463 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2a_bn_s I0830 19:10:39.618666 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2a_bn_s I0830 19:10:39.618841 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2a_bn_s I0830 19:10:39.619014 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2a_bn_s I0830 19:10:39.619184 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2a_bn_s I0830 19:10:39.619362 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2a_bn_s I0830 19:10:39.619533 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2a_bn_s I0830 19:10:39.619735 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2a_bn_b to I0830 19:10:39.619836 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2a_bn_b I0830 19:10:39.619998 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2a_bn_b I0830 19:10:39.620172 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2a_bn_b I0830 19:10:39.620349 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2a_bn_b I0830 19:10:39.620526 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2a_bn_b I0830 19:10:39.620732 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2a_bn_b I0830 19:10:39.620963 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2a_bn_b I0830 19:10:39.621226 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2b_w to I0830 19:10:39.621356 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2b_w I0830 19:10:39.621589 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2b_w I0830 19:10:39.621834 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2b_w I0830 19:10:39.622057 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2b_w I0830 19:10:39.622267 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2b_w I0830 19:10:39.622499 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2b_w I0830 19:10:39.622742 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2b_w I0830 19:10:39.622989 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2b_bn_s to I0830 19:10:39.623092 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2b_bn_s I0830 19:10:39.623260 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2b_bn_s I0830 19:10:39.623486 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2b_bn_s I0830 19:10:39.623677 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2b_bn_s I0830 19:10:39.623850 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2b_bn_s I0830 19:10:39.624020 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2b_bn_s I0830 19:10:39.624188 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2b_bn_s I0830 19:10:39.624401 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2b_bn_b to I0830 19:10:39.624514 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2b_bn_b I0830 19:10:39.624694 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2b_bn_b I0830 19:10:39.624921 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2b_bn_b I0830 19:10:39.625176 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2b_bn_b I0830 19:10:39.625403 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2b_bn_b I0830 19:10:39.625619 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2b_bn_b I0830 19:10:39.625854 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2b_bn_b I0830 19:10:39.626089 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2c_w to I0830 19:10:39.626231 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2c_w I0830 19:10:39.626456 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2c_w I0830 19:10:39.626692 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2c_w I0830 19:10:39.626931 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2c_w I0830 19:10:39.627156 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2c_w I0830 19:10:39.627377 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2c_w I0830 19:10:39.627573 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2c_w I0830 19:10:39.627804 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2c_bn_s to I0830 19:10:39.627928 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2c_bn_s I0830 19:10:39.628101 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2c_bn_s I0830 19:10:39.628304 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2c_bn_s I0830 19:10:39.628494 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2c_bn_s I0830 19:10:39.628674 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2c_bn_s I0830 19:10:39.628860 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2c_bn_s I0830 19:10:39.629049 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2c_bn_s I0830 19:10:39.629256 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2c_bn_b to I0830 19:10:39.629365 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2c_bn_b I0830 19:10:39.629530 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2c_bn_b I0830 19:10:39.629702 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2c_bn_b I0830 19:10:39.629873 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2c_bn_b I0830 19:10:39.630068 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2c_bn_b I0830 19:10:39.630242 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2c_bn_b I0830 19:10:39.630439 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2c_bn_b I0830 19:10:39.630692 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2a_w to I0830 19:10:39.630793 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2a_w I0830 19:10:39.631015 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2a_w I0830 19:10:39.631237 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2a_w I0830 19:10:39.631510 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2a_w I0830 19:10:39.631782 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2a_w I0830 19:10:39.632033 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2a_w I0830 19:10:39.632283 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2a_w I0830 19:10:39.632567 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2a_bn_s to I0830 19:10:39.632707 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2a_bn_s I0830 19:10:39.632926 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2a_bn_s I0830 19:10:39.633132 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2a_bn_s I0830 19:10:39.633336 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2a_bn_s I0830 19:10:39.633532 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2a_bn_s I0830 19:10:39.633724 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2a_bn_s I0830 19:10:39.633934 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2a_bn_s I0830 19:10:39.634156 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2a_bn_b to I0830 19:10:39.634258 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2a_bn_b I0830 19:10:39.634437 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2a_bn_b I0830 19:10:39.634624 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2a_bn_b I0830 19:10:39.634797 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2a_bn_b I0830 19:10:39.634979 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2a_bn_b I0830 19:10:39.635174 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2a_bn_b I0830 19:10:39.635391 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2a_bn_b I0830 19:10:39.635670 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2b_w to I0830 19:10:39.635840 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2b_w I0830 19:10:39.636075 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2b_w I0830 19:10:39.636349 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2b_w I0830 19:10:39.636600 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2b_w I0830 19:10:39.636827 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2b_w I0830 19:10:39.637053 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2b_w I0830 19:10:39.637324 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2b_w I0830 19:10:39.637633 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2b_bn_s to I0830 19:10:39.637758 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2b_bn_s I0830 19:10:39.637942 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2b_bn_s I0830 19:10:39.638114 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2b_bn_s I0830 19:10:39.638293 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2b_bn_s I0830 19:10:39.638467 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2b_bn_s I0830 19:10:39.638636 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2b_bn_s I0830 19:10:39.638812 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2b_bn_s I0830 19:10:39.639035 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2b_bn_b to I0830 19:10:39.639137 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2b_bn_b I0830 19:10:39.639328 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2b_bn_b I0830 19:10:39.639518 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2b_bn_b I0830 19:10:39.639714 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2b_bn_b I0830 19:10:39.639902 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2b_bn_b I0830 19:10:39.640089 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2b_bn_b I0830 19:10:39.640259 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2b_bn_b I0830 19:10:39.640489 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2c_w to I0830 19:10:39.640589 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2c_w I0830 19:10:39.640784 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2c_w I0830 19:10:39.640984 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2c_w I0830 19:10:39.641201 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2c_w I0830 19:10:39.641402 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2c_w I0830 19:10:39.641602 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2c_w I0830 19:10:39.641813 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2c_w I0830 19:10:39.642045 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2c_bn_s to I0830 19:10:39.642148 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2c_bn_s I0830 19:10:39.642318 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2c_bn_s I0830 19:10:39.642493 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2c_bn_s I0830 19:10:39.642666 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2c_bn_s I0830 19:10:39.642837 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2c_bn_s I0830 19:10:39.643004 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2c_bn_s I0830 19:10:39.643173 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2c_bn_s I0830 19:10:39.643389 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2c_bn_b to I0830 19:10:39.643505 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2c_bn_b I0830 19:10:39.643675 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2c_bn_b I0830 19:10:39.643872 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2c_bn_b I0830 19:10:39.644089 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2c_bn_b I0830 19:10:39.644280 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2c_bn_b I0830 19:10:39.644462 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2c_bn_b I0830 19:10:39.644632 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2c_bn_b I0830 19:10:39.644903 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2a_w to I0830 19:10:39.645009 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2a_w I0830 19:10:39.645302 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2a_w I0830 19:10:39.645572 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2a_w I0830 19:10:39.645878 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2a_w I0830 19:10:39.646156 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2a_w I0830 19:10:39.646476 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2a_w I0830 19:10:39.646806 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2a_w I0830 19:10:39.647125 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2a_bn_s to I0830 19:10:39.647231 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2a_bn_s I0830 19:10:39.647413 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2a_bn_s I0830 19:10:39.647633 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2a_bn_s I0830 19:10:39.647857 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2a_bn_s I0830 19:10:39.648066 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2a_bn_s I0830 19:10:39.648262 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2a_bn_s I0830 19:10:39.648443 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2a_bn_s I0830 19:10:39.648647 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2a_bn_b to I0830 19:10:39.648746 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2a_bn_b I0830 19:10:39.648909 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2a_bn_b I0830 19:10:39.649102 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2a_bn_b I0830 19:10:39.649276 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2a_bn_b I0830 19:10:39.649468 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2a_bn_b I0830 19:10:39.649642 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2a_bn_b I0830 19:10:39.649827 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2a_bn_b I0830 19:10:39.650147 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2b_w to I0830 19:10:39.650262 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2b_w I0830 19:10:39.650608 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2b_w I0830 19:10:39.650917 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2b_w I0830 19:10:39.651206 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2b_w I0830 19:10:39.651525 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2b_w I0830 19:10:39.651834 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2b_w I0830 19:10:39.652136 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2b_w I0830 19:10:39.652467 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2b_bn_s to I0830 19:10:39.652573 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2b_bn_s I0830 19:10:39.652745 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2b_bn_s I0830 19:10:39.652920 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2b_bn_s I0830 19:10:39.653096 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2b_bn_s I0830 19:10:39.653311 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2b_bn_s I0830 19:10:39.653486 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2b_bn_s I0830 19:10:39.653657 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2b_bn_s I0830 19:10:39.653866 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2b_bn_b to I0830 19:10:39.653980 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2b_bn_b I0830 19:10:39.654160 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2b_bn_b I0830 19:10:39.654362 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2b_bn_b I0830 19:10:39.654581 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2b_bn_b I0830 19:10:39.654771 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2b_bn_b I0830 19:10:39.654967 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2b_bn_b I0830 19:10:39.655138 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2b_bn_b I0830 19:10:39.655395 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2c_w to I0830 19:10:39.655514 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2c_w I0830 19:10:39.655771 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2c_w I0830 19:10:39.656054 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2c_w I0830 19:10:39.656317 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2c_w I0830 19:10:39.656573 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2c_w I0830 19:10:39.656841 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2c_w I0830 19:10:39.657149 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2c_w I0830 19:10:39.657480 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2c_bn_s to I0830 19:10:39.657588 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2c_bn_s I0830 19:10:39.657763 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2c_bn_s I0830 19:10:39.657977 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2c_bn_s I0830 19:10:39.658202 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2c_bn_s I0830 19:10:39.658428 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2c_bn_s I0830 19:10:39.658624 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2c_bn_s I0830 19:10:39.658798 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2c_bn_s I0830 19:10:39.659008 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2c_bn_b to I0830 19:10:39.659109 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2c_bn_b I0830 19:10:39.659270 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2c_bn_b I0830 19:10:39.659467 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2c_bn_b I0830 19:10:39.659656 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2c_bn_b I0830 19:10:39.659827 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2c_bn_b I0830 19:10:39.659996 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2c_bn_b I0830 19:10:39.660170 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2c_bn_b I0830 19:10:39.660509 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch1_w to I0830 19:10:39.660633 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch1_w I0830 19:10:39.660942 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch1_w I0830 19:10:39.661214 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch1_w I0830 19:10:39.661497 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch1_w I0830 19:10:39.661787 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch1_w I0830 19:10:39.662074 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch1_w I0830 19:10:39.662352 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch1_w I0830 19:10:39.662672 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch1_bn_s to I0830 19:10:39.662779 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch1_bn_s I0830 19:10:39.662947 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch1_bn_s I0830 19:10:39.663124 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch1_bn_s I0830 19:10:39.663305 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch1_bn_s I0830 19:10:39.663489 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch1_bn_s I0830 19:10:39.663667 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch1_bn_s I0830 19:10:39.663840 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch1_bn_s I0830 19:10:39.664047 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch1_bn_b to I0830 19:10:39.664148 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch1_bn_b I0830 19:10:39.664323 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch1_bn_b I0830 19:10:39.664511 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch1_bn_b I0830 19:10:39.664727 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch1_bn_b I0830 19:10:39.664937 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch1_bn_b I0830 19:10:39.665107 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch1_bn_b I0830 19:10:39.665277 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch1_bn_b I0830 19:10:39.665546 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2a_w to I0830 19:10:39.665648 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2a_w I0830 19:10:39.665893 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2a_w I0830 19:10:39.666138 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2a_w I0830 19:10:39.666408 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2a_w I0830 19:10:39.666656 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2a_w I0830 19:10:39.666937 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2a_w I0830 19:10:39.667231 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2a_w I0830 19:10:39.667573 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2a_bn_s to I0830 19:10:39.667680 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2a_bn_s I0830 19:10:39.667853 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2a_bn_s I0830 19:10:39.668067 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2a_bn_s I0830 19:10:39.668251 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2a_bn_s I0830 19:10:39.668452 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2a_bn_s I0830 19:10:39.668684 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2a_bn_s I0830 19:10:39.668872 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2a_bn_s I0830 19:10:39.669080 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2a_bn_b to I0830 19:10:39.669188 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2a_bn_b I0830 19:10:39.669358 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2a_bn_b I0830 19:10:39.669532 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2a_bn_b I0830 19:10:39.669703 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2a_bn_b I0830 19:10:39.669879 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2a_bn_b I0830 19:10:39.670062 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2a_bn_b I0830 19:10:39.670254 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2a_bn_b I0830 19:10:39.670609 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2b_w to I0830 19:10:39.670742 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2b_w I0830 19:10:39.671066 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2b_w I0830 19:10:39.671359 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2b_w I0830 19:10:39.671659 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2b_w I0830 19:10:39.671947 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2b_w I0830 19:10:39.672244 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2b_w I0830 19:10:39.672564 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2b_w I0830 19:10:39.672892 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2b_bn_s to I0830 19:10:39.672998 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2b_bn_s I0830 19:10:39.673167 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2b_bn_s I0830 19:10:39.673347 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2b_bn_s I0830 19:10:39.673521 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2b_bn_s I0830 19:10:39.673691 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2b_bn_s I0830 19:10:39.673861 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2b_bn_s I0830 19:10:39.674031 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2b_bn_s I0830 19:10:39.674236 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2b_bn_b to I0830 19:10:39.674355 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2b_bn_b I0830 19:10:39.674530 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2b_bn_b I0830 19:10:39.674721 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2b_bn_b I0830 19:10:39.674946 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2b_bn_b I0830 19:10:39.675148 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2b_bn_b I0830 19:10:39.675327 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2b_bn_b I0830 19:10:39.675503 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2b_bn_b I0830 19:10:39.675770 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2c_w to I0830 19:10:39.675872 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2c_w I0830 19:10:39.676117 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2c_w I0830 19:10:39.676359 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2c_w I0830 19:10:39.676616 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2c_w I0830 19:10:39.676850 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2c_w I0830 19:10:39.677115 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2c_w I0830 19:10:39.677390 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2c_w I0830 19:10:39.677722 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2c_bn_s to I0830 19:10:39.677876 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2c_bn_s I0830 19:10:39.678056 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2c_bn_s I0830 19:10:39.678271 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2c_bn_s I0830 19:10:39.678479 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2c_bn_s I0830 19:10:39.678667 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2c_bn_s I0830 19:10:39.678858 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2c_bn_s I0830 19:10:39.679083 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2c_bn_s I0830 19:10:39.679313 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2c_bn_b to I0830 19:10:39.679418 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2c_bn_b I0830 19:10:39.679598 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2c_bn_b I0830 19:10:39.679788 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2c_bn_b I0830 19:10:39.679960 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2c_bn_b I0830 19:10:39.680133 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2c_bn_b I0830 19:10:39.680320 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2c_bn_b I0830 19:10:39.680517 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2c_bn_b I0830 19:10:39.680850 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_theta_w to I0830 19:10:39.680953 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_theta_w I0830 19:10:39.681269 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_theta_w I0830 19:10:39.681585 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_theta_w I0830 19:10:39.681865 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_theta_w I0830 19:10:39.682162 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_theta_w I0830 19:10:39.682441 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_theta_w I0830 19:10:39.682737 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_theta_w I0830 19:10:39.683060 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_theta_b to I0830 19:10:39.683166 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_theta_b I0830 19:10:39.683341 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_theta_b I0830 19:10:39.683518 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_theta_b I0830 19:10:39.683691 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_theta_b I0830 19:10:39.683862 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_theta_b I0830 19:10:39.684032 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_theta_b I0830 19:10:39.684202 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_theta_b I0830 19:10:39.684534 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_phi_w to I0830 19:10:39.684654 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_phi_w I0830 19:10:39.684982 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_phi_w I0830 19:10:39.685297 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_phi_w I0830 19:10:39.685620 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_phi_w I0830 19:10:39.685918 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_phi_w I0830 19:10:39.686202 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_phi_w I0830 19:10:39.686501 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_phi_w I0830 19:10:39.686835 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_phi_b to I0830 19:10:39.686942 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_phi_b I0830 19:10:39.687125 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_phi_b I0830 19:10:39.687333 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_phi_b I0830 19:10:39.687550 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_phi_b I0830 19:10:39.687750 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_phi_b I0830 19:10:39.687992 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_phi_b I0830 19:10:39.688208 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_phi_b I0830 19:10:39.688575 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_g_w to I0830 19:10:39.688724 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_g_w I0830 19:10:39.689038 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_g_w I0830 19:10:39.689341 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_g_w I0830 19:10:39.689670 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_g_w I0830 19:10:39.689979 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_g_w I0830 19:10:39.690267 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_g_w I0830 19:10:39.690550 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_g_w I0830 19:10:39.690864 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_g_b to I0830 19:10:39.690979 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_g_b I0830 19:10:39.691164 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_g_b I0830 19:10:39.691369 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_g_b I0830 19:10:39.691550 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_g_b I0830 19:10:39.691741 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_g_b I0830 19:10:39.691930 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_g_b I0830 19:10:39.692126 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_g_b I0830 19:10:39.692473 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_out_w to I0830 19:10:39.692595 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_out_w I0830 19:10:39.692908 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_out_w I0830 19:10:39.693217 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_out_w I0830 19:10:39.693523 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_out_w I0830 19:10:39.693859 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_out_w I0830 19:10:39.694191 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_out_w I0830 19:10:39.694494 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_out_w I0830 19:10:39.694865 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_out_b to I0830 19:10:39.695012 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_out_b I0830 19:10:39.695226 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_out_b I0830 19:10:39.695492 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_out_b I0830 19:10:39.695811 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_out_b I0830 19:10:39.696031 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_out_b I0830 19:10:39.696243 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_out_b I0830 19:10:39.696456 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_out_b I0830 19:10:39.696669 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_bn_s to I0830 19:10:39.696793 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_bn_s I0830 19:10:39.696962 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_bn_s I0830 19:10:39.697135 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_bn_s I0830 19:10:39.697316 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_bn_s I0830 19:10:39.697493 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_bn_s I0830 19:10:39.697689 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_bn_s I0830 19:10:39.697900 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_bn_s I0830 19:10:39.698153 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_bn_b to I0830 19:10:39.698297 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_bn_b I0830 19:10:39.698535 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_bn_b I0830 19:10:39.698731 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_bn_b I0830 19:10:39.698920 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_bn_b I0830 19:10:39.699140 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_bn_b I0830 19:10:39.699343 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_bn_b I0830 19:10:39.699533 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_bn_b I0830 19:10:39.699908 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2a_w to I0830 19:10:39.700050 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2a_w I0830 19:10:39.700460 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2a_w I0830 19:10:39.700804 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2a_w I0830 19:10:39.701128 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2a_w I0830 19:10:39.701463 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2a_w I0830 19:10:39.701819 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2a_w I0830 19:10:39.702184 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2a_w I0830 19:10:39.702558 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2a_bn_s to I0830 19:10:39.702682 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2a_bn_s I0830 19:10:39.702858 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2a_bn_s I0830 19:10:39.703051 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2a_bn_s I0830 19:10:39.703226 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2a_bn_s I0830 19:10:39.703404 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2a_bn_s I0830 19:10:39.703577 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2a_bn_s I0830 19:10:39.703747 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2a_bn_s I0830 19:10:39.703950 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2a_bn_b to I0830 19:10:39.704070 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2a_bn_b I0830 19:10:39.704236 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2a_bn_b I0830 19:10:39.704416 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2a_bn_b I0830 19:10:39.704586 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2a_bn_b I0830 19:10:39.704755 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2a_bn_b I0830 19:10:39.704924 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2a_bn_b I0830 19:10:39.705094 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2a_bn_b I0830 19:10:39.705421 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2b_w to I0830 19:10:39.705524 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2b_w I0830 19:10:39.705853 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2b_w I0830 19:10:39.706171 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2b_w I0830 19:10:39.706518 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2b_w I0830 19:10:39.706826 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2b_w I0830 19:10:39.707133 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2b_w I0830 19:10:39.707449 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2b_w I0830 19:10:39.707787 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2b_bn_s to I0830 19:10:39.707932 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2b_bn_s I0830 19:10:39.708134 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2b_bn_s I0830 19:10:39.708334 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2b_bn_s I0830 19:10:39.708606 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2b_bn_s I0830 19:10:39.708827 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2b_bn_s I0830 19:10:39.709001 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2b_bn_s I0830 19:10:39.709178 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2b_bn_s I0830 19:10:39.709405 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2b_bn_b to I0830 19:10:39.709522 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2b_bn_b I0830 19:10:39.709707 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2b_bn_b I0830 19:10:39.709928 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2b_bn_b I0830 19:10:39.710104 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2b_bn_b I0830 19:10:39.710299 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2b_bn_b I0830 19:10:39.710472 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2b_bn_b I0830 19:10:39.710644 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2b_bn_b I0830 19:10:39.710947 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2c_w to I0830 19:10:39.711069 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2c_w I0830 19:10:39.711322 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2c_w I0830 19:10:39.711550 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2c_w I0830 19:10:39.711777 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2c_w I0830 19:10:39.712025 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2c_w I0830 19:10:39.712265 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2c_w I0830 19:10:39.712503 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2c_w I0830 19:10:39.712765 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2c_bn_s to I0830 19:10:39.712868 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2c_bn_s I0830 19:10:39.713037 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2c_bn_s I0830 19:10:39.713217 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2c_bn_s I0830 19:10:39.713413 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2c_bn_s I0830 19:10:39.713586 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2c_bn_s I0830 19:10:39.713756 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2c_bn_s I0830 19:10:39.713924 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2c_bn_s I0830 19:10:39.714129 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2c_bn_b to I0830 19:10:39.714241 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2c_bn_b I0830 19:10:39.714431 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2c_bn_b I0830 19:10:39.714626 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2c_bn_b I0830 19:10:39.714797 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2c_bn_b I0830 19:10:39.714968 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2c_bn_b I0830 19:10:39.715137 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2c_bn_b I0830 19:10:39.715315 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2c_bn_b I0830 19:10:39.715578 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2a_w to I0830 19:10:39.715679 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2a_w I0830 19:10:39.715924 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2a_w I0830 19:10:39.716187 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2a_w I0830 19:10:39.716449 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2a_w I0830 19:10:39.716692 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2a_w I0830 19:10:39.716955 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2a_w I0830 19:10:39.717184 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2a_w I0830 19:10:39.717457 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2a_bn_s to I0830 19:10:39.717577 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2a_bn_s I0830 19:10:39.717747 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2a_bn_s I0830 19:10:39.717921 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2a_bn_s I0830 19:10:39.718113 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2a_bn_s I0830 19:10:39.718314 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2a_bn_s I0830 19:10:39.718523 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2a_bn_s I0830 19:10:39.718780 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2a_bn_s I0830 19:10:39.719017 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2a_bn_b to I0830 19:10:39.719168 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2a_bn_b I0830 19:10:39.719367 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2a_bn_b I0830 19:10:39.719588 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2a_bn_b I0830 19:10:39.719789 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2a_bn_b I0830 19:10:39.719978 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2a_bn_b I0830 19:10:39.720199 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2a_bn_b I0830 19:10:39.720407 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2a_bn_b I0830 19:10:39.720736 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2b_w to I0830 19:10:39.720894 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2b_w I0830 19:10:39.721221 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2b_w I0830 19:10:39.721520 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2b_w I0830 19:10:39.721834 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2b_w I0830 19:10:39.722162 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2b_w I0830 19:10:39.722506 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2b_w I0830 19:10:39.722839 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2b_w I0830 19:10:39.723192 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2b_bn_s to I0830 19:10:39.723309 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2b_bn_s I0830 19:10:39.723483 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2b_bn_s I0830 19:10:39.723658 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2b_bn_s I0830 19:10:39.723836 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2b_bn_s I0830 19:10:39.724006 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2b_bn_s I0830 19:10:39.724176 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2b_bn_s I0830 19:10:39.724359 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2b_bn_s I0830 19:10:39.724581 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2b_bn_b to I0830 19:10:39.724708 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2b_bn_b I0830 19:10:39.724878 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2b_bn_b I0830 19:10:39.725050 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2b_bn_b I0830 19:10:39.725234 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2b_bn_b I0830 19:10:39.725416 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2b_bn_b I0830 19:10:39.725586 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2b_bn_b I0830 19:10:39.725765 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2b_bn_b I0830 19:10:39.726043 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2c_w to I0830 19:10:39.726146 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2c_w I0830 19:10:39.726406 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2c_w I0830 19:10:39.726654 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2c_w I0830 19:10:39.726900 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2c_w I0830 19:10:39.727183 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2c_w I0830 19:10:39.727420 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2c_w I0830 19:10:39.727648 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2c_w I0830 19:10:39.727912 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2c_bn_s to I0830 19:10:39.728029 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2c_bn_s I0830 19:10:39.728209 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2c_bn_s I0830 19:10:39.728408 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2c_bn_s I0830 19:10:39.728605 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2c_bn_s I0830 19:10:39.728810 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2c_bn_s I0830 19:10:39.729024 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2c_bn_s I0830 19:10:39.729234 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2c_bn_s I0830 19:10:39.729490 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2c_bn_b to I0830 19:10:39.729630 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2c_bn_b I0830 19:10:39.729810 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2c_bn_b I0830 19:10:39.730045 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2c_bn_b I0830 19:10:39.730263 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2c_bn_b I0830 19:10:39.730532 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2c_bn_b I0830 19:10:39.730719 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2c_bn_b I0830 19:10:39.730890 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2c_bn_b I0830 19:10:39.731232 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_theta_w to I0830 19:10:39.731383 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_theta_w I0830 19:10:39.731719 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_theta_w I0830 19:10:39.732017 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_theta_w I0830 19:10:39.732300 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_theta_w I0830 19:10:39.732588 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_theta_w I0830 19:10:39.732862 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_theta_w I0830 19:10:39.733143 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_theta_w I0830 19:10:39.733522 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_theta_b to I0830 19:10:39.733669 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_theta_b I0830 19:10:39.733874 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_theta_b I0830 19:10:39.734110 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_theta_b I0830 19:10:39.734332 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_theta_b I0830 19:10:39.734554 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_theta_b I0830 19:10:39.734751 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_theta_b I0830 19:10:39.734950 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_theta_b I0830 19:10:39.735300 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_phi_w to I0830 19:10:39.735426 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_phi_w I0830 19:10:39.735754 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_phi_w I0830 19:10:39.736036 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_phi_w I0830 19:10:39.736316 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_phi_w I0830 19:10:39.736592 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_phi_w I0830 19:10:39.736865 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_phi_w I0830 19:10:39.737143 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_phi_w I0830 19:10:39.737500 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_phi_b to I0830 19:10:39.737618 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_phi_b I0830 19:10:39.737802 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_phi_b I0830 19:10:39.737993 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_phi_b I0830 19:10:39.738181 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_phi_b I0830 19:10:39.738375 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_phi_b I0830 19:10:39.738559 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_phi_b I0830 19:10:39.738748 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_phi_b I0830 19:10:39.739063 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_g_w to I0830 19:10:39.739166 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_g_w I0830 19:10:39.739483 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_g_w I0830 19:10:39.739763 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_g_w I0830 19:10:39.740067 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_g_w I0830 19:10:39.740427 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_g_w I0830 19:10:39.740722 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_g_w I0830 19:10:39.741051 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_g_w I0830 19:10:39.741435 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_g_b to I0830 19:10:39.741621 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_g_b I0830 19:10:39.741815 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_g_b I0830 19:10:39.741990 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_g_b I0830 19:10:39.742165 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_g_b I0830 19:10:39.742366 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_g_b I0830 19:10:39.742553 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_g_b I0830 19:10:39.742753 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_g_b I0830 19:10:39.743096 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_out_w to I0830 19:10:39.743217 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_out_w I0830 19:10:39.743535 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_out_w I0830 19:10:39.743818 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_out_w I0830 19:10:39.744106 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_out_w I0830 19:10:39.744385 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_out_w I0830 19:10:39.744688 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_out_w I0830 19:10:39.744992 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_out_w I0830 19:10:39.745342 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_out_b to I0830 19:10:39.745462 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_out_b I0830 19:10:39.745646 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_out_b I0830 19:10:39.745841 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_out_b I0830 19:10:39.746016 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_out_b I0830 19:10:39.746186 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_out_b I0830 19:10:39.746365 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_out_b I0830 19:10:39.746537 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_out_b I0830 19:10:39.746762 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_bn_s to I0830 19:10:39.746864 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_bn_s I0830 19:10:39.747030 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_bn_s I0830 19:10:39.747201 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_bn_s I0830 19:10:39.747385 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_bn_s I0830 19:10:39.747582 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_bn_s I0830 19:10:39.747786 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_bn_s I0830 19:10:39.747980 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_bn_s I0830 19:10:39.748203 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_bn_b to I0830 19:10:39.748312 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_bn_b I0830 19:10:39.748481 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_bn_b I0830 19:10:39.748665 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_bn_b I0830 19:10:39.748853 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_bn_b I0830 19:10:39.749059 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_bn_b I0830 19:10:39.749252 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_bn_b I0830 19:10:39.749435 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_bn_b I0830 19:10:39.749893 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2a_w to I0830 19:10:39.750014 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2a_w I0830 19:10:39.750519 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2a_w I0830 19:10:39.750958 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2a_w I0830 19:10:39.751471 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2a_w I0830 19:10:39.751996 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2a_w I0830 19:10:39.752492 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2a_w I0830 19:10:39.752986 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2a_w I0830 19:10:39.753489 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2a_bn_s to I0830 19:10:39.753628 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2a_bn_s I0830 19:10:39.753809 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2a_bn_s I0830 19:10:39.753988 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2a_bn_s I0830 19:10:39.754167 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2a_bn_s I0830 19:10:39.754364 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2a_bn_s I0830 19:10:39.754554 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2a_bn_s I0830 19:10:39.754729 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2a_bn_s I0830 19:10:39.754954 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2a_bn_b to I0830 19:10:39.755057 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2a_bn_b I0830 19:10:39.755228 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2a_bn_b I0830 19:10:39.755434 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2a_bn_b I0830 19:10:39.755611 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2a_bn_b I0830 19:10:39.755805 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2a_bn_b I0830 19:10:39.755975 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2a_bn_b I0830 19:10:39.756145 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2a_bn_b I0830 19:10:39.756661 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2b_w to I0830 19:10:39.756787 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2b_w I0830 19:10:39.757422 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2b_w I0830 19:10:39.757925 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2b_w I0830 19:10:39.758440 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2b_w I0830 19:10:39.758986 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2b_w I0830 19:10:39.759500 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2b_w I0830 19:10:39.760037 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2b_w I0830 19:10:39.760602 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2b_bn_s to I0830 19:10:39.760731 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2b_bn_s I0830 19:10:39.760919 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2b_bn_s I0830 19:10:39.761111 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2b_bn_s I0830 19:10:39.761353 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2b_bn_s I0830 19:10:39.761568 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2b_bn_s I0830 19:10:39.761780 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2b_bn_s I0830 19:10:39.761971 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2b_bn_s I0830 19:10:39.762182 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2b_bn_b to I0830 19:10:39.762337 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2b_bn_b I0830 19:10:39.762525 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2b_bn_b I0830 19:10:39.762742 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2b_bn_b I0830 19:10:39.762952 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2b_bn_b I0830 19:10:39.763170 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2b_bn_b I0830 19:10:39.763362 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2b_bn_b I0830 19:10:39.763596 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2b_bn_b I0830 19:10:39.764015 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2c_w to I0830 19:10:39.764121 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2c_w I0830 19:10:39.764579 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2c_w I0830 19:10:39.764986 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2c_w I0830 19:10:39.765377 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2c_w I0830 19:10:39.765758 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2c_w I0830 19:10:39.766160 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2c_w I0830 19:10:39.766570 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2c_w I0830 19:10:39.766983 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2c_bn_s to I0830 19:10:39.767122 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2c_bn_s I0830 19:10:39.767307 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2c_bn_s I0830 19:10:39.767487 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2c_bn_s I0830 19:10:39.767661 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2c_bn_s I0830 19:10:39.767833 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2c_bn_s I0830 19:10:39.768007 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2c_bn_s I0830 19:10:39.768178 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2c_bn_s I0830 19:10:39.768395 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2c_bn_b to I0830 19:10:39.768498 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2c_bn_b I0830 19:10:39.768698 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2c_bn_b I0830 19:10:39.768872 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2c_bn_b I0830 19:10:39.769043 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2c_bn_b I0830 19:10:39.769233 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2c_bn_b I0830 19:10:39.769421 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2c_bn_b I0830 19:10:39.769593 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2c_bn_b I0830 19:10:39.770143 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch1_w to I0830 19:10:39.770265 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch1_w I0830 19:10:39.770884 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch1_w I0830 19:10:39.771380 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch1_w I0830 19:10:39.771872 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch1_w I0830 19:10:39.772454 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch1_w I0830 19:10:39.772948 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch1_w I0830 19:10:39.773468 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch1_w I0830 19:10:39.774033 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch1_bn_s to I0830 19:10:39.774197 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch1_bn_s I0830 19:10:39.774401 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch1_bn_s I0830 19:10:39.774600 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch1_bn_s I0830 19:10:39.774840 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch1_bn_s I0830 19:10:39.775074 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch1_bn_s I0830 19:10:39.775273 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch1_bn_s I0830 19:10:39.775473 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch1_bn_s I0830 19:10:39.775707 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch1_bn_b to I0830 19:10:39.775821 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch1_bn_b I0830 19:10:39.775995 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch1_bn_b I0830 19:10:39.776206 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch1_bn_b I0830 19:10:39.776401 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch1_bn_b I0830 19:10:39.776576 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch1_bn_b I0830 19:10:39.776746 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch1_bn_b I0830 19:10:39.776917 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch1_bn_b I0830 19:10:39.777348 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2a_w to I0830 19:10:39.777453 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2a_w I0830 19:10:39.777888 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2a_w I0830 19:10:39.778297 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2a_w I0830 19:10:39.778709 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2a_w I0830 19:10:39.779097 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2a_w I0830 19:10:39.779514 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2a_w I0830 19:10:39.779884 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2a_w I0830 19:10:39.780297 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2a_bn_s to I0830 19:10:39.780426 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2a_bn_s I0830 19:10:39.780601 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2a_bn_s I0830 19:10:39.780776 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2a_bn_s I0830 19:10:39.780951 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2a_bn_s I0830 19:10:39.781121 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2a_bn_s I0830 19:10:39.781299 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2a_bn_s I0830 19:10:39.781471 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2a_bn_s

[Scofieldtangsiwei]

I0830 19:10:39.781678 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2a_bn_b to I0830 19:10:39.781780 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2a_bn_b I0830 19:10:39.781944 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2a_bn_b I0830 19:10:39.782123 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2a_bn_b I0830 19:10:39.782331 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2a_bn_b I0830 19:10:39.782510 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2a_bn_b I0830 19:10:39.782716 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2a_bn_b I0830 19:10:39.782888 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2a_bn_b I0830 19:10:39.783488 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2b_w to I0830 19:10:39.783631 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2b_w I0830 19:10:39.784276 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2b_w I0830 19:10:39.784801 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2b_w I0830 19:10:39.785382 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2b_w I0830 19:10:39.785964 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2b_w I0830 19:10:39.786521 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2b_w I0830 19:10:39.787084 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2b_w I0830 19:10:39.787667 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2b_bn_s to I0830 19:10:39.787780 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2b_bn_s I0830 19:10:39.787956 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2b_bn_s I0830 19:10:39.788140 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2b_bn_s I0830 19:10:39.788333 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2b_bn_s I0830 19:10:39.788532 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2b_bn_s I0830 19:10:39.788710 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2b_bn_s I0830 19:10:39.788897 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2b_bn_s I0830 19:10:39.789107 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2b_bn_b to I0830 19:10:39.789210 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2b_bn_b I0830 19:10:39.789410 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2b_bn_b I0830 19:10:39.789624 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2b_bn_b I0830 19:10:39.789797 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2b_bn_b I0830 19:10:39.789967 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2b_bn_b I0830 19:10:39.790136 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2b_bn_b I0830 19:10:39.790313 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2b_bn_b I0830 19:10:39.790736 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2c_w to I0830 19:10:39.790869 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2c_w I0830 19:10:39.791337 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2c_w I0830 19:10:39.791712 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2c_w I0830 19:10:39.792083 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2c_w I0830 19:10:39.792464 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2c_w I0830 19:10:39.792854 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2c_w I0830 19:10:39.793250 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2c_w I0830 19:10:39.793687 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2c_bn_s to I0830 19:10:39.793824 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2c_bn_s I0830 19:10:39.794017 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2c_bn_s I0830 19:10:39.794229 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2c_bn_s I0830 19:10:39.794416 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2c_bn_s I0830 19:10:39.794591 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2c_bn_s I0830 19:10:39.794763 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2c_bn_s I0830 19:10:39.794939 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2c_bn_s I0830 19:10:39.795201 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2c_bn_b to I0830 19:10:39.795325 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2c_bn_b I0830 19:10:39.795521 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2c_bn_b I0830 19:10:39.795794 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2c_bn_b I0830 19:10:39.796025 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2c_bn_b I0830 19:10:39.796262 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2c_bn_b I0830 19:10:39.796509 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2c_bn_b I0830 19:10:39.796792 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2c_bn_b I0830 19:10:39.797533 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_theta_w to I0830 19:10:39.797715 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_theta_w I0830 19:10:39.798347 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_theta_w I0830 19:10:39.798888 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_theta_w I0830 19:10:39.799411 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_theta_w I0830 19:10:39.799890 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_theta_w I0830 19:10:39.800391 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_theta_w I0830 19:10:39.800919 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_theta_w I0830 19:10:39.801461 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_theta_b to I0830 19:10:39.801574 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_theta_b I0830 19:10:39.801758 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_theta_b I0830 19:10:39.801954 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_theta_b I0830 19:10:39.802134 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_theta_b I0830 19:10:39.802334 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_theta_b I0830 19:10:39.802509 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_theta_b I0830 19:10:39.802681 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_theta_b I0830 19:10:39.803239 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_phi_w to I0830 19:10:39.803371 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_phi_w I0830 19:10:39.803982 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_phi_w I0830 19:10:39.804474 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_phi_w I0830 19:10:39.804992 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_phi_w I0830 19:10:39.805517 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_phi_w I0830 19:10:39.806037 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_phi_w I0830 19:10:39.806607 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_phi_w I0830 19:10:39.807159 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_phi_b to I0830 19:10:39.807333 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_phi_b I0830 19:10:39.807542 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_phi_b I0830 19:10:39.807755 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_phi_b I0830 19:10:39.807976 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_phi_b I0830 19:10:39.808183 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_phi_b I0830 19:10:39.808398 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_phi_b I0830 19:10:39.808603 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_phi_b I0830 19:10:39.809181 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_g_w to I0830 19:10:39.809312 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_g_w I0830 19:10:39.809940 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_g_w I0830 19:10:39.810450 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_g_w I0830 19:10:39.810957 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_g_w I0830 19:10:39.811453 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_g_w I0830 19:10:39.811969 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_g_w I0830 19:10:39.812448 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_g_w I0830 19:10:39.812989 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_g_b to I0830 19:10:39.813118 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_g_b I0830 19:10:39.813306 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_g_b I0830 19:10:39.813503 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_g_b I0830 19:10:39.813683 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_g_b I0830 19:10:39.813858 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_g_b I0830 19:10:39.814029 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_g_b I0830 19:10:39.814203 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_g_b I0830 19:10:39.814768 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_out_w to I0830 19:10:39.814901 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_out_w I0830 19:10:39.815515 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_out_w I0830 19:10:39.816006 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_out_w I0830 19:10:39.816493 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_out_w I0830 19:10:39.817061 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_out_w I0830 19:10:39.817601 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_out_w I0830 19:10:39.818170 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_out_w I0830 19:10:39.818731 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_out_b to I0830 19:10:39.818914 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_out_b I0830 19:10:39.819096 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_out_b I0830 19:10:39.819276 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_out_b I0830 19:10:39.819473 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_out_b I0830 19:10:39.819658 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_out_b I0830 19:10:39.819854 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_out_b I0830 19:10:39.820046 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_out_b I0830 19:10:39.820257 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_bn_s to I0830 19:10:39.820369 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_bn_s I0830 19:10:39.820538 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_bn_s I0830 19:10:39.820713 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_bn_s I0830 19:10:39.820888 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_bn_s I0830 19:10:39.821086 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_bn_s I0830 19:10:39.821259 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_bn_s I0830 19:10:39.821440 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_bn_s I0830 19:10:39.821652 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_bn_b to I0830 19:10:39.821767 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_bn_b I0830 19:10:39.821947 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_bn_b I0830 19:10:39.822151 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_bn_b I0830 19:10:39.822370 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_bn_b I0830 19:10:39.822544 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_bn_b I0830 19:10:39.822714 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_bn_b I0830 19:10:39.822885 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_bn_b I0830 19:10:39.823554 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2a_w to I0830 19:10:39.823672 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2a_w I0830 19:10:39.824449 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2a_w I0830 19:10:39.825076 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2a_w I0830 19:10:39.825711 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2a_w I0830 19:10:39.826323 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2a_w I0830 19:10:39.826941 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2a_w I0830 19:10:39.827560 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2a_w I0830 19:10:39.828210 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2a_bn_s to I0830 19:10:39.828450 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2a_bn_s I0830 19:10:39.828669 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2a_bn_s I0830 19:10:39.828887 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2a_bn_s I0830 19:10:39.829125 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2a_bn_s I0830 19:10:39.829363 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2a_bn_s I0830 19:10:39.829566 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2a_bn_s I0830 19:10:39.829758 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2a_bn_s I0830 19:10:39.829968 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2a_bn_b to I0830 19:10:39.830082 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2a_bn_b I0830 19:10:39.830265 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2a_bn_b I0830 19:10:39.830462 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2a_bn_b I0830 19:10:39.830650 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2a_bn_b I0830 19:10:39.830821 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2a_bn_b I0830 19:10:39.830991 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2a_bn_b I0830 19:10:39.831177 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2a_bn_b I0830 19:10:39.831779 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2b_w to I0830 19:10:39.831895 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2b_w I0830 19:10:39.832541 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2b_w I0830 19:10:39.833092 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2b_w I0830 19:10:39.833614 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2b_w I0830 19:10:39.834160 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2b_w I0830 19:10:39.834671 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2b_w I0830 19:10:39.835215 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2b_w I0830 19:10:39.835800 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2b_bn_s to I0830 19:10:39.835933 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2b_bn_s I0830 19:10:39.836144 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2b_bn_s I0830 19:10:39.836373 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2b_bn_s I0830 19:10:39.836607 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2b_bn_s I0830 19:10:39.836827 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2b_bn_s I0830 19:10:39.837049 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2b_bn_s I0830 19:10:39.837248 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2b_bn_s I0830 19:10:39.837486 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2b_bn_b to I0830 19:10:39.837615 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2b_bn_b I0830 19:10:39.837805 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2b_bn_b I0830 19:10:39.837994 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2b_bn_b I0830 19:10:39.838207 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2b_bn_b I0830 19:10:39.838423 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2b_bn_b I0830 19:10:39.838635 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2b_bn_b I0830 19:10:39.838808 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2b_bn_b I0830 19:10:39.839248 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2c_w to I0830 19:10:39.839456 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2c_w I0830 19:10:39.839946 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2c_w I0830 19:10:39.840373 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2c_w I0830 19:10:39.840794 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2c_w I0830 19:10:39.841182 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2c_w I0830 19:10:39.841581 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2c_w I0830 19:10:39.842000 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2c_w I0830 19:10:39.842427 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2c_bn_s to I0830 19:10:39.842551 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2c_bn_s I0830 19:10:39.842746 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2c_bn_s I0830 19:10:39.842942 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2c_bn_s I0830 19:10:39.843120 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2c_bn_s I0830 19:10:39.843300 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2c_bn_s I0830 19:10:39.843476 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2c_bn_s I0830 19:10:39.843647 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2c_bn_s I0830 19:10:39.843863 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2c_bn_b to I0830 19:10:39.844029 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2c_bn_b I0830 19:10:39.844213 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2c_bn_b I0830 19:10:39.844410 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2c_bn_b I0830 19:10:39.844585 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2c_bn_b I0830 19:10:39.844758 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2c_bn_b I0830 19:10:39.844928 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2c_bn_b I0830 19:10:39.845098 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2c_bn_b I0830 19:10:39.845521 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2a_w to I0830 19:10:39.845638 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2a_w I0830 19:10:39.846115 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2a_w I0830 19:10:39.846514 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2a_w I0830 19:10:39.846906 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2a_w I0830 19:10:39.847307 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2a_w I0830 19:10:39.847683 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2a_w I0830 19:10:39.848062 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2a_w I0830 19:10:39.848524 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2a_bn_s to I0830 19:10:39.848679 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2a_bn_s I0830 19:10:39.848903 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2a_bn_s I0830 19:10:39.849109 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2a_bn_s I0830 19:10:39.849308 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2a_bn_s I0830 19:10:39.849489 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2a_bn_s I0830 19:10:39.849730 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2a_bn_s I0830 19:10:39.849961 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2a_bn_s I0830 19:10:39.850224 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2a_bn_b to I0830 19:10:39.850353 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2a_bn_b I0830 19:10:39.850527 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2a_bn_b I0830 19:10:39.850727 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2a_bn_b I0830 19:10:39.850938 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2a_bn_b I0830 19:10:39.851128 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2a_bn_b I0830 19:10:39.851327 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2a_bn_b I0830 19:10:39.851501 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2a_bn_b I0830 19:10:39.852094 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2b_w to I0830 19:10:39.852221 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2b_w I0830 19:10:39.852878 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2b_w I0830 19:10:39.853403 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2b_w I0830 19:10:39.853921 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2b_w I0830 19:10:39.854460 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2b_w I0830 19:10:39.855024 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2b_w I0830 19:10:39.855540 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2b_w I0830 19:10:39.856091 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2b_bn_s to I0830 19:10:39.856204 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2b_bn_s I0830 19:10:39.856398 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2b_bn_s I0830 19:10:39.856590 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2b_bn_s I0830 19:10:39.856774 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2b_bn_s I0830 19:10:39.856962 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2b_bn_s I0830 19:10:39.857161 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2b_bn_s I0830 19:10:39.857361 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2b_bn_s I0830 19:10:39.857572 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2b_bn_b to I0830 19:10:39.857676 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2b_bn_b I0830 19:10:39.857840 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2b_bn_b I0830 19:10:39.858011 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2b_bn_b I0830 19:10:39.858211 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2b_bn_b I0830 19:10:39.858390 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2b_bn_b I0830 19:10:39.858561 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2b_bn_b I0830 19:10:39.858738 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2b_bn_b I0830 19:10:39.859178 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2c_w to I0830 19:10:39.859302 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2c_w I0830 19:10:39.859760 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2c_w I0830 19:10:39.860241 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2c_w I0830 19:10:39.860677 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2c_w I0830 19:10:39.861129 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2c_w I0830 19:10:39.861562 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2c_w I0830 19:10:39.861974 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2c_w I0830 19:10:39.862427 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2c_bn_s to I0830 19:10:39.862554 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2c_bn_s I0830 19:10:39.862736 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2c_bn_s I0830 19:10:39.862929 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2c_bn_s I0830 19:10:39.863106 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2c_bn_s I0830 19:10:39.863305 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2c_bn_s I0830 19:10:39.863481 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2c_bn_s I0830 19:10:39.863656 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2c_bn_s I0830 19:10:39.863879 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2c_bn_b to I0830 19:10:39.863995 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2c_bn_b I0830 19:10:39.864167 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2c_bn_b I0830 19:10:39.864379 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2c_bn_b I0830 19:10:39.864584 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2c_bn_b I0830 19:10:39.864790 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2c_bn_b I0830 19:10:39.865003 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2c_bn_b I0830 19:10:39.865195 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2c_bn_b I0830 19:10:39.865773 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_theta_w to I0830 19:10:39.865899 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_theta_w I0830 19:10:39.866532 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_theta_w I0830 19:10:39.867021 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_theta_w I0830 19:10:39.867511 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_theta_w I0830 19:10:39.867993 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_theta_w I0830 19:10:39.868494 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_theta_w I0830 19:10:39.869003 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_theta_w I0830 19:10:39.869549 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_theta_b to I0830 19:10:39.869673 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_theta_b I0830 19:10:39.869864 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_theta_b I0830 19:10:39.870124 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_theta_b I0830 19:10:39.870310 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_theta_b I0830 19:10:39.870492 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_theta_b I0830 19:10:39.870692 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_theta_b I0830 19:10:39.870912 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_theta_b I0830 19:10:39.871510 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_phi_w to I0830 19:10:39.871634 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_phi_w I0830 19:10:39.872306 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_phi_w I0830 19:10:39.872855 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_phi_w I0830 19:10:39.873336 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_phi_w I0830 19:10:39.873815 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_phi_w I0830 19:10:39.874339 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_phi_w I0830 19:10:39.874835 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_phi_w I0830 19:10:39.875374 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_phi_b to I0830 19:10:39.875524 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_phi_b I0830 19:10:39.875705 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_phi_b I0830 19:10:39.875884 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_phi_b I0830 19:10:39.876069 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_phi_b I0830 19:10:39.876260 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_phi_b I0830 19:10:39.876473 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_phi_b I0830 19:10:39.876665 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_phi_b I0830 19:10:39.877222 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_g_w to I0830 19:10:39.877335 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_g_w I0830 19:10:39.877933 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_g_w I0830 19:10:39.878427 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_g_w I0830 19:10:39.878922 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_g_w I0830 19:10:39.879411 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_g_w I0830 19:10:39.879920 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_g_w I0830 19:10:39.880415 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_g_w I0830 19:10:39.881015 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_g_b to I0830 19:10:39.881200 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_g_b I0830 19:10:39.881402 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_g_b I0830 19:10:39.881614 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_g_b I0830 19:10:39.881839 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_g_b I0830 19:10:39.882045 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_g_b I0830 19:10:39.882225 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_g_b I0830 19:10:39.882420 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_g_b I0830 19:10:39.883022 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_out_w to I0830 19:10:39.883203 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_out_w I0830 19:10:39.883842 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_out_w I0830 19:10:39.884350 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_out_w I0830 19:10:39.884865 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_out_w I0830 19:10:39.885370 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_out_w I0830 19:10:39.885862 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_out_w I0830 19:10:39.886388 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_out_w I0830 19:10:39.886939 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_out_b to I0830 19:10:39.887079 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_out_b I0830 19:10:39.887259 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_out_b I0830 19:10:39.887449 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_out_b I0830 19:10:39.887633 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_out_b I0830 19:10:39.887814 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_out_b I0830 19:10:39.887989 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_out_b I0830 19:10:39.888164 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_out_b I0830 19:10:39.888381 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_bn_s to I0830 19:10:39.888484 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_bn_s I0830 19:10:39.888681 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_bn_s I0830 19:10:39.888871 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_bn_s I0830 19:10:39.889045 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_bn_s I0830 19:10:39.889218 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_bn_s I0830 19:10:39.889398 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_bn_s I0830 19:10:39.889569 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_bn_s I0830 19:10:39.889779 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_bn_b to I0830 19:10:39.889899 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_bn_b I0830 19:10:39.890077 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_bn_b I0830 19:10:39.890249 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_bn_b I0830 19:10:39.890429 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_bn_b I0830 19:10:39.890616 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_bn_b I0830 19:10:39.890831 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_bn_b I0830 19:10:39.891005 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_bn_b I0830 19:10:39.891707 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2a_w to I0830 19:10:39.891829 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2a_w I0830 19:10:39.892625 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2a_w I0830 19:10:39.893282 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2a_w I0830 19:10:39.893964 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2a_w I0830 19:10:39.894583 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2a_w I0830 19:10:39.895236 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2a_w I0830 19:10:39.895857 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2a_w I0830 19:10:39.896501 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2a_bn_s to I0830 19:10:39.896652 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2a_bn_s I0830 19:10:39.896848 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2a_bn_s I0830 19:10:39.897092 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2a_bn_s I0830 19:10:39.897382 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2a_bn_s I0830 19:10:39.897619 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2a_bn_s I0830 19:10:39.897845 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2a_bn_s I0830 19:10:39.898089 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2a_bn_s I0830 19:10:39.898327 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2a_bn_b to I0830 19:10:39.898468 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2a_bn_b I0830 19:10:39.898688 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2a_bn_b I0830 19:10:39.898905 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2a_bn_b I0830 19:10:39.899079 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2a_bn_b I0830 19:10:39.899291 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2a_bn_b I0830 19:10:39.899524 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2a_bn_b I0830 19:10:39.899732 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2a_bn_b I0830 19:10:39.900362 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2b_w to I0830 19:10:39.900579 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2b_w I0830 19:10:39.901231 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2b_w I0830 19:10:39.901820 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2b_w I0830 19:10:39.902347 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2b_w I0830 19:10:39.902856 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2b_w I0830 19:10:39.903388 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2b_w I0830 19:10:39.903907 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2b_w I0830 19:10:39.904508 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2b_bn_s to I0830 19:10:39.904660 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2b_bn_s I0830 19:10:39.904870 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2b_bn_s I0830 19:10:39.905100 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2b_bn_s I0830 19:10:39.905304 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2b_bn_s I0830 19:10:39.905488 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2b_bn_s I0830 19:10:39.905668 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2b_bn_s I0830 19:10:39.905846 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2b_bn_s I0830 19:10:39.906073 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2b_bn_b to I0830 19:10:39.906215 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2b_bn_b I0830 19:10:39.906400 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2b_bn_b I0830 19:10:39.906574 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2b_bn_b I0830 19:10:39.906747 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2b_bn_b I0830 19:10:39.906918 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2b_bn_b I0830 19:10:39.907134 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2b_bn_b I0830 19:10:39.907371 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2b_bn_b I0830 19:10:39.907847 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2c_w to I0830 19:10:39.907954 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2c_w I0830 19:10:39.908404 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2c_w I0830 19:10:39.908816 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2c_w I0830 19:10:39.909220 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2c_w I0830 19:10:39.909611 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2c_w I0830 19:10:39.910001 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2c_w I0830 19:10:39.910372 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2c_w I0830 19:10:39.910783 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2c_bn_s to I0830 19:10:39.910907 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2c_bn_s I0830 19:10:39.911103 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2c_bn_s I0830 19:10:39.911307 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2c_bn_s I0830 19:10:39.911485 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2c_bn_s I0830 19:10:39.911659 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2c_bn_s I0830 19:10:39.911832 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2c_bn_s I0830 19:10:39.912007 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2c_bn_s I0830 19:10:39.912230 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2c_bn_b to I0830 19:10:39.912373 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2c_bn_b I0830 19:10:39.912547 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2c_bn_b I0830 19:10:39.912741 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2c_bn_b I0830 19:10:39.912915 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2c_bn_b I0830 19:10:39.913099 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2c_bn_b I0830 19:10:39.913270 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2c_bn_b I0830 19:10:39.913465 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2c_bn_b I0830 19:10:39.913891 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2a_w to I0830 19:10:39.914011 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2a_w I0830 19:10:39.914460 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2a_w I0830 19:10:39.914885 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2a_w I0830 19:10:39.915306 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2a_w I0830 19:10:39.915751 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2a_w I0830 19:10:39.916142 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2a_w I0830 19:10:39.916531 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2a_w I0830 19:10:39.917006 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2a_bn_s to I0830 19:10:39.917129 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2a_bn_s I0830 19:10:39.917346 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2a_bn_s I0830 19:10:39.917550 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2a_bn_s I0830 19:10:39.917773 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2a_bn_s I0830 19:10:39.917999 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2a_bn_s I0830 19:10:39.918200 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2a_bn_s I0830 19:10:39.918386 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2a_bn_s I0830 19:10:39.918608 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2a_bn_b to I0830 19:10:39.918725 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2a_bn_b I0830 19:10:39.918910 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2a_bn_b I0830 19:10:39.919117 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2a_bn_b I0830 19:10:39.919297 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2a_bn_b I0830 19:10:39.919472 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2a_bn_b I0830 19:10:39.919651 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2a_bn_b I0830 19:10:39.919821 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2a_bn_b I0830 19:10:39.920424 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2b_w to I0830 19:10:39.920542 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2b_w I0830 19:10:39.921184 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2b_w I0830 19:10:39.921724 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2b_w I0830 19:10:39.922228 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2b_w I0830 19:10:39.922758 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2b_w I0830 19:10:39.923326 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2b_w I0830 19:10:39.923854 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2b_w I0830 19:10:39.924391 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2b_bn_s to I0830 19:10:39.924515 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2b_bn_s I0830 19:10:39.924700 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2b_bn_s I0830 19:10:39.924916 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2b_bn_s I0830 19:10:39.925100 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2b_bn_s I0830 19:10:39.925293 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2b_bn_s I0830 19:10:39.925535 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2b_bn_s I0830 19:10:39.925755 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2b_bn_s I0830 19:10:39.925970 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2b_bn_b to I0830 19:10:39.926085 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2b_bn_b I0830 19:10:39.926253 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2b_bn_b I0830 19:10:39.926435 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2b_bn_b I0830 19:10:39.926625 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2b_bn_b I0830 19:10:39.926852 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2b_bn_b I0830 19:10:39.927023 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2b_bn_b I0830 19:10:39.927198 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2b_bn_b I0830 19:10:39.927691 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2c_w to I0830 19:10:39.927871 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2c_w I0830 19:10:39.928380 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2c_w I0830 19:10:39.928791 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2c_w I0830 19:10:39.929213 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2c_w I0830 19:10:39.929626 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2c_w I0830 19:10:39.930015 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2c_w I0830 19:10:39.930434 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2c_w I0830 19:10:39.930852 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2c_bn_s to I0830 19:10:39.930984 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2c_bn_s I0830 19:10:39.931178 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2c_bn_s I0830 19:10:39.931370 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2c_bn_s I0830 19:10:39.931566 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2c_bn_s I0830 19:10:39.931742 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2c_bn_s I0830 19:10:39.931916 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2c_bn_s I0830 19:10:39.932090 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2c_bn_s I0830 19:10:39.932308 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2c_bn_b to I0830 19:10:39.932411 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2c_bn_b I0830 19:10:39.932585 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2c_bn_b I0830 19:10:39.932774 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2c_bn_b I0830 19:10:39.932971 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2c_bn_b I0830 19:10:39.933163 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2c_bn_b I0830 19:10:39.933394 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2c_bn_b I0830 19:10:39.933569 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2c_bn_b I0830 19:10:39.934130 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_theta_w to I0830 19:10:39.934257 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_theta_w I0830 19:10:39.934876 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_theta_w I0830 19:10:39.935405 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_theta_w I0830 19:10:39.935937 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_theta_w I0830 19:10:39.936470 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_theta_w I0830 19:10:39.936970 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_theta_w I0830 19:10:39.937517 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_theta_w I0830 19:10:39.938096 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_theta_b to I0830 19:10:39.938266 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_theta_b I0830 19:10:39.938585 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_theta_b I0830 19:10:39.938801 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_theta_b I0830 19:10:39.938994 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_theta_b I0830 19:10:39.939199 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_theta_b I0830 19:10:39.939403 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_theta_b I0830 19:10:39.939620 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_theta_b I0830 19:10:39.940193 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_phi_w to I0830 19:10:39.940347 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_phi_w I0830 19:10:39.940967 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_phi_w I0830 19:10:39.941469 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_phi_w I0830 19:10:39.941966 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_phi_w I0830 19:10:39.942473 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_phi_w I0830 19:10:39.942975 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_phi_w I0830 19:10:39.943469 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_phi_w I0830 19:10:39.944032 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_phi_b to I0830 19:10:39.944145 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_phi_b I0830 19:10:39.944346 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_phi_b I0830 19:10:39.944545 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_phi_b I0830 19:10:39.944729 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_phi_b I0830 19:10:39.944915 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_phi_b I0830 19:10:39.945101 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_phi_b I0830 19:10:39.945312 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_phi_b I0830 19:10:39.945924 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_g_w to I0830 19:10:39.946074 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_g_w I0830 19:10:39.946734 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_g_w I0830 19:10:39.947280 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_g_w I0830 19:10:39.947810 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_g_w I0830 19:10:39.948338 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_g_w I0830 19:10:39.948860 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_g_w I0830 19:10:39.949383 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_g_w I0830 19:10:39.949973 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_g_b to I0830 19:10:39.950112 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_g_b I0830 19:10:39.950311 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_g_b I0830 19:10:39.950516 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_g_b I0830 19:10:39.950721 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_g_b I0830 19:10:39.950913 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_g_b I0830 19:10:39.951096 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_g_b I0830 19:10:39.951283 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_g_b I0830 19:10:39.951854 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_out_w to I0830 19:10:39.951973 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_out_w I0830 19:10:39.952608 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_out_w I0830 19:10:39.953099 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_out_w I0830 19:10:39.953619 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_out_w I0830 19:10:39.954137 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_out_w I0830 19:10:39.954642 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_out_w I0830 19:10:39.955120 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_out_w I0830 19:10:39.955650 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_out_b to I0830 19:10:39.955796 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_out_b I0830 19:10:39.955976 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_out_b I0830 19:10:39.956155 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_out_b I0830 19:10:39.956377 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_out_b I0830 19:10:39.956553 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_out_b I0830 19:10:39.956729 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_out_b I0830 19:10:39.956931 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_out_b I0830 19:10:39.957186 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_bn_s to I0830 19:10:39.957330 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_bn_s I0830 19:10:39.957505 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_bn_s I0830 19:10:39.957680 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_bn_s I0830 19:10:39.957859 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_bn_s I0830 19:10:39.958060 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_bn_s I0830 19:10:39.958260 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_bn_s I0830 19:10:39.958484 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_bn_s I0830 19:10:39.958714 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_bn_b to I0830 19:10:39.958817 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_bn_b I0830 19:10:39.958985 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_bn_b I0830 19:10:39.959192 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_bn_b I0830 19:10:39.959378 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_bn_b I0830 19:10:39.959568 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_bn_b I0830 19:10:39.959784 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_bn_b I0830 19:10:39.959990 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_bn_b I0830 19:10:39.960563 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2a_w to I0830 19:10:39.960728 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2a_w I0830 19:10:39.961349 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2a_w I0830 19:10:39.961842 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2a_w I0830 19:10:39.962353 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2a_w I0830 19:10:39.962852 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2a_w I0830 19:10:39.963351 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2a_w I0830 19:10:39.963830 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2a_w I0830 19:10:39.964400 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2a_bn_s to I0830 19:10:39.964529 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2a_bn_s I0830 19:10:39.964713 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2a_bn_s I0830 19:10:39.964910 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2a_bn_s I0830 19:10:39.965094 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2a_bn_s I0830 19:10:39.965281 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2a_bn_s I0830 19:10:39.965466 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2a_bn_s I0830 19:10:39.965637 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2a_bn_s I0830 19:10:39.965846 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2a_bn_b to I0830 19:10:39.965951 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2a_bn_b I0830 19:10:39.966136 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2a_bn_b I0830 19:10:39.966321 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2a_bn_b I0830 19:10:39.966531 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2a_bn_b I0830 19:10:39.966721 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2a_bn_b I0830 19:10:39.966892 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2a_bn_b I0830 19:10:39.967092 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2a_bn_b I0830 19:10:39.968231 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2b_w to I0830 19:10:39.968401 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2b_w I0830 19:10:39.970154 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2b_w I0830 19:10:39.971383 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2b_w I0830 19:10:39.972626 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2b_w I0830 19:10:39.973876 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2b_w I0830 19:10:39.975079 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2b_w I0830 19:10:39.976305 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2b_w I0830 19:10:39.977544 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2b_bn_s to I0830 19:10:39.977744 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2b_bn_s I0830 19:10:39.977930 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2b_bn_s I0830 19:10:39.978116 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2b_bn_s I0830 19:10:39.978318 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2b_bn_s I0830 19:10:39.978502 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2b_bn_s I0830 19:10:39.978693 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2b_bn_s I0830 19:10:39.978908 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2b_bn_s I0830 19:10:39.979134 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2b_bn_b to I0830 19:10:39.979264 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2b_bn_b I0830 19:10:39.979443 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2b_bn_b I0830 19:10:39.979615 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2b_bn_b I0830 19:10:39.979793 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2b_bn_b I0830 19:10:39.979981 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2b_bn_b I0830 19:10:39.980154 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2b_bn_b I0830 19:10:39.980336 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2b_bn_b I0830 19:10:39.981024 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2c_w to I0830 19:10:39.981157 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2c_w I0830 19:10:39.982094 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2c_w I0830 19:10:39.982790 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2c_w I0830 19:10:39.983545 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2c_w I0830 19:10:39.984297 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2c_w I0830 19:10:39.985021 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2c_w I0830 19:10:39.985752 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2c_w I0830 19:10:39.986507 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2c_bn_s to I0830 19:10:39.986637 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2c_bn_s I0830 19:10:39.986835 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2c_bn_s I0830 19:10:39.987054 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2c_bn_s I0830 19:10:39.987257 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2c_bn_s I0830 19:10:39.987468 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2c_bn_s I0830 19:10:39.987646 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2c_bn_s I0830 19:10:39.987828 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2c_bn_s I0830 19:10:39.988084 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2c_bn_b to I0830 19:10:39.988224 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2c_bn_b I0830 19:10:39.988425 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2c_bn_b I0830 19:10:39.988603 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2c_bn_b I0830 19:10:39.988775 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2c_bn_b I0830 19:10:39.988948 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2c_bn_b I0830 19:10:39.989152 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2c_bn_b I0830 19:10:39.989357 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2c_bn_b I0830 19:10:39.990430 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch1_w to I0830 19:10:39.990550 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch1_w I0830 19:10:39.992130 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch1_w I0830 19:10:39.993325 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch1_w I0830 19:10:39.994451 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch1_w I0830 19:10:39.995542 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch1_w I0830 19:10:39.996669 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch1_w I0830 19:10:39.997796 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch1_w I0830 19:10:39.999049 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch1_bn_s to I0830 19:10:39.999279 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch1_bn_s I0830 19:10:39.999531 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch1_bn_s I0830 19:10:39.999746 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch1_bn_s I0830 19:10:40.000010 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch1_bn_s I0830 19:10:40.000247 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch1_bn_s I0830 19:10:40.000505 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch1_bn_s I0830 19:10:40.000720 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch1_bn_s I0830 19:10:40.001003 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch1_bn_b to I0830 19:10:40.001130 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch1_bn_b I0830 19:10:40.001326 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch1_bn_b I0830 19:10:40.001539 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch1_bn_b I0830 19:10:40.001779 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch1_bn_b I0830 19:10:40.002002 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch1_bn_b I0830 19:10:40.002197 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch1_bn_b I0830 19:10:40.002401 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch1_bn_b I0830 19:10:40.003880 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2a_w to I0830 19:10:40.004007 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2a_w I0830 19:10:40.006284 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2a_w I0830 19:10:40.008007 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2a_w I0830 19:10:40.009613 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2a_w I0830 19:10:40.011126 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2a_w I0830 19:10:40.012677 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2a_w I0830 19:10:40.014204 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2a_w I0830 19:10:40.015743 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2a_bn_s to I0830 19:10:40.015918 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2a_bn_s I0830 19:10:40.016103 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2a_bn_s I0830 19:10:40.016284 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2a_bn_s I0830 19:10:40.016469 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2a_bn_s I0830 19:10:40.016663 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2a_bn_s I0830 19:10:40.016881 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2a_bn_s I0830 19:10:40.017085 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2a_bn_s I0830 19:10:40.017328 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2a_bn_b to I0830 19:10:40.017433 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2a_bn_b I0830 19:10:40.017601 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2a_bn_b I0830 19:10:40.017776 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2a_bn_b I0830 19:10:40.017957 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2a_bn_b I0830 19:10:40.018131 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2a_bn_b I0830 19:10:40.018318 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2a_bn_b I0830 19:10:40.018565 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2a_bn_b I0830 19:10:40.019988 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2b_w to I0830 19:10:40.020111 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2b_w I0830 19:10:40.021873 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2b_w I0830 19:10:40.023102 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2b_w I0830 19:10:40.024365 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2b_w I0830 19:10:40.025588 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2b_w I0830 19:10:40.026817 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2b_w I0830 19:10:40.028040 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2b_w I0830 19:10:40.029318 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2b_bn_s to I0830 19:10:40.029495 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2b_bn_s I0830 19:10:40.029711 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2b_bn_s I0830 19:10:40.029938 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2b_bn_s I0830 19:10:40.030138 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2b_bn_s I0830 19:10:40.030359 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2b_bn_s I0830 19:10:40.030580 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2b_bn_s I0830 19:10:40.030771 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2b_bn_s I0830 19:10:40.031024 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2b_bn_b to I0830 19:10:40.031152 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2b_bn_b I0830 19:10:40.031366 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2b_bn_b I0830 19:10:40.031568 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2b_bn_b I0830 19:10:40.031776 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2b_bn_b I0830 19:10:40.031950 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2b_bn_b I0830 19:10:40.032123 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2b_bn_b I0830 19:10:40.032318 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2b_bn_b I0830 19:10:40.033082 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2c_w to I0830 19:10:40.033210 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2c_w I0830 19:10:40.034179 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2c_w I0830 19:10:40.034915 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2c_w I0830 19:10:40.035612 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2c_w I0830 19:10:40.036358 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2c_w I0830 19:10:40.037086 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2c_w I0830 19:10:40.037812 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2c_w I0830 19:10:40.038594 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2c_bn_s to I0830 19:10:40.038743 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2c_bn_s I0830 19:10:40.038942 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2c_bn_s I0830 19:10:40.039195 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2c_bn_s I0830 19:10:40.039417 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2c_bn_s I0830 19:10:40.039647 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2c_bn_s I0830 19:10:40.039908 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2c_bn_s I0830 19:10:40.040117 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2c_bn_s I0830 19:10:40.040364 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2c_bn_b to I0830 19:10:40.040537 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2c_bn_b I0830 19:10:40.040747 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2c_bn_b I0830 19:10:40.040956 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2c_bn_b I0830 19:10:40.041177 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2c_bn_b I0830 19:10:40.041383 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2c_bn_b I0830 19:10:40.041594 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2c_bn_b I0830 19:10:40.041800 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2c_bn_b I0830 19:10:40.042665 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2a_w to I0830 19:10:40.042793 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2a_w I0830 19:10:40.043725 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2a_w I0830 19:10:40.044431 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2a_w I0830 19:10:40.045124 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2a_w I0830 19:10:40.045851 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2a_w I0830 19:10:40.046585 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2a_w I0830 19:10:40.047275 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2a_w I0830 19:10:40.048043 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2a_bn_s to I0830 19:10:40.048212 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2a_bn_s I0830 19:10:40.048403 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2a_bn_s I0830 19:10:40.048599 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2a_bn_s I0830 19:10:40.048789 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2a_bn_s I0830 19:10:40.049016 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2a_bn_s I0830 19:10:40.049200 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2a_bn_s I0830 19:10:40.049397 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2a_bn_s I0830 19:10:40.049624 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2a_bn_b to I0830 19:10:40.049729 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2a_bn_b I0830 19:10:40.049899 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2a_bn_b I0830 19:10:40.050103 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2a_bn_b I0830 19:10:40.050332 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2a_bn_b I0830 19:10:40.050559 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2a_bn_b I0830 19:10:40.050735 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2a_bn_b I0830 19:10:40.050949 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2a_bn_b I0830 19:10:40.052459 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2b_w to I0830 19:10:40.052582 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2b_w I0830 19:10:40.054353 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2b_w I0830 19:10:40.055540 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2b_w I0830 19:10:40.056793 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2b_w I0830 19:10:40.058000 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2b_w I0830 19:10:40.059197 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2b_w I0830 19:10:40.060417 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2b_w I0830 19:10:40.061693 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2b_bn_s to I0830 19:10:40.061856 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2b_bn_s I0830 19:10:40.062058 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2b_bn_s I0830 19:10:40.062283 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2b_bn_s I0830 19:10:40.062527 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2b_bn_s I0830 19:10:40.062757 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2b_bn_s I0830 19:10:40.062973 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2b_bn_s I0830 19:10:40.063149 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2b_bn_s I0830 19:10:40.063385 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2b_bn_b to I0830 19:10:40.063500 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2b_bn_b I0830 19:10:40.063697 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2b_bn_b I0830 19:10:40.063886 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2b_bn_b I0830 19:10:40.064086 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2b_bn_b I0830 19:10:40.064276 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2b_bn_b I0830 19:10:40.064456 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2b_bn_b I0830 19:10:40.064629 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2b_bn_b I0830 19:10:40.065464 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2c_w to I0830 19:10:40.065586 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2c_w I0830 19:10:40.066534 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2c_w I0830 19:10:40.067251 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2c_w I0830 19:10:40.067995 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2c_w I0830 19:10:40.068708 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2c_w I0830 19:10:40.069387 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2c_w I0830 19:10:40.070096 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2c_w I0830 19:10:40.070833 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2c_bn_s to I0830 19:10:40.070966 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2c_bn_s I0830 19:10:40.071151 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2c_bn_s I0830 19:10:40.071340 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2c_bn_s I0830 19:10:40.071537 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2c_bn_s I0830 19:10:40.071732 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2c_bn_s I0830 19:10:40.071928 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2c_bn_s I0830 19:10:40.072151 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2c_bn_s I0830 19:10:40.072422 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2c_bn_b to I0830 19:10:40.072576 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2c_bn_b I0830 19:10:40.072767 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2c_bn_b I0830 19:10:40.072990 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2c_bn_b I0830 19:10:40.073250 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2c_bn_b I0830 19:10:40.073452 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2c_bn_b I0830 19:10:40.073661 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2c_bn_b I0830 19:10:40.073853 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2c_bn_b I0830 19:10:40.074187 52234 checkpoints.py:410] Broadcasting gpu_0/pred_w to I0830 19:10:40.074326 52234 checkpoints.py:415] |-> gpu_1/pred_w I0830 19:10:40.074693 52234 checkpoints.py:415] |-> gpu_2/pred_w I0830 19:10:40.075012 52234 checkpoints.py:415] |-> gpu_3/pred_w I0830 19:10:40.075318 52234 checkpoints.py:415] |-> gpu_4/pred_w I0830 19:10:40.075620 52234 checkpoints.py:415] |-> gpu_5/pred_w I0830 19:10:40.075938 52234 checkpoints.py:415] |-> gpu_6/pred_w I0830 19:10:40.076263 52234 checkpoints.py:415] |-> gpu_7/pred_w I0830 19:10:40.076634 52234 checkpoints.py:410] Broadcasting gpu_0/pred_b to I0830 19:10:40.076761 52234 checkpoints.py:415] |-> gpu_1/pred_b I0830 19:10:40.076942 52234 checkpoints.py:415] |-> gpu_2/pred_b I0830 19:10:40.077135 52234 checkpoints.py:415] |-> gpu_3/pred_b I0830 19:10:40.077315 52234 checkpoints.py:415] |-> gpu_4/pred_b I0830 19:10:40.077488 52234 checkpoints.py:415] |-> gpu_5/pred_b I0830 19:10:40.077661 52234 checkpoints.py:415] |-> gpu_6/pred_b I0830 19:10:40.077832 52234 checkpoints.py:415] |-> gpu_7/pred_b I0830 19:10:40.078075 52234 checkpoints.py:410] Broadcasting gpu_0/conv1_w_momentum to I0830 19:10:40.078237 52234 checkpoints.py:415] |-> gpu_1/conv1_w_momentum I0830 19:10:40.078478 52234 checkpoints.py:415] |-> gpu_2/conv1_w_momentum I0830 19:10:40.078691 52234 checkpoints.py:415] |-> gpu_3/conv1_w_momentum I0830 19:10:40.078907 52234 checkpoints.py:415] |-> gpu_4/conv1_w_momentum I0830 19:10:40.079121 52234 checkpoints.py:415] |-> gpu_5/conv1_w_momentum I0830 19:10:40.079344 52234 checkpoints.py:415] |-> gpu_6/conv1_w_momentum I0830 19:10:40.079590 52234 checkpoints.py:415] |-> gpu_7/conv1_w_momentum I0830 19:10:40.079854 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2a_w_momentum to I0830 19:10:40.079973 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2a_w_momentum I0830 19:10:40.080174 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2a_w_momentum I0830 19:10:40.080399 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2a_w_momentum I0830 19:10:40.080610 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2a_w_momentum I0830 19:10:40.080806 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2a_w_momentum I0830 19:10:40.081012 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2a_w_momentum I0830 19:10:40.081214 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2a_w_momentum I0830 19:10:40.081482 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2b_w_momentum to I0830 19:10:40.081602 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2b_w_momentum I0830 19:10:40.081840 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2b_w_momentum I0830 19:10:40.082071 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2b_w_momentum I0830 19:10:40.082320 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2b_w_momentum I0830 19:10:40.082542 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2b_w_momentum I0830 19:10:40.082798 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2b_w_momentum I0830 19:10:40.083020 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2b_w_momentum I0830 19:10:40.083321 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch2c_w_momentum to I0830 19:10:40.083493 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch2c_w_momentum I0830 19:10:40.083740 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch2c_w_momentum I0830 19:10:40.083980 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch2c_w_momentum I0830 19:10:40.084176 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch2c_w_momentum I0830 19:10:40.084377 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch2c_w_momentum I0830 19:10:40.084578 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch2c_w_momentum I0830 19:10:40.084805 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch2c_w_momentum I0830 19:10:40.085052 52234 checkpoints.py:410] Broadcasting gpu_0/res2_0_branch1_w_momentum to I0830 19:10:40.085172 52234 checkpoints.py:415] |-> gpu_1/res2_0_branch1_w_momentum I0830 19:10:40.085376 52234 checkpoints.py:415] |-> gpu_2/res2_0_branch1_w_momentum I0830 19:10:40.085575 52234 checkpoints.py:415] |-> gpu_3/res2_0_branch1_w_momentum I0830 19:10:40.085776 52234 checkpoints.py:415] |-> gpu_4/res2_0_branch1_w_momentum I0830 19:10:40.085992 52234 checkpoints.py:415] |-> gpu_5/res2_0_branch1_w_momentum I0830 19:10:40.086206 52234 checkpoints.py:415] |-> gpu_6/res2_0_branch1_w_momentum I0830 19:10:40.086409 52234 checkpoints.py:415] |-> gpu_7/res2_0_branch1_w_momentum I0830 19:10:40.086673 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2a_w_momentum to I0830 19:10:40.086786 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2a_w_momentum I0830 19:10:40.087052 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2a_w_momentum I0830 19:10:40.087285 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2a_w_momentum I0830 19:10:40.087527 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2a_w_momentum I0830 19:10:40.087743 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2a_w_momentum I0830 19:10:40.087956 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2a_w_momentum I0830 19:10:40.088169 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2a_w_momentum I0830 19:10:40.088462 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2b_w_momentum to I0830 19:10:40.088630 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2b_w_momentum I0830 19:10:40.088860 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2b_w_momentum I0830 19:10:40.089067 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2b_w_momentum I0830 19:10:40.089273 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2b_w_momentum I0830 19:10:40.089489 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2b_w_momentum I0830 19:10:40.089725 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2b_w_momentum I0830 19:10:40.089976 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2b_w_momentum I0830 19:10:40.090249 52234 checkpoints.py:410] Broadcasting gpu_0/res2_1_branch2c_w_momentum to I0830 19:10:40.090374 52234 checkpoints.py:415] |-> gpu_1/res2_1_branch2c_w_momentum I0830 19:10:40.090575 52234 checkpoints.py:415] |-> gpu_2/res2_1_branch2c_w_momentum I0830 19:10:40.090771 52234 checkpoints.py:415] |-> gpu_3/res2_1_branch2c_w_momentum I0830 19:10:40.090971 52234 checkpoints.py:415] |-> gpu_4/res2_1_branch2c_w_momentum I0830 19:10:40.091190 52234 checkpoints.py:415] |-> gpu_5/res2_1_branch2c_w_momentum I0830 19:10:40.091413 52234 checkpoints.py:415] |-> gpu_6/res2_1_branch2c_w_momentum I0830 19:10:40.091624 52234 checkpoints.py:415] |-> gpu_7/res2_1_branch2c_w_momentum I0830 19:10:40.091896 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2a_w_momentum to I0830 19:10:40.092024 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2a_w_momentum I0830 19:10:40.092258 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2a_w_momentum I0830 19:10:40.092525 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2a_w_momentum I0830 19:10:40.092749 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2a_w_momentum I0830 19:10:40.092988 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2a_w_momentum I0830 19:10:40.093236 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2a_w_momentum I0830 19:10:40.093485 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2a_w_momentum I0830 19:10:40.093823 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2b_w_momentum to I0830 19:10:40.093971 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2b_w_momentum I0830 19:10:40.094250 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2b_w_momentum I0830 19:10:40.094521 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2b_w_momentum I0830 19:10:40.094758 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2b_w_momentum I0830 19:10:40.094965 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2b_w_momentum I0830 19:10:40.095175 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2b_w_momentum I0830 19:10:40.095412 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2b_w_momentum I0830 19:10:40.095695 52234 checkpoints.py:410] Broadcasting gpu_0/res2_2_branch2c_w_momentum to I0830 19:10:40.095813 52234 checkpoints.py:415] |-> gpu_1/res2_2_branch2c_w_momentum I0830 19:10:40.096033 52234 checkpoints.py:415] |-> gpu_2/res2_2_branch2c_w_momentum I0830 19:10:40.096249 52234 checkpoints.py:415] |-> gpu_3/res2_2_branch2c_w_momentum I0830 19:10:40.096462 52234 checkpoints.py:415] |-> gpu_4/res2_2_branch2c_w_momentum I0830 19:10:40.096677 52234 checkpoints.py:415] |-> gpu_5/res2_2_branch2c_w_momentum I0830 19:10:40.096887 52234 checkpoints.py:415] |-> gpu_6/res2_2_branch2c_w_momentum I0830 19:10:40.097080 52234 checkpoints.py:415] |-> gpu_7/res2_2_branch2c_w_momentum I0830 19:10:40.097394 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2a_w_momentum to I0830 19:10:40.097512 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2a_w_momentum I0830 19:10:40.097810 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2a_w_momentum I0830 19:10:40.098068 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2a_w_momentum I0830 19:10:40.098349 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2a_w_momentum I0830 19:10:40.098604 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2a_w_momentum I0830 19:10:40.098911 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2a_w_momentum I0830 19:10:40.099195 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2a_w_momentum I0830 19:10:40.099601 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2b_w_momentum to I0830 19:10:40.099706 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2b_w_momentum I0830 19:10:40.100047 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2b_w_momentum I0830 19:10:40.100359 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2b_w_momentum I0830 19:10:40.100677 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2b_w_momentum I0830 19:10:40.101059 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2b_w_momentum I0830 19:10:40.101430 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2b_w_momentum I0830 19:10:40.101802 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2b_w_momentum I0830 19:10:40.102233 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch2c_w_momentum to I0830 19:10:40.102359 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch2c_w_momentum I0830 19:10:40.102623 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch2c_w_momentum I0830 19:10:40.102905 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch2c_w_momentum I0830 19:10:40.103212 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch2c_w_momentum I0830 19:10:40.103511 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch2c_w_momentum I0830 19:10:40.103795 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch2c_w_momentum I0830 19:10:40.104043 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch2c_w_momentum I0830 19:10:40.104480 52234 checkpoints.py:410] Broadcasting gpu_0/res3_0_branch1_w_momentum to I0830 19:10:40.104613 52234 checkpoints.py:415] |-> gpu_1/res3_0_branch1_w_momentum I0830 19:10:40.104939 52234 checkpoints.py:415] |-> gpu_2/res3_0_branch1_w_momentum I0830 19:10:40.105314 52234 checkpoints.py:415] |-> gpu_3/res3_0_branch1_w_momentum I0830 19:10:40.105631 52234 checkpoints.py:415] |-> gpu_4/res3_0_branch1_w_momentum I0830 19:10:40.105906 52234 checkpoints.py:415] |-> gpu_5/res3_0_branch1_w_momentum I0830 19:10:40.106184 52234 checkpoints.py:415] |-> gpu_6/res3_0_branch1_w_momentum I0830 19:10:40.106502 52234 checkpoints.py:415] |-> gpu_7/res3_0_branch1_w_momentum I0830 19:10:40.106892 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2a_w_momentum to I0830 19:10:40.107016 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2a_w_momentum I0830 19:10:40.107275 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2a_w_momentum I0830 19:10:40.107534 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2a_w_momentum I0830 19:10:40.107765 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2a_w_momentum I0830 19:10:40.107991 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2a_w_momentum I0830 19:10:40.108218 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2a_w_momentum I0830 19:10:40.108460 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2a_w_momentum I0830 19:10:40.108901 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2b_w_momentum to I0830 19:10:40.109038 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2b_w_momentum I0830 19:10:40.109391 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2b_w_momentum I0830 19:10:40.109724 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2b_w_momentum I0830 19:10:40.110054 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2b_w_momentum I0830 19:10:40.110382 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2b_w_momentum I0830 19:10:40.110704 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2b_w_momentum I0830 19:10:40.111022 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2b_w_momentum I0830 19:10:40.111449 52234 checkpoints.py:410] Broadcasting gpu_0/res3_1_branch2c_w_momentum to I0830 19:10:40.111581 52234 checkpoints.py:415] |-> gpu_1/res3_1_branch2c_w_momentum I0830 19:10:40.111829 52234 checkpoints.py:415] |-> gpu_2/res3_1_branch2c_w_momentum I0830 19:10:40.112059 52234 checkpoints.py:415] |-> gpu_3/res3_1_branch2c_w_momentum I0830 19:10:40.112293 52234 checkpoints.py:415] |-> gpu_4/res3_1_branch2c_w_momentum I0830 19:10:40.112523 52234 checkpoints.py:415] |-> gpu_5/res3_1_branch2c_w_momentum I0830 19:10:40.112752 52234 checkpoints.py:415] |-> gpu_6/res3_1_branch2c_w_momentum I0830 19:10:40.112997 52234 checkpoints.py:415] |-> gpu_7/res3_1_branch2c_w_momentum I0830 19:10:40.113411 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_theta_w_momentum to I0830 19:10:40.113567 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_theta_w_momentum I0830 19:10:40.113903 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_theta_w_momentum I0830 19:10:40.114219 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_theta_w_momentum I0830 19:10:40.114521 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_theta_w_momentum I0830 19:10:40.114813 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_theta_w_momentum I0830 19:10:40.115134 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_theta_w_momentum I0830 19:10:40.115514 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_theta_w_momentum I0830 19:10:40.115865 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_theta_b_momentum to I0830 19:10:40.115993 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_theta_b_momentum I0830 19:10:40.116178 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_theta_b_momentum I0830 19:10:40.116362 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_theta_b_momentum I0830 19:10:40.116538 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_theta_b_momentum I0830 19:10:40.116717 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_theta_b_momentum I0830 19:10:40.116919 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_theta_b_momentum I0830 19:10:40.117100 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_theta_b_momentum I0830 19:10:40.117440 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_phi_w_momentum to I0830 19:10:40.117561 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_phi_w_momentum I0830 19:10:40.117877 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_phi_w_momentum I0830 19:10:40.118181 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_phi_w_momentum I0830 19:10:40.118475 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_phi_w_momentum I0830 19:10:40.118752 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_phi_w_momentum I0830 19:10:40.119045 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_phi_w_momentum I0830 19:10:40.119343 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_phi_w_momentum I0830 19:10:40.119678 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_phi_b_momentum to I0830 19:10:40.119785 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_phi_b_momentum I0830 19:10:40.119955 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_phi_b_momentum I0830 19:10:40.120129 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_phi_b_momentum I0830 19:10:40.120352 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_phi_b_momentum I0830 19:10:40.120573 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_phi_b_momentum I0830 19:10:40.120745 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_phi_b_momentum I0830 19:10:40.120918 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_phi_b_momentum I0830 19:10:40.121232 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_g_w_momentum to I0830 19:10:40.121344 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_g_w_momentum I0830 19:10:40.121660 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_g_w_momentum I0830 19:10:40.121970 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_g_w_momentum I0830 19:10:40.122262 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_g_w_momentum I0830 19:10:40.122558 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_g_w_momentum I0830 19:10:40.122839 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_g_w_momentum I0830 19:10:40.123123 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_g_w_momentum I0830 19:10:40.123457 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_g_b_momentum to I0830 19:10:40.123585 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_g_b_momentum I0830 19:10:40.123774 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_g_b_momentum I0830 19:10:40.123980 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_g_b_momentum I0830 19:10:40.124202 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_g_b_momentum I0830 19:10:40.124406 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_g_b_momentum I0830 19:10:40.124594 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_g_b_momentum I0830 19:10:40.124800 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_g_b_momentum I0830 19:10:40.125118 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_out_w_momentum to I0830 19:10:40.125272 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_out_w_momentum I0830 19:10:40.125612 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_out_w_momentum I0830 19:10:40.125967 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_out_w_momentum I0830 19:10:40.126271 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_out_w_momentum I0830 19:10:40.126604 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_out_w_momentum I0830 19:10:40.126895 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_out_w_momentum I0830 19:10:40.127192 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_out_w_momentum I0830 19:10:40.127536 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_1_out_b_momentum to I0830 19:10:40.127672 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_1_out_b_momentum I0830 19:10:40.127897 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_1_out_b_momentum I0830 19:10:40.128101 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_1_out_b_momentum I0830 19:10:40.128281 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_1_out_b_momentum I0830 19:10:40.128478 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_1_out_b_momentum I0830 19:10:40.128654 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_1_out_b_momentum I0830 19:10:40.128828 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_1_out_b_momentum I0830 19:10:40.129199 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2a_w_momentum to I0830 19:10:40.129328 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2a_w_momentum I0830 19:10:40.129724 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2a_w_momentum I0830 19:10:40.130053 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2a_w_momentum I0830 19:10:40.130398 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2a_w_momentum I0830 19:10:40.130742 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2a_w_momentum I0830 19:10:40.131136 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2a_w_momentum I0830 19:10:40.131469 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2a_w_momentum I0830 19:10:40.131947 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2b_w_momentum to I0830 19:10:40.132102 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2b_w_momentum I0830 19:10:40.132500 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2b_w_momentum I0830 19:10:40.132843 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2b_w_momentum I0830 19:10:40.133162 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2b_w_momentum I0830 19:10:40.133485 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2b_w_momentum I0830 19:10:40.133800 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2b_w_momentum I0830 19:10:40.134090 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2b_w_momentum I0830 19:10:40.134491 52234 checkpoints.py:410] Broadcasting gpu_0/res3_2_branch2c_w_momentum to I0830 19:10:40.134641 52234 checkpoints.py:415] |-> gpu_1/res3_2_branch2c_w_momentum I0830 19:10:40.134911 52234 checkpoints.py:415] |-> gpu_2/res3_2_branch2c_w_momentum I0830 19:10:40.135198 52234 checkpoints.py:415] |-> gpu_3/res3_2_branch2c_w_momentum I0830 19:10:40.135498 52234 checkpoints.py:415] |-> gpu_4/res3_2_branch2c_w_momentum I0830 19:10:40.135746 52234 checkpoints.py:415] |-> gpu_5/res3_2_branch2c_w_momentum I0830 19:10:40.136019 52234 checkpoints.py:415] |-> gpu_6/res3_2_branch2c_w_momentum I0830 19:10:40.136251 52234 checkpoints.py:415] |-> gpu_7/res3_2_branch2c_w_momentum I0830 19:10:40.136600 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2a_w_momentum to I0830 19:10:40.136743 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2a_w_momentum I0830 19:10:40.137044 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2a_w_momentum I0830 19:10:40.137336 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2a_w_momentum I0830 19:10:40.137567 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2a_w_momentum I0830 19:10:40.137794 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2a_w_momentum I0830 19:10:40.138021 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2a_w_momentum I0830 19:10:40.138253 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2a_w_momentum I0830 19:10:40.138674 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2b_w_momentum to I0830 19:10:40.138807 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2b_w_momentum I0830 19:10:40.139138 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2b_w_momentum I0830 19:10:40.139450 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2b_w_momentum I0830 19:10:40.139772 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2b_w_momentum I0830 19:10:40.140062 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2b_w_momentum I0830 19:10:40.140347 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2b_w_momentum I0830 19:10:40.140649 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2b_w_momentum I0830 19:10:40.141067 52234 checkpoints.py:410] Broadcasting gpu_0/res3_3_branch2c_w_momentum to I0830 19:10:40.141192 52234 checkpoints.py:415] |-> gpu_1/res3_3_branch2c_w_momentum I0830 19:10:40.141460 52234 checkpoints.py:415] |-> gpu_2/res3_3_branch2c_w_momentum I0830 19:10:40.141774 52234 checkpoints.py:415] |-> gpu_3/res3_3_branch2c_w_momentum I0830 19:10:40.142066 52234 checkpoints.py:415] |-> gpu_4/res3_3_branch2c_w_momentum I0830 19:10:40.142338 52234 checkpoints.py:415] |-> gpu_5/res3_3_branch2c_w_momentum I0830 19:10:40.142637 52234 checkpoints.py:415] |-> gpu_6/res3_3_branch2c_w_momentum I0830 19:10:40.142890 52234 checkpoints.py:415] |-> gpu_7/res3_3_branch2c_w_momentum I0830 19:10:40.143306 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_theta_w_momentum to I0830 19:10:40.143435 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_theta_w_momentum I0830 19:10:40.143763 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_theta_w_momentum I0830 19:10:40.144079 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_theta_w_momentum I0830 19:10:40.144391 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_theta_w_momentum I0830 19:10:40.144695 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_theta_w_momentum I0830 19:10:40.145001 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_theta_w_momentum I0830 19:10:40.145395 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_theta_w_momentum I0830 19:10:40.145740 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_theta_b_momentum to I0830 19:10:40.145864 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_theta_b_momentum I0830 19:10:40.146056 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_theta_b_momentum I0830 19:10:40.146294 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_theta_b_momentum I0830 19:10:40.146513 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_theta_b_momentum I0830 19:10:40.146717 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_theta_b_momentum I0830 19:10:40.146889 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_theta_b_momentum I0830 19:10:40.147062 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_theta_b_momentum I0830 19:10:40.147387 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_phi_w_momentum to I0830 19:10:40.147505 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_phi_w_momentum I0830 19:10:40.147835 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_phi_w_momentum I0830 19:10:40.148200 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_phi_w_momentum I0830 19:10:40.148487 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_phi_w_momentum I0830 19:10:40.148770 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_phi_w_momentum I0830 19:10:40.149074 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_phi_w_momentum I0830 19:10:40.149384 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_phi_w_momentum I0830 19:10:40.149709 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_phi_b_momentum to I0830 19:10:40.149816 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_phi_b_momentum I0830 19:10:40.149990 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_phi_b_momentum I0830 19:10:40.150187 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_phi_b_momentum I0830 19:10:40.150371 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_phi_b_momentum I0830 19:10:40.150546 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_phi_b_momentum I0830 19:10:40.150717 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_phi_b_momentum I0830 19:10:40.150888 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_phi_b_momentum I0830 19:10:40.151202 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_g_w_momentum to I0830 19:10:40.151331 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_g_w_momentum I0830 19:10:40.151652 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_g_w_momentum I0830 19:10:40.151968 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_g_w_momentum I0830 19:10:40.152292 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_g_w_momentum I0830 19:10:40.152585 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_g_w_momentum I0830 19:10:40.152868 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_g_w_momentum I0830 19:10:40.153156 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_g_w_momentum I0830 19:10:40.153511 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_g_b_momentum to I0830 19:10:40.153635 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_g_b_momentum I0830 19:10:40.153811 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_g_b_momentum I0830 19:10:40.153986 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_g_b_momentum I0830 19:10:40.154165 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_g_b_momentum I0830 19:10:40.154360 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_g_b_momentum I0830 19:10:40.154535 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_g_b_momentum I0830 19:10:40.154706 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_g_b_momentum I0830 19:10:40.155019 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_out_w_momentum to I0830 19:10:40.155122 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_out_w_momentum I0830 19:10:40.155441 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_out_w_momentum I0830 19:10:40.155802 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_out_w_momentum I0830 19:10:40.156102 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_out_w_momentum I0830 19:10:40.156388 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_out_w_momentum I0830 19:10:40.156691 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_out_w_momentum I0830 19:10:40.157036 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_out_w_momentum I0830 19:10:40.157371 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv3_3_out_b_momentum to I0830 19:10:40.157495 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv3_3_out_b_momentum I0830 19:10:40.157668 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv3_3_out_b_momentum I0830 19:10:40.157846 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv3_3_out_b_momentum I0830 19:10:40.158046 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv3_3_out_b_momentum I0830 19:10:40.158262 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv3_3_out_b_momentum I0830 19:10:40.158480 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv3_3_out_b_momentum I0830 19:10:40.158689 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv3_3_out_b_momentum I0830 19:10:40.159171 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2a_w_momentum to I0830 19:10:40.159276 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2a_w_momentum I0830 19:10:40.159784 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2a_w_momentum I0830 19:10:40.160243 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2a_w_momentum I0830 19:10:40.160696 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2a_w_momentum I0830 19:10:40.161156 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2a_w_momentum I0830 19:10:40.161602 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2a_w_momentum I0830 19:10:40.162033 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2a_w_momentum I0830 19:10:40.162837 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2b_w_momentum to I0830 19:10:40.162970 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2b_w_momentum I0830 19:10:40.163623 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2b_w_momentum I0830 19:10:40.164180 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2b_w_momentum I0830 19:10:40.164715 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2b_w_momentum I0830 19:10:40.165226 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2b_w_momentum I0830 19:10:40.165776 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2b_w_momentum I0830 19:10:40.166325 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2b_w_momentum I0830 19:10:40.167135 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch2c_w_momentum to I0830 19:10:40.167313 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch2c_w_momentum I0830 19:10:40.167797 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch2c_w_momentum I0830 19:10:40.168255 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch2c_w_momentum I0830 19:10:40.168669 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch2c_w_momentum I0830 19:10:40.169064 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch2c_w_momentum I0830 19:10:40.169493 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch2c_w_momentum I0830 19:10:40.169904 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch2c_w_momentum I0830 19:10:40.170689 52234 checkpoints.py:410] Broadcasting gpu_0/res4_0_branch1_w_momentum to I0830 19:10:40.170846 52234 checkpoints.py:415] |-> gpu_1/res4_0_branch1_w_momentum I0830 19:10:40.171503 52234 checkpoints.py:415] |-> gpu_2/res4_0_branch1_w_momentum I0830 19:10:40.172013 52234 checkpoints.py:415] |-> gpu_3/res4_0_branch1_w_momentum I0830 19:10:40.172513 52234 checkpoints.py:415] |-> gpu_4/res4_0_branch1_w_momentum I0830 19:10:40.173036 52234 checkpoints.py:415] |-> gpu_5/res4_0_branch1_w_momentum I0830 19:10:40.173552 52234 checkpoints.py:415] |-> gpu_6/res4_0_branch1_w_momentum I0830 19:10:40.174034 52234 checkpoints.py:415] |-> gpu_7/res4_0_branch1_w_momentum I0830 19:10:40.174787 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2a_w_momentum to I0830 19:10:40.174917 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2a_w_momentum I0830 19:10:40.175379 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2a_w_momentum I0830 19:10:40.175791 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2a_w_momentum I0830 19:10:40.176186 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2a_w_momentum I0830 19:10:40.176573 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2a_w_momentum I0830 19:10:40.176990 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2a_w_momentum I0830 19:10:40.177381 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2a_w_momentum I0830 19:10:40.178210 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2b_w_momentum to I0830 19:10:40.178348 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2b_w_momentum I0830 19:10:40.179001 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2b_w_momentum I0830 19:10:40.179603 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2b_w_momentum I0830 19:10:40.180171 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2b_w_momentum I0830 19:10:40.180730 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2b_w_momentum I0830 19:10:40.181236 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2b_w_momentum I0830 19:10:40.181826 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2b_w_momentum I0830 19:10:40.182585 52234 checkpoints.py:410] Broadcasting gpu_0/res4_1_branch2c_w_momentum to I0830 19:10:40.182720 52234 checkpoints.py:415] |-> gpu_1/res4_1_branch2c_w_momentum I0830 19:10:40.183168 52234 checkpoints.py:415] |-> gpu_2/res4_1_branch2c_w_momentum I0830 19:10:40.183599 52234 checkpoints.py:415] |-> gpu_3/res4_1_branch2c_w_momentum I0830 19:10:40.183996 52234 checkpoints.py:415] |-> gpu_4/res4_1_branch2c_w_momentum I0830 19:10:40.184408 52234 checkpoints.py:415] |-> gpu_5/res4_1_branch2c_w_momentum I0830 19:10:40.184803 52234 checkpoints.py:415] |-> gpu_6/res4_1_branch2c_w_momentum I0830 19:10:40.185235 52234 checkpoints.py:415] |-> gpu_7/res4_1_branch2c_w_momentum I0830 19:10:40.186017 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_theta_w_momentum to I0830 19:10:40.186147 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_theta_w_momentum I0830 19:10:40.186791 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_theta_w_momentum I0830 19:10:40.187335 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_theta_w_momentum I0830 19:10:40.187813 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_theta_w_momentum I0830 19:10:40.188303 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_theta_w_momentum I0830 19:10:40.188818 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_theta_w_momentum I0830 19:10:40.189327 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_theta_w_momentum I0830 19:10:40.189888 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_theta_b_momentum to I0830 19:10:40.190001 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_theta_b_momentum I0830 19:10:40.190195 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_theta_b_momentum I0830 19:10:40.190392 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_theta_b_momentum I0830 19:10:40.190637 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_theta_b_momentum I0830 19:10:40.190881 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_theta_b_momentum I0830 19:10:40.191063 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_theta_b_momentum I0830 19:10:40.191273 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_theta_b_momentum I0830 19:10:40.191832 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_phi_w_momentum to I0830 19:10:40.192004 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_phi_w_momentum I0830 19:10:40.192636 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_phi_w_momentum I0830 19:10:40.193147 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_phi_w_momentum I0830 19:10:40.193650 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_phi_w_momentum I0830 19:10:40.194175 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_phi_w_momentum I0830 19:10:40.194689 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_phi_w_momentum I0830 19:10:40.195187 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_phi_w_momentum I0830 19:10:40.195755 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_phi_b_momentum to I0830 19:10:40.195868 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_phi_b_momentum I0830 19:10:40.196046 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_phi_b_momentum I0830 19:10:40.196225 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_phi_b_momentum I0830 19:10:40.196414 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_phi_b_momentum I0830 19:10:40.196618 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_phi_b_momentum I0830 19:10:40.196810 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_phi_b_momentum I0830 19:10:40.197033 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_phi_b_momentum I0830 19:10:40.197597 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_g_w_momentum to I0830 19:10:40.197721 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_g_w_momentum I0830 19:10:40.198357 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_g_w_momentum I0830 19:10:40.198880 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_g_w_momentum I0830 19:10:40.199371 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_g_w_momentum I0830 19:10:40.199851 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_g_w_momentum I0830 19:10:40.200365 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_g_w_momentum I0830 19:10:40.200848 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_g_w_momentum I0830 19:10:40.201486 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_g_b_momentum to I0830 19:10:40.201617 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_g_b_momentum I0830 19:10:40.201798 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_g_b_momentum I0830 19:10:40.201978 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_g_b_momentum I0830 19:10:40.202246 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_g_b_momentum I0830 19:10:40.202470 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_g_b_momentum I0830 19:10:40.202740 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_g_b_momentum I0830 19:10:40.202953 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_g_b_momentum I0830 19:10:40.203560 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_out_w_momentum to I0830 19:10:40.203715 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_out_w_momentum I0830 19:10:40.204386 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_out_w_momentum I0830 19:10:40.204952 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_out_w_momentum I0830 19:10:40.205522 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_out_w_momentum I0830 19:10:40.206039 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_out_w_momentum I0830 19:10:40.206537 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_out_w_momentum I0830 19:10:40.207098 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_out_w_momentum I0830 19:10:40.207643 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_1_out_b_momentum to I0830 19:10:40.207789 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_1_out_b_momentum I0830 19:10:40.207988 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_1_out_b_momentum I0830 19:10:40.208208 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_1_out_b_momentum I0830 19:10:40.208420 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_1_out_b_momentum I0830 19:10:40.208606 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_1_out_b_momentum I0830 19:10:40.208796 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_1_out_b_momentum I0830 19:10:40.208972 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_1_out_b_momentum I0830 19:10:40.209676 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2a_w_momentum to I0830 19:10:40.209819 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2a_w_momentum I0830 19:10:40.210606 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2a_w_momentum I0830 19:10:40.211222 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2a_w_momentum I0830 19:10:40.211822 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2a_w_momentum I0830 19:10:40.212568 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2a_w_momentum I0830 19:10:40.213181 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2a_w_momentum I0830 19:10:40.213798 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2a_w_momentum I0830 19:10:40.214871 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2b_w_momentum to I0830 19:10:40.215013 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2b_w_momentum I0830 19:10:40.215707 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2b_w_momentum I0830 19:10:40.216237 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2b_w_momentum I0830 19:10:40.216801 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2b_w_momentum I0830 19:10:40.217360 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2b_w_momentum I0830 19:10:40.217879 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2b_w_momentum I0830 19:10:40.218411 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2b_w_momentum I0830 19:10:40.219207 52234 checkpoints.py:410] Broadcasting gpu_0/res4_2_branch2c_w_momentum to I0830 19:10:40.219350 52234 checkpoints.py:415] |-> gpu_1/res4_2_branch2c_w_momentum I0830 19:10:40.219794 52234 checkpoints.py:415] |-> gpu_2/res4_2_branch2c_w_momentum I0830 19:10:40.220175 52234 checkpoints.py:415] |-> gpu_3/res4_2_branch2c_w_momentum I0830 19:10:40.220567 52234 checkpoints.py:415] |-> gpu_4/res4_2_branch2c_w_momentum I0830 19:10:40.220957 52234 checkpoints.py:415] |-> gpu_5/res4_2_branch2c_w_momentum I0830 19:10:40.221369 52234 checkpoints.py:415] |-> gpu_6/res4_2_branch2c_w_momentum I0830 19:10:40.221751 52234 checkpoints.py:415] |-> gpu_7/res4_2_branch2c_w_momentum I0830 19:10:40.222387 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2a_w_momentum to I0830 19:10:40.222589 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2a_w_momentum I0830 19:10:40.223067 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2a_w_momentum I0830 19:10:40.223495 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2a_w_momentum I0830 19:10:40.223898 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2a_w_momentum I0830 19:10:40.224337 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2a_w_momentum I0830 19:10:40.224756 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2a_w_momentum I0830 19:10:40.225128 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2a_w_momentum I0830 19:10:40.225972 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2b_w_momentum to I0830 19:10:40.226099 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2b_w_momentum I0830 19:10:40.226759 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2b_w_momentum I0830 19:10:40.227294 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2b_w_momentum I0830 19:10:40.227869 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2b_w_momentum I0830 19:10:40.228399 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2b_w_momentum I0830 19:10:40.228902 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2b_w_momentum I0830 19:10:40.229409 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2b_w_momentum I0830 19:10:40.230195 52234 checkpoints.py:410] Broadcasting gpu_0/res4_3_branch2c_w_momentum to I0830 19:10:40.230354 52234 checkpoints.py:415] |-> gpu_1/res4_3_branch2c_w_momentum I0830 19:10:40.230819 52234 checkpoints.py:415] |-> gpu_2/res4_3_branch2c_w_momentum I0830 19:10:40.231211 52234 checkpoints.py:415] |-> gpu_3/res4_3_branch2c_w_momentum I0830 19:10:40.231597 52234 checkpoints.py:415] |-> gpu_4/res4_3_branch2c_w_momentum I0830 19:10:40.231993 52234 checkpoints.py:415] |-> gpu_5/res4_3_branch2c_w_momentum I0830 19:10:40.232412 52234 checkpoints.py:415] |-> gpu_6/res4_3_branch2c_w_momentum I0830 19:10:40.232795 52234 checkpoints.py:415] |-> gpu_7/res4_3_branch2c_w_momentum I0830 19:10:40.233575 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_theta_w_momentum to I0830 19:10:40.233838 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_theta_w_momentum I0830 19:10:40.234479 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_theta_w_momentum I0830 19:10:40.235038 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_theta_w_momentum I0830 19:10:40.235539 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_theta_w_momentum I0830 19:10:40.236119 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_theta_w_momentum I0830 19:10:40.236665 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_theta_w_momentum I0830 19:10:40.237176 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_theta_w_momentum I0830 19:10:40.237713 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_theta_b_momentum to I0830 19:10:40.237843 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_theta_b_momentum I0830 19:10:40.238029 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_theta_b_momentum I0830 19:10:40.238224 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_theta_b_momentum I0830 19:10:40.238441 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_theta_b_momentum I0830 19:10:40.238663 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_theta_b_momentum I0830 19:10:40.238852 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_theta_b_momentum I0830 19:10:40.239052 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_theta_b_momentum I0830 19:10:40.239623 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_phi_w_momentum to I0830 19:10:40.239758 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_phi_w_momentum I0830 19:10:40.240369 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_phi_w_momentum I0830 19:10:40.240839 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_phi_w_momentum I0830 19:10:40.241328 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_phi_w_momentum I0830 19:10:40.241842 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_phi_w_momentum I0830 19:10:40.242341 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_phi_w_momentum I0830 19:10:40.242839 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_phi_w_momentum I0830 19:10:40.243392 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_phi_b_momentum to I0830 19:10:40.243541 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_phi_b_momentum I0830 19:10:40.243762 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_phi_b_momentum I0830 19:10:40.244059 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_phi_b_momentum I0830 19:10:40.244261 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_phi_b_momentum I0830 19:10:40.244447 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_phi_b_momentum I0830 19:10:40.244622 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_phi_b_momentum I0830 19:10:40.244834 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_phi_b_momentum I0830 19:10:40.245423 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_g_w_momentum to I0830 19:10:40.245543 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_g_w_momentum I0830 19:10:40.246202 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_g_w_momentum I0830 19:10:40.246736 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_g_w_momentum I0830 19:10:40.247315 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_g_w_momentum I0830 19:10:40.247828 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_g_w_momentum I0830 19:10:40.248358 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_g_w_momentum I0830 19:10:40.248850 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_g_w_momentum I0830 19:10:40.249392 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_g_b_momentum to I0830 19:10:40.249527 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_g_b_momentum I0830 19:10:40.249722 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_g_b_momentum I0830 19:10:40.249934 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_g_b_momentum I0830 19:10:40.250153 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_g_b_momentum I0830 19:10:40.250354 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_g_b_momentum I0830 19:10:40.250545 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_g_b_momentum I0830 19:10:40.250720 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_g_b_momentum I0830 19:10:40.251280 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_out_w_momentum to I0830 19:10:40.251413 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_out_w_momentum I0830 19:10:40.252017 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_out_w_momentum I0830 19:10:40.252545 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_out_w_momentum I0830 19:10:40.253021 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_out_w_momentum I0830 19:10:40.253516 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_out_w_momentum I0830 19:10:40.254028 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_out_w_momentum I0830 19:10:40.254590 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_out_w_momentum I0830 19:10:40.255136 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_3_out_b_momentum to I0830 19:10:40.255292 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_3_out_b_momentum I0830 19:10:40.255476 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_3_out_b_momentum I0830 19:10:40.255656 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_3_out_b_momentum I0830 19:10:40.255861 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_3_out_b_momentum I0830 19:10:40.256051 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_3_out_b_momentum I0830 19:10:40.256242 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_3_out_b_momentum I0830 19:10:40.256464 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_3_out_b_momentum I0830 19:10:40.257154 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2a_w_momentum to I0830 19:10:40.257344 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2a_w_momentum I0830 19:10:40.258147 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2a_w_momentum I0830 19:10:40.258774 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2a_w_momentum I0830 19:10:40.259376 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2a_w_momentum I0830 19:10:40.259974 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2a_w_momentum I0830 19:10:40.260551 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2a_w_momentum I0830 19:10:40.261169 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2a_w_momentum I0830 19:10:40.262187 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2b_w_momentum to I0830 19:10:40.262353 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2b_w_momentum I0830 19:10:40.263015 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2b_w_momentum I0830 19:10:40.263552 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2b_w_momentum I0830 19:10:40.264058 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2b_w_momentum I0830 19:10:40.264565 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2b_w_momentum I0830 19:10:40.265098 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2b_w_momentum I0830 19:10:40.265665 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2b_w_momentum I0830 19:10:40.266463 52234 checkpoints.py:410] Broadcasting gpu_0/res4_4_branch2c_w_momentum to I0830 19:10:40.266649 52234 checkpoints.py:415] |-> gpu_1/res4_4_branch2c_w_momentum I0830 19:10:40.267113 52234 checkpoints.py:415] |-> gpu_2/res4_4_branch2c_w_momentum I0830 19:10:40.267529 52234 checkpoints.py:415] |-> gpu_3/res4_4_branch2c_w_momentum I0830 19:10:40.267973 52234 checkpoints.py:415] |-> gpu_4/res4_4_branch2c_w_momentum I0830 19:10:40.268372 52234 checkpoints.py:415] |-> gpu_5/res4_4_branch2c_w_momentum I0830 19:10:40.268831 52234 checkpoints.py:415] |-> gpu_6/res4_4_branch2c_w_momentum I0830 19:10:40.269212 52234 checkpoints.py:415] |-> gpu_7/res4_4_branch2c_w_momentum I0830 19:10:40.269848 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2a_w_momentum to I0830 19:10:40.270006 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2a_w_momentum I0830 19:10:40.270477 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2a_w_momentum I0830 19:10:40.270872 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2a_w_momentum I0830 19:10:40.271306 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2a_w_momentum I0830 19:10:40.271708 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2a_w_momentum I0830 19:10:40.272088 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2a_w_momentum I0830 19:10:40.272499 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2a_w_momentum I0830 19:10:40.273324 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2b_w_momentum to I0830 19:10:40.273452 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2b_w_momentum I0830 19:10:40.274121 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2b_w_momentum I0830 19:10:40.274662 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2b_w_momentum I0830 19:10:40.275187 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2b_w_momentum I0830 19:10:40.275751 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2b_w_momentum I0830 19:10:40.276267 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2b_w_momentum I0830 19:10:40.276791 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2b_w_momentum I0830 19:10:40.277632 52234 checkpoints.py:410] Broadcasting gpu_0/res4_5_branch2c_w_momentum to I0830 19:10:40.277791 52234 checkpoints.py:415] |-> gpu_1/res4_5_branch2c_w_momentum I0830 19:10:40.278275 52234 checkpoints.py:415] |-> gpu_2/res4_5_branch2c_w_momentum I0830 19:10:40.278686 52234 checkpoints.py:415] |-> gpu_3/res4_5_branch2c_w_momentum I0830 19:10:40.279081 52234 checkpoints.py:415] |-> gpu_4/res4_5_branch2c_w_momentum I0830 19:10:40.279510 52234 checkpoints.py:415] |-> gpu_5/res4_5_branch2c_w_momentum I0830 19:10:40.279948 52234 checkpoints.py:415] |-> gpu_6/res4_5_branch2c_w_momentum I0830 19:10:40.280350 52234 checkpoints.py:415] |-> gpu_7/res4_5_branch2c_w_momentum I0830 19:10:40.281106 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_theta_w_momentum to I0830 19:10:40.281282 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_theta_w_momentum I0830 19:10:40.281913 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_theta_w_momentum I0830 19:10:40.282411 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_theta_w_momentum I0830 19:10:40.282898 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_theta_w_momentum I0830 19:10:40.283381 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_theta_w_momentum I0830 19:10:40.283896 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_theta_w_momentum I0830 19:10:40.284411 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_theta_w_momentum I0830 19:10:40.284934 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_theta_b_momentum to I0830 19:10:40.285068 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_theta_b_momentum I0830 19:10:40.285264 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_theta_b_momentum I0830 19:10:40.285494 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_theta_b_momentum I0830 19:10:40.285728 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_theta_b_momentum I0830 19:10:40.285942 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_theta_b_momentum I0830 19:10:40.286150 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_theta_b_momentum I0830 19:10:40.286348 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_theta_b_momentum I0830 19:10:40.286921 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_phi_w_momentum to I0830 19:10:40.287058 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_phi_w_momentum I0830 19:10:40.287750 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_phi_w_momentum I0830 19:10:40.288278 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_phi_w_momentum I0830 19:10:40.288785 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_phi_w_momentum I0830 19:10:40.289325 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_phi_w_momentum I0830 19:10:40.289834 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_phi_w_momentum I0830 19:10:40.290343 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_phi_w_momentum I0830 19:10:40.290909 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_phi_b_momentum to I0830 19:10:40.291076 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_phi_b_momentum I0830 19:10:40.291265 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_phi_b_momentum I0830 19:10:40.291498 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_phi_b_momentum I0830 19:10:40.291692 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_phi_b_momentum I0830 19:10:40.291882 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_phi_b_momentum I0830 19:10:40.292072 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_phi_b_momentum I0830 19:10:40.292273 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_phi_b_momentum I0830 19:10:40.292861 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_g_w_momentum to I0830 19:10:40.292992 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_g_w_momentum I0830 19:10:40.293602 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_g_w_momentum I0830 19:10:40.294079 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_g_w_momentum I0830 19:10:40.294570 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_g_w_momentum I0830 19:10:40.295067 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_g_w_momentum I0830 19:10:40.295619 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_g_w_momentum I0830 19:10:40.296138 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_g_w_momentum I0830 19:10:40.296700 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_g_b_momentum to I0830 19:10:40.296843 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_g_b_momentum I0830 19:10:40.297025 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_g_b_momentum I0830 19:10:40.297219 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_g_b_momentum I0830 19:10:40.297455 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_g_b_momentum I0830 19:10:40.297651 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_g_b_momentum I0830 19:10:40.297826 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_g_b_momentum I0830 19:10:40.298031 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_g_b_momentum I0830 19:10:40.298639 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_out_w_momentum to I0830 19:10:40.298768 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_out_w_momentum I0830 19:10:40.299386 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_out_w_momentum I0830 19:10:40.299937 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_out_w_momentum I0830 19:10:40.300431 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_out_w_momentum I0830 19:10:40.300937 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_out_w_momentum I0830 19:10:40.301434 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_out_w_momentum I0830 19:10:40.301961 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_out_w_momentum I0830 19:10:40.302534 52234 checkpoints.py:410] Broadcasting gpu_0/nonlocal_conv4_5_out_b_momentum to I0830 19:10:40.302689 52234 checkpoints.py:415] |-> gpu_1/nonlocal_conv4_5_out_b_momentum I0830 19:10:40.302931 52234 checkpoints.py:415] |-> gpu_2/nonlocal_conv4_5_out_b_momentum I0830 19:10:40.303165 52234 checkpoints.py:415] |-> gpu_3/nonlocal_conv4_5_out_b_momentum I0830 19:10:40.303386 52234 checkpoints.py:415] |-> gpu_4/nonlocal_conv4_5_out_b_momentum I0830 19:10:40.303602 52234 checkpoints.py:415] |-> gpu_5/nonlocal_conv4_5_out_b_momentum I0830 19:10:40.303838 52234 checkpoints.py:415] |-> gpu_6/nonlocal_conv4_5_out_b_momentum I0830 19:10:40.304066 52234 checkpoints.py:415] |-> gpu_7/nonlocal_conv4_5_out_b_momentum I0830 19:10:40.304662 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2a_w_momentum to I0830 19:10:40.304791 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2a_w_momentum I0830 19:10:40.305404 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2a_w_momentum I0830 19:10:40.305893 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2a_w_momentum I0830 19:10:40.306370 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2a_w_momentum I0830 19:10:40.306866 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2a_w_momentum I0830 19:10:40.307466 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2a_w_momentum I0830 19:10:40.308018 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2a_w_momentum I0830 19:10:40.309834 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2b_w_momentum to I0830 19:10:40.309967 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2b_w_momentum I0830 19:10:40.311773 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2b_w_momentum I0830 19:10:40.313014 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2b_w_momentum I0830 19:10:40.314249 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2b_w_momentum I0830 19:10:40.315484 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2b_w_momentum I0830 19:10:40.316725 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2b_w_momentum I0830 19:10:40.317962 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2b_w_momentum I0830 19:10:40.319947 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch2c_w_momentum to I0830 19:10:40.320142 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch2c_w_momentum I0830 19:10:40.321101 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch2c_w_momentum I0830 19:10:40.321862 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch2c_w_momentum I0830 19:10:40.322571 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch2c_w_momentum I0830 19:10:40.323293 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch2c_w_momentum I0830 19:10:40.324031 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch2c_w_momentum I0830 19:10:40.324764 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch2c_w_momentum I0830 19:10:40.326365 52234 checkpoints.py:410] Broadcasting gpu_0/res5_0_branch1_w_momentum to I0830 19:10:40.326536 52234 checkpoints.py:415] |-> gpu_1/res5_0_branch1_w_momentum I0830 19:10:40.328150 52234 checkpoints.py:415] |-> gpu_2/res5_0_branch1_w_momentum I0830 19:10:40.329426 52234 checkpoints.py:415] |-> gpu_3/res5_0_branch1_w_momentum I0830 19:10:40.330586 52234 checkpoints.py:415] |-> gpu_4/res5_0_branch1_w_momentum I0830 19:10:40.331727 52234 checkpoints.py:415] |-> gpu_5/res5_0_branch1_w_momentum I0830 19:10:40.332851 52234 checkpoints.py:415] |-> gpu_6/res5_0_branch1_w_momentum I0830 19:10:40.333976 52234 checkpoints.py:415] |-> gpu_7/res5_0_branch1_w_momentum I0830 19:10:40.336499 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2a_w_momentum to I0830 19:10:40.336752 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2a_w_momentum I0830 19:10:40.339047 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2a_w_momentum I0830 19:10:40.340766 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2a_w_momentum I0830 19:10:40.342356 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2a_w_momentum I0830 19:10:40.343902 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2a_w_momentum I0830 19:10:40.345453 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2a_w_momentum I0830 19:10:40.346980 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2a_w_momentum I0830 19:10:40.349747 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2b_w_momentum to I0830 19:10:40.350034 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2b_w_momentum I0830 19:10:40.351857 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2b_w_momentum I0830 19:10:40.353090 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2b_w_momentum I0830 19:10:40.354357 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2b_w_momentum I0830 19:10:40.355590 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2b_w_momentum I0830 19:10:40.356862 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2b_w_momentum I0830 19:10:40.358073 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2b_w_momentum I0830 19:10:40.359852 52234 checkpoints.py:410] Broadcasting gpu_0/res5_1_branch2c_w_momentum to I0830 19:10:40.360018 52234 checkpoints.py:415] |-> gpu_1/res5_1_branch2c_w_momentum I0830 19:10:40.361005 52234 checkpoints.py:415] |-> gpu_2/res5_1_branch2c_w_momentum I0830 19:10:40.361800 52234 checkpoints.py:415] |-> gpu_3/res5_1_branch2c_w_momentum I0830 19:10:40.362543 52234 checkpoints.py:415] |-> gpu_4/res5_1_branch2c_w_momentum I0830 19:10:40.363250 52234 checkpoints.py:415] |-> gpu_5/res5_1_branch2c_w_momentum I0830 19:10:40.364022 52234 checkpoints.py:415] |-> gpu_6/res5_1_branch2c_w_momentum I0830 19:10:40.364702 52234 checkpoints.py:415] |-> gpu_7/res5_1_branch2c_w_momentum I0830 19:10:40.365881 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2a_w_momentum to I0830 19:10:40.366044 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2a_w_momentum I0830 19:10:40.367018 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2a_w_momentum I0830 19:10:40.367878 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2a_w_momentum I0830 19:10:40.368594 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2a_w_momentum I0830 19:10:40.369317 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2a_w_momentum I0830 19:10:40.369998 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2a_w_momentum I0830 19:10:40.370712 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2a_w_momentum I0830 19:10:40.372714 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2b_w_momentum to I0830 19:10:40.372844 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2b_w_momentum I0830 19:10:40.374645 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2b_w_momentum I0830 19:10:40.375874 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2b_w_momentum I0830 19:10:40.377121 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2b_w_momentum I0830 19:10:40.378350 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2b_w_momentum I0830 19:10:40.379590 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2b_w_momentum I0830 19:10:40.380852 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2b_w_momentum I0830 19:10:40.382731 52234 checkpoints.py:410] Broadcasting gpu_0/res5_2_branch2c_w_momentum to I0830 19:10:40.382878 52234 checkpoints.py:415] |-> gpu_1/res5_2_branch2c_w_momentum I0830 19:10:40.383836 52234 checkpoints.py:415] |-> gpu_2/res5_2_branch2c_w_momentum I0830 19:10:40.384563 52234 checkpoints.py:415] |-> gpu_3/res5_2_branch2c_w_momentum I0830 19:10:40.385272 52234 checkpoints.py:415] |-> gpu_4/res5_2_branch2c_w_momentum I0830 19:10:40.385977 52234 checkpoints.py:415] |-> gpu_5/res5_2_branch2c_w_momentum I0830 19:10:40.386759 52234 checkpoints.py:415] |-> gpu_6/res5_2_branch2c_w_momentum I0830 19:10:40.387506 52234 checkpoints.py:415] |-> gpu_7/res5_2_branch2c_w_momentum I0830 19:10:40.388351 52234 checkpoints.py:410] Broadcasting gpu_0/pred_w_momentum to I0830 19:10:40.388501 52234 checkpoints.py:415] |-> gpu_1/pred_w_momentum I0830 19:10:40.388870 52234 checkpoints.py:415] |-> gpu_2/pred_w_momentum I0830 19:10:40.389214 52234 checkpoints.py:415] |-> gpu_3/pred_w_momentum I0830 19:10:40.389523 52234 checkpoints.py:415] |-> gpu_4/pred_w_momentum I0830 19:10:40.389828 52234 checkpoints.py:415] |-> gpu_5/pred_w_momentum I0830 19:10:40.390162 52234 checkpoints.py:415] |-> gpu_6/pred_w_momentum I0830 19:10:40.390503 52234 checkpoints.py:415] |-> gpu_7/pred_w_momentum I0830 19:10:40.390865 52234 checkpoints.py:410] Broadcasting gpu_0/pred_b_momentum to I0830 19:10:40.391045 52234 checkpoints.py:415] |-> gpu_1/pred_b_momentum I0830 19:10:40.391259 52234 checkpoints.py:415] |-> gpu_2/pred_b_momentum I0830 19:10:40.391479 52234 checkpoints.py:415] |-> gpu_3/pred_b_momentum I0830 19:10:40.391654 52234 checkpoints.py:415] |-> gpu_4/pred_b_momentum I0830 19:10:40.391829 52234 checkpoints.py:415] |-> gpu_5/pred_b_momentum I0830 19:10:40.392003 52234 checkpoints.py:415] |-> gpu_6/pred_b_momentum I0830 19:10:40.392179 52234 checkpoints.py:415] |-> gpu_7/pred_b_momentum I0830 19:10:40.392477 52234 checkpoints.py:234] Loaded: start_model_iter: 8000; prev_lr: 0.04000000 I0830 19:10:40.392575 52234 train_net.py:143] ------------- Training model... ------------- I0830 19:10:40.392646 52234 metrics.py:136] Resetting train metrics... [I net_async_base.h:206] Using specified CPU pool size: 32; device id: -1 [I net_async_base.h:211] Created new CPU pool, size: 32; device id: -1 [I cuda_nccl_gpu.cc:91] Creating NCCLContext for key: 0:0,1,2,3,4,5,6,7, I0830 19:10:53.683886 52234 misc.py:114] Printing Model: CNN_1 I0830 19:10:53.773133 52234 misc.py:142] data_train : (2, 3, 32, 224, 224) => data_train : (2, 3, 32, 224, 224) ------- (op: StopGradient) I0830 19:10:54.082587 52234 misc.py:142] data_train : (2, 3, 32, 224, 224) => conv1 : (2, 64, 32, 112, 112) ------- (op: Conv) I0830 19:10:54.664244 52234 misc.py:142] conv1 : (2, 64, 32, 112, 112) => res_conv1_bn : (2, 64, 32, 112, 112) ------- (op: AffineNd) I0830 19:10:55.243000 52234 misc.py:142] res_conv1_bn : (2, 64, 32, 112, 112) => res_conv1_bn : (2, 64, 32, 112, 112) ------- (op: Relu) I0830 19:10:55.554739 52234 misc.py:142] res_conv1_bn : (2, 64, 32, 112, 112) => pool1 : (2, 64, 32, 56, 56) ------- (op: MaxPool) I0830 19:10:55.595096 52234 misc.py:142] pool1 : (2, 64, 32, 56, 56) => res2_0_branch2a : (2, 64, 32, 56, 56) ------- (op: Conv) I0830 19:10:55.636472 52234 misc.py:142] res2_0_branch2a : (2, 64, 32, 56, 56) => res2_0_branch2a_bn : (2, 64, 32, 56, 56) ------- (op: AffineNd) I0830 19:10:55.677793 52234 misc.py:142] res2_0_branch2a_bn : (2, 64, 32, 56, 56) => res2_0_branch2a_bn : (2, 64, 32, 56, 56) ------- (op: Relu) I0830 19:10:55.719039 52234 misc.py:142] res2_0_branch2a_bn : (2, 64, 32, 56, 56) => res2_0_branch2b : (2, 64, 32, 56, 56) ------- (op: Conv) I0830 19:10:55.760315 52234 misc.py:142] res2_0_branch2b : (2, 64, 32, 56, 56) => res2_0_branch2b_bn : (2, 64, 32, 56, 56) ------- (op: AffineNd) I0830 19:10:55.801410 52234 misc.py:142] res2_0_branch2b_bn : (2, 64, 32, 56, 56) => res2_0_branch2b_bn : (2, 64, 32, 56, 56) ------- (op: Relu) I0830 19:10:56.113610 52234 misc.py:142] res2_0_branch2b_bn : (2, 64, 32, 56, 56) => res2_0_branch2c : (2, 256, 32, 56, 56) ------- (op: Conv) I0830 19:10:56.694873 52234 misc.py:142] res2_0_branch2c : (2, 256, 32, 56, 56) => res2_0_branch2c_bn : (2, 256, 32, 56, 56) ------- (op: AffineNd) I0830 19:10:57.004296 52234 misc.py:142] pool1 : (2, 64, 32, 56, 56) => res2_0_branch1 : (2, 256, 32, 56, 56) ------- (op: Conv) I0830 19:10:57.585314 52234 misc.py:142] res2_0_branch1 : (2, 256, 32, 56, 56) => res2_0_branch1_bn : (2, 256, 32, 56, 56) ------- (op: AffineNd) I0830 19:10:58.163988 52234 misc.py:142] res2_0_branch2c_bn : (2, 256, 32, 56, 56) => res2_0_branch2c_bn : (2, 256, 32, 56, 56) ------- (op: Sum) I0830 19:10:58.453496 52234 misc.py:142] res2_0_branch1_bn : (2, 256, 32, 56, 56) => res2_0_branch2c_bn : (2, 256, 32, 56, 56) ------| I0830 19:10:59.032018 52234 misc.py:142] res2_0_branch2c_bn : (2, 256, 32, 56, 56) => res2_0_branch2c_bn : (2, 256, 32, 56, 56) ------- (op: Relu) I0830 19:10:59.344924 52234 misc.py:142] res2_0_branch2c_bn : (2, 256, 32, 56, 56) => res2_1_branch2a : (2, 64, 32, 56, 56) ------- (op: Conv) I0830 19:10:59.386230 52234 misc.py:142] res2_1_branch2a : (2, 64, 32, 56, 56) => res2_1_branch2a_bn : (2, 64, 32, 56, 56) ------- (op: AffineNd) I0830 19:10:59.427566 52234 misc.py:142] res2_1_branch2a_bn : (2, 64, 32, 56, 56) => res2_1_branch2a_bn : (2, 64, 32, 56, 56) ------- (op: Relu) I0830 19:10:59.468463 52234 misc.py:142] res2_1_branch2a_bn : (2, 64, 32, 56, 56) => res2_1_branch2b : (2, 64, 32, 56, 56) ------- (op: Conv) I0830 19:10:59.509322 52234 misc.py:142] res2_1_branch2b : (2, 64, 32, 56, 56) => res2_1_branch2b_bn : (2, 64, 32, 56, 56) ------- (op: AffineNd) I0830 19:10:59.550621 52234 misc.py:142] res2_1_branch2b_bn : (2, 64, 32, 56, 56) => res2_1_branch2b_bn : (2, 64, 32, 56, 56) ------- (op: Relu) I0830 19:10:59.862364 52234 misc.py:142] res2_1_branch2b_bn : (2, 64, 32, 56, 56) => res2_1_branch2c : (2, 256, 32, 56, 56) ------- (op: Conv) I0830 19:11:00.443335 52234 misc.py:142] res2_1_branch2c : (2, 256, 32, 56, 56) => res2_1_branch2c_bn : (2, 256, 32, 56, 56) ------- (op: AffineNd) I0830 19:11:01.022062 52234 misc.py:142] res2_1_branch2c_bn : (2, 256, 32, 56, 56) => res2_1_branch2c_bn : (2, 256, 32, 56, 56) ------- (op: Sum) I0830 19:11:01.311734 52234 misc.py:142] res2_0_branch2c_bn : (2, 256, 32, 56, 56) => res2_1_branch2c_bn : (2, 256, 32, 56, 56) ------| I0830 19:11:01.890431 52234 misc.py:142] res2_1_branch2c_bn : (2, 256, 32, 56, 56) => res2_1_branch2c_bn : (2, 256, 32, 56, 56) ------- (op: Relu) I0830 19:11:02.203084 52234 misc.py:142] res2_1_branch2c_bn : (2, 256, 32, 56, 56) => res2_2_branch2a : (2, 64, 32, 56, 56) ------- (op: Conv) I0830 19:11:02.243967 52234 misc.py:142] res2_2_branch2a : (2, 64, 32, 56, 56) => res2_2_branch2a_bn : (2, 64, 32, 56, 56) ------- (op: AffineNd) I0830 19:11:02.285342 52234 misc.py:142] res2_2_branch2a_bn : (2, 64, 32, 56, 56) => res2_2_branch2a_bn : (2, 64, 32, 56, 56) ------- (op: Relu) I0830 19:11:02.326663 52234 misc.py:142] res2_2_branch2a_bn : (2, 64, 32, 56, 56) => res2_2_branch2b : (2, 64, 32, 56, 56) ------- (op: Conv) I0830 19:11:02.367958 52234 misc.py:142] res2_2_branch2b : (2, 64, 32, 56, 56) => res2_2_branch2b_bn : (2, 64, 32, 56, 56) ------- (op: AffineNd) I0830 19:11:02.409277 52234 misc.py:142] res2_2_branch2b_bn : (2, 64, 32, 56, 56) => res2_2_branch2b_bn : (2, 64, 32, 56, 56) ------- (op: Relu) I0830 19:11:02.720830 52234 misc.py:142] res2_2_branch2b_bn : (2, 64, 32, 56, 56) => res2_2_branch2c : (2, 256, 32, 56, 56) ------- (op: Conv) I0830 19:11:03.301590 52234 misc.py:142] res2_2_branch2c : (2, 256, 32, 56, 56) => res2_2_branch2c_bn : (2, 256, 32, 56, 56) ------- (op: AffineNd) I0830 19:11:03.880139 52234 misc.py:142] res2_2_branch2c_bn : (2, 256, 32, 56, 56) => res2_2_branch2c_bn : (2, 256, 32, 56, 56) ------- (op: Sum) I0830 19:11:04.169631 52234 misc.py:142] res2_1_branch2c_bn : (2, 256, 32, 56, 56) => res2_2_branch2c_bn : (2, 256, 32, 56, 56) ------| I0830 19:11:04.747808 52234 misc.py:142] res2_2_branch2c_bn : (2, 256, 32, 56, 56) => res2_2_branch2c_bn : (2, 256, 32, 56, 56) ------- (op: Relu) I0830 19:11:05.179276 52234 misc.py:142] res2_2_branch2c_bn : (2, 256, 32, 56, 56) => pool2 : (2, 256, 16, 56, 56) ------- (op: MaxPool) I0830 19:11:05.346865 52234 misc.py:142] pool2 : (2, 256, 16, 56, 56) => res3_0_branch2a : (2, 128, 16, 56, 56) ------- (op: Conv) I0830 19:11:05.387429 52234 misc.py:142] res3_0_branch2a : (2, 128, 16, 56, 56) => res3_0_branch2a_bn : (2, 128, 16, 56, 56) ------- (op: AffineNd) I0830 19:11:05.428817 52234 misc.py:142] res3_0_branch2a_bn : (2, 128, 16, 56, 56) => res3_0_branch2a_bn : (2, 128, 16, 56, 56) ------- (op: Relu) I0830 19:11:05.454312 52234 misc.py:142] res3_0_branch2a_bn : (2, 128, 16, 56, 56) => res3_0_branch2b : (2, 128, 16, 28, 28) ------- (op: Conv) I0830 19:11:05.464059 52234 misc.py:142] res3_0_branch2b : (2, 128, 16, 28, 28) => res3_0_branch2b_bn : (2, 128, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:05.472462 52234 misc.py:142] res3_0_branch2b_bn : (2, 128, 16, 28, 28) => res3_0_branch2b_bn : (2, 128, 16, 28, 28) ------- (op: Relu) I0830 19:11:05.497777 52234 misc.py:142] res3_0_branch2b_bn : (2, 128, 16, 28, 28) => res3_0_branch2c : (2, 512, 16, 28, 28) ------- (op: Conv) I0830 19:11:05.538532 52234 misc.py:142] res3_0_branch2c : (2, 512, 16, 28, 28) => res3_0_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:05.707036 52234 misc.py:142] pool2 : (2, 256, 16, 56, 56) => res3_0_branch1 : (2, 512, 16, 28, 28) ------- (op: Conv) I0830 19:11:05.747912 52234 misc.py:142] res3_0_branch1 : (2, 512, 16, 28, 28) => res3_0_branch1_bn : (2, 512, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:05.789396 52234 misc.py:142] res3_0_branch2c_bn : (2, 512, 16, 28, 28) => res3_0_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: Sum) I0830 19:11:05.809743 52234 misc.py:142] res3_0_branch1_bn : (2, 512, 16, 28, 28) => res3_0_branch2c_bn : (2, 512, 16, 28, 28) ------| I0830 19:11:05.851059 52234 misc.py:142] res3_0_branch2c_bn : (2, 512, 16, 28, 28) => res3_0_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: Relu) I0830 19:11:05.876497 52234 misc.py:142] res3_0_branch2c_bn : (2, 512, 16, 28, 28) => res3_1_branch2a : (2, 128, 16, 28, 28) ------- (op: Conv) I0830 19:11:05.886231 52234 misc.py:142] res3_1_branch2a : (2, 128, 16, 28, 28) => res3_1_branch2a_bn : (2, 128, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:05.894582 52234 misc.py:142] res3_1_branch2a_bn : (2, 128, 16, 28, 28) => res3_1_branch2a_bn : (2, 128, 16, 28, 28) ------- (op: Relu) I0830 19:11:05.902755 52234 misc.py:142] res3_1_branch2a_bn : (2, 128, 16, 28, 28) => res3_1_branch2b : (2, 128, 16, 28, 28) ------- (op: Conv) I0830 19:11:05.910933 52234 misc.py:142] res3_1_branch2b : (2, 128, 16, 28, 28) => res3_1_branch2b_bn : (2, 128, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:05.919243 52234 misc.py:142] res3_1_branch2b_bn : (2, 128, 16, 28, 28) => res3_1_branch2b_bn : (2, 128, 16, 28, 28) ------- (op: Relu) I0830 19:11:05.944515 52234 misc.py:142] res3_1_branch2b_bn : (2, 128, 16, 28, 28) => res3_1_branch2c : (2, 512, 16, 28, 28) ------- (op: Conv) I0830 19:11:05.985353 52234 misc.py:142] res3_1_branch2c : (2, 512, 16, 28, 28) => res3_1_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:06.026647 52234 misc.py:142] res3_1_branch2c_bn : (2, 512, 16, 28, 28) => res3_1_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: Sum) I0830 19:11:06.047538 52234 misc.py:142] res3_0_branch2c_bn : (2, 512, 16, 28, 28) => res3_1_branch2c_bn : (2, 512, 16, 28, 28) ------| I0830 19:11:06.088761 52234 misc.py:142] res3_1_branch2c_bn : (2, 512, 16, 28, 28) => res3_1_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: Relu) I0830 19:11:06.130043 52234 misc.py:142] res3_1_branch2c_bn : (2, 512, 16, 28, 28) => res3_1_branch2c_bn_trans : (8, 4, 512, 28, 28) ------- (op: Transpose) I0830 19:11:06.171334 52234 misc.py:142] res3_1_branch2c_bn_trans : (8, 4, 512, 28, 28) => res3_1_branch2c_bn_trans : (8, 4, 512, 28, 28) ------- (op: Reshape) I0830 19:11:06.212660 52234 misc.py:142] res3_1_branch2c_bn_trans : (8, 4, 512, 28, 28) => res3_1_branch2c_bn_trans_trans : (8, 512, 4, 28, 28) ------- (op: Transpose) I0830 19:11:06.242801 52234 misc.py:142] res3_1_branch2c_bn_trans_trans: (8, 512, 4, 28, 28) => nonlocal_conv3_1_theta : (8, 256, 3136) ------- (op: Conv) I0830 19:11:06.268372 52234 misc.py:142] res3_1_branch2c_bn_trans_trans: (8, 512, 4, 28, 28) => nonlocal_conv3_1_pool : (8, 512, 4, 14, 14) ------- (op: MaxPool) I0830 19:11:06.276712 52234 misc.py:142] nonlocal_conv3_1_pool : (8, 512, 4, 14, 14) => nonlocal_conv3_1_phi : (8, 256, 784) ------- (op: Conv) I0830 19:11:06.283603 52234 misc.py:142] nonlocal_conv3_1_pool : (8, 512, 4, 14, 14) => nonlocal_conv3_1_g : (8, 256, 784) ------- (op: Conv) I0830 19:11:06.303414 52234 misc.py:142] nonlocal_conv3_1_theta : (8, 256, 3136) => nonlocal_conv3_1_theta : (8, 256, 3136) ------- (op: Reshape) I0830 19:11:06.308569 52234 misc.py:142] nonlocal_conv3_1_phi : (8, 256, 784) => nonlocal_conv3_1_phi : (8, 256, 784) ------- (op: Reshape) I0830 19:11:06.312956 52234 misc.py:142] nonlocal_conv3_1_g : (8, 256, 784) => nonlocal_conv3_1_g : (8, 256, 784) ------- (op: Reshape) I0830 19:11:06.384919 52234 misc.py:142] nonlocal_conv3_1_theta : (8, 256, 3136) => nonlocal_conv3_1_affinity : (8, 3136, 784) ------- (op: BatchMatMul) I0830 19:11:06.388181 52234 misc.py:142] nonlocal_conv3_1_phi : (8, 256, 784) => nonlocal_conv3_1_affinity : (8, 3136, 784) ------| I0830 19:11:06.511181 52234 misc.py:142] nonlocal_conv3_1_affinity : (8, 3136, 784) => nonlocal_conv3_1_affinity : (8, 3136, 784) ------- (op: Scale) I0830 19:11:06.633918 52234 misc.py:142] nonlocal_conv3_1_affinity : (8, 3136, 784) => nonlocal_conv3_1_affinity_prob : (8, 3136, 784) ------- (op: Softmax) I0830 19:11:06.646986 52234 misc.py:142] nonlocal_conv3_1_g : (8, 256, 784) => nonlocal_conv3_1_y : (8, 256, 4, 28, 28) ------- (op: BatchMatMul) I0830 19:11:06.709053 52234 misc.py:142] nonlocal_conv3_1_affinity_prob: (8, 3136, 784) => nonlocal_conv3_1_y : (8, 256, 4, 28, 28) ------| I0830 19:11:06.728341 52234 misc.py:142] nonlocal_conv3_1_y : (8, 256, 4, 28, 28) => nonlocal_conv3_1_y : (8, 256, 4, 28, 28) ------- (op: Reshape) I0830 19:11:06.728921 52234 misc.py:142] nonlocal_conv3_1_theta_shape5d: (5,) => nonlocal_conv3_1_y : (8, 256, 4, 28, 28) ------| I0830 19:11:06.758918 52234 misc.py:142] nonlocal_conv3_1_y : (8, 256, 4, 28, 28) => nonlocal_conv3_1_out : (8, 512, 4, 28, 28) ------- (op: Conv) I0830 19:11:06.799535 52234 misc.py:142] nonlocal_conv3_1_out : (8, 512, 4, 28, 28) => nonlocal_conv3_1_bn : (8, 512, 4, 28, 28) ------- (op: AffineNd) I0830 19:11:06.841029 52234 misc.py:142] res3_1_branch2c_bn_trans_trans: (8, 512, 4, 28, 28) => nonlocal_conv3_1_sum : (8, 512, 4, 28, 28) ------- (op: Sum) I0830 19:11:06.862061 52234 misc.py:142] nonlocal_conv3_1_bn : (8, 512, 4, 28, 28) => nonlocal_conv3_1_sum : (8, 512, 4, 28, 28) ------| I0830 19:11:06.903230 52234 misc.py:142] nonlocal_conv3_1_sum : (8, 512, 4, 28, 28) => nonlocal_conv3_1_sum_trans : (2, 16, 512, 28, 28) ------- (op: Transpose) I0830 19:11:06.943954 52234 misc.py:142] nonlocal_conv3_1_sum_trans : (2, 16, 512, 28, 28) => nonlocal_conv3_1_sum_trans : (2, 16, 512, 28, 28) ------- (op: Reshape) I0830 19:11:06.944645 52234 misc.py:142] res3_1_branch2c_bn_trans_shape5d: (5,) => nonlocal_conv3_1_sum_trans : (2, 16, 512, 28, 28) ------| I0830 19:11:06.985800 52234 misc.py:142] nonlocal_conv3_1_sum_trans : (2, 16, 512, 28, 28) => nonlocal_conv3_1_sum_trans_trans : (2, 512, 16, 28, 28) ------- (op: Transpose) I0830 19:11:07.011281 52234 misc.py:142] nonlocal_conv3_1_sum_trans_trans: (2, 512, 16, 28, 28) => res3_2_branch2a : (2, 128, 16, 28, 28) ------- (op: Conv) I0830 19:11:07.020939 52234 misc.py:142] res3_2_branch2a : (2, 128, 16, 28, 28) => res3_2_branch2a_bn : (2, 128, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:07.029330 52234 misc.py:142] res3_2_branch2a_bn : (2, 128, 16, 28, 28) => res3_2_branch2a_bn : (2, 128, 16, 28, 28) ------- (op: Relu) I0830 19:11:07.037477 52234 misc.py:142] res3_2_branch2a_bn : (2, 128, 16, 28, 28) => res3_2_branch2b : (2, 128, 16, 28, 28) ------- (op: Conv) I0830 19:11:07.045773 52234 misc.py:142] res3_2_branch2b : (2, 128, 16, 28, 28) => res3_2_branch2b_bn : (2, 128, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:07.054068 52234 misc.py:142] res3_2_branch2b_bn : (2, 128, 16, 28, 28) => res3_2_branch2b_bn : (2, 128, 16, 28, 28) ------- (op: Relu) I0830 19:11:07.079425 52234 misc.py:142] res3_2_branch2b_bn : (2, 128, 16, 28, 28) => res3_2_branch2c : (2, 512, 16, 28, 28) ------- (op: Conv) I0830 19:11:07.120256 52234 misc.py:142] res3_2_branch2c : (2, 512, 16, 28, 28) => res3_2_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:07.161820 52234 misc.py:142] res3_2_branch2c_bn : (2, 512, 16, 28, 28) => res3_2_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: Sum) I0830 19:11:07.182826 52234 misc.py:142] nonlocal_conv3_1_sum_trans_trans: (2, 512, 16, 28, 28) => res3_2_branch2c_bn : (2, 512, 16, 28, 28) ------| I0830 19:11:07.224132 52234 misc.py:142] res3_2_branch2c_bn : (2, 512, 16, 28, 28) => res3_2_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: Relu) I0830 19:11:07.249629 52234 misc.py:142] res3_2_branch2c_bn : (2, 512, 16, 28, 28) => res3_3_branch2a : (2, 128, 16, 28, 28) ------- (op: Conv) I0830 19:11:07.259329 52234 misc.py:142] res3_3_branch2a : (2, 128, 16, 28, 28) => res3_3_branch2a_bn : (2, 128, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:07.267664 52234 misc.py:142] res3_3_branch2a_bn : (2, 128, 16, 28, 28) => res3_3_branch2a_bn : (2, 128, 16, 28, 28) ------- (op: Relu) I0830 19:11:07.275809 52234 misc.py:142] res3_3_branch2a_bn : (2, 128, 16, 28, 28) => res3_3_branch2b : (2, 128, 16, 28, 28) ------- (op: Conv) I0830 19:11:07.284060 52234 misc.py:142] res3_3_branch2b : (2, 128, 16, 28, 28) => res3_3_branch2b_bn : (2, 128, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:07.292380 52234 misc.py:142] res3_3_branch2b_bn : (2, 128, 16, 28, 28) => res3_3_branch2b_bn : (2, 128, 16, 28, 28) ------- (op: Relu) I0830 19:11:07.317692 52234 misc.py:142] res3_3_branch2b_bn : (2, 128, 16, 28, 28) => res3_3_branch2c : (2, 512, 16, 28, 28) ------- (op: Conv) I0830 19:11:07.358498 52234 misc.py:142] res3_3_branch2c : (2, 512, 16, 28, 28) => res3_3_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:07.400073 52234 misc.py:142] res3_3_branch2c_bn : (2, 512, 16, 28, 28) => res3_3_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: Sum) I0830 19:11:07.421030 52234 misc.py:142] res3_2_branch2c_bn : (2, 512, 16, 28, 28) => res3_3_branch2c_bn : (2, 512, 16, 28, 28) ------| I0830 19:11:07.462240 52234 misc.py:142] res3_3_branch2c_bn : (2, 512, 16, 28, 28) => res3_3_branch2c_bn : (2, 512, 16, 28, 28) ------- (op: Relu) I0830 19:11:07.503552 52234 misc.py:142] res3_3_branch2c_bn : (2, 512, 16, 28, 28) => res3_3_branch2c_bn_trans : (8, 4, 512, 28, 28) ------- (op: Transpose) I0830 19:11:07.544862 52234 misc.py:142] res3_3_branch2c_bn_trans : (8, 4, 512, 28, 28) => res3_3_branch2c_bn_trans : (8, 4, 512, 28, 28) ------- (op: Reshape) I0830 19:11:07.586229 52234 misc.py:142] res3_3_branch2c_bn_trans : (8, 4, 512, 28, 28) => res3_3_branch2c_bn_trans_trans : (8, 512, 4, 28, 28) ------- (op: Transpose) I0830 19:11:07.616501 52234 misc.py:142] res3_3_branch2c_bn_trans_trans: (8, 512, 4, 28, 28) => nonlocal_conv3_3_theta : (8, 256, 3136) ------- (op: Conv) I0830 19:11:07.641990 52234 misc.py:142] res3_3_branch2c_bn_trans_trans: (8, 512, 4, 28, 28) => nonlocal_conv3_3_pool : (8, 512, 4, 14, 14) ------- (op: MaxPool) I0830 19:11:07.650332 52234 misc.py:142] nonlocal_conv3_3_pool : (8, 512, 4, 14, 14) => nonlocal_conv3_3_phi : (8, 256, 784) ------- (op: Conv) I0830 19:11:07.657316 52234 misc.py:142] nonlocal_conv3_3_pool : (8, 512, 4, 14, 14) => nonlocal_conv3_3_g : (8, 256, 784) ------- (op: Conv) I0830 19:11:07.677069 52234 misc.py:142] nonlocal_conv3_3_theta : (8, 256, 3136) => nonlocal_conv3_3_theta : (8, 256, 3136) ------- (op: Reshape) I0830 19:11:07.682205 52234 misc.py:142] nonlocal_conv3_3_phi : (8, 256, 784) => nonlocal_conv3_3_phi : (8, 256, 784) ------- (op: Reshape) I0830 19:11:07.686633 52234 misc.py:142] nonlocal_conv3_3_g : (8, 256, 784) => nonlocal_conv3_3_g : (8, 256, 784) ------- (op: Reshape) I0830 19:11:07.758537 52234 misc.py:142] nonlocal_conv3_3_theta : (8, 256, 3136) => nonlocal_conv3_3_affinity : (8, 3136, 784) ------- (op: BatchMatMul) I0830 19:11:07.761760 52234 misc.py:142] nonlocal_conv3_3_phi : (8, 256, 784) => nonlocal_conv3_3_affinity : (8, 3136, 784) ------| I0830 19:11:07.884711 52234 misc.py:142] nonlocal_conv3_3_affinity : (8, 3136, 784) => nonlocal_conv3_3_affinity : (8, 3136, 784) ------- (op: Scale) I0830 19:11:08.007363 52234 misc.py:142] nonlocal_conv3_3_affinity : (8, 3136, 784) => nonlocal_conv3_3_affinity_prob : (8, 3136, 784) ------- (op: Softmax) I0830 19:11:08.020647 52234 misc.py:142] nonlocal_conv3_3_g : (8, 256, 784) => nonlocal_conv3_3_y : (8, 256, 4, 28, 28) ------- (op: BatchMatMul) I0830 19:11:08.082607 52234 misc.py:142] nonlocal_conv3_3_affinity_prob: (8, 3136, 784) => nonlocal_conv3_3_y : (8, 256, 4, 28, 28) ------| I0830 19:11:08.101974 52234 misc.py:142] nonlocal_conv3_3_y : (8, 256, 4, 28, 28) => nonlocal_conv3_3_y : (8, 256, 4, 28, 28) ------- (op: Reshape) I0830 19:11:08.102544 52234 misc.py:142] nonlocal_conv3_3_theta_shape5d: (5,) => nonlocal_conv3_3_y : (8, 256, 4, 28, 28) ------| I0830 19:11:08.132550 52234 misc.py:142] nonlocal_conv3_3_y : (8, 256, 4, 28, 28) => nonlocal_conv3_3_out : (8, 512, 4, 28, 28) ------- (op: Conv) I0830 19:11:08.173379 52234 misc.py:142] nonlocal_conv3_3_out : (8, 512, 4, 28, 28) => nonlocal_conv3_3_bn : (8, 512, 4, 28, 28) ------- (op: AffineNd) I0830 19:11:08.214936 52234 misc.py:142] res3_3_branch2c_bn_trans_trans: (8, 512, 4, 28, 28) => nonlocal_conv3_3_sum : (8, 512, 4, 28, 28) ------- (op: Sum) I0830 19:11:08.235882 52234 misc.py:142] nonlocal_conv3_3_bn : (8, 512, 4, 28, 28) => nonlocal_conv3_3_sum : (8, 512, 4, 28, 28) ------| I0830 19:11:08.277203 52234 misc.py:142] nonlocal_conv3_3_sum : (8, 512, 4, 28, 28) => nonlocal_conv3_3_sum_trans : (2, 16, 512, 28, 28) ------- (op: Transpose) I0830 19:11:08.318510 52234 misc.py:142] nonlocal_conv3_3_sum_trans : (2, 16, 512, 28, 28) => nonlocal_conv3_3_sum_trans : (2, 16, 512, 28, 28) ------- (op: Reshape) I0830 19:11:08.319192 52234 misc.py:142] res3_3_branch2c_bn_trans_shape5d: (5,) => nonlocal_conv3_3_sum_trans : (2, 16, 512, 28, 28) ------| I0830 19:11:08.360378 52234 misc.py:142] nonlocal_conv3_3_sum_trans : (2, 16, 512, 28, 28) => nonlocal_conv3_3_sum_trans_trans : (2, 512, 16, 28, 28) ------- (op: Transpose) I0830 19:11:08.390564 52234 misc.py:142] nonlocal_conv3_3_sum_trans_trans: (2, 512, 16, 28, 28) => res4_0_branch2a : (2, 256, 16, 28, 28) ------- (op: Conv) I0830 19:11:08.409813 52234 misc.py:142] res4_0_branch2a : (2, 256, 16, 28, 28) => res4_0_branch2a_bn : (2, 256, 16, 28, 28) ------- (op: AffineNd) I0830 19:11:08.427484 52234 misc.py:142] res4_0_branch2a_bn : (2, 256, 16, 28, 28) => res4_0_branch2a_bn : (2, 256, 16, 28, 28) ------- (op: Relu) I0830 19:11:08.440103 52234 misc.py:142] res4_0_branch2a_bn : (2, 256, 16, 28, 28) => res4_0_branch2b : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.445285 52234 misc.py:142] res4_0_branch2b : (2, 256, 16, 14, 14) => res4_0_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.449950 52234 misc.py:142] res4_0_branch2b_bn : (2, 256, 16, 14, 14) => res4_0_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:08.462565 52234 misc.py:142] res4_0_branch2b_bn : (2, 256, 16, 14, 14) => res4_0_branch2c : (2, 1024, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.481476 52234 misc.py:142] res4_0_branch2c : (2, 1024, 16, 14, 14) => res4_0_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.510324 52234 misc.py:142] nonlocal_conv3_3_sum_trans_trans: (2, 512, 16, 28, 28) => res4_0_branch1 : (2, 1024, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.529648 52234 misc.py:142] res4_0_branch1 : (2, 1024, 16, 14, 14) => res4_0_branch1_bn : (2, 1024, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.547317 52234 misc.py:142] res4_0_branch2c_bn : (2, 1024, 16, 14, 14) => res4_0_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Sum) I0830 19:11:08.556127 52234 misc.py:142] res4_0_branch1_bn : (2, 1024, 16, 14, 14) => res4_0_branch2c_bn : (2, 1024, 16, 14, 14) ------| I0830 19:11:08.573187 52234 misc.py:142] res4_0_branch2c_bn : (2, 1024, 16, 14, 14) => res4_0_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Relu) I0830 19:11:08.585633 52234 misc.py:142] res4_0_branch2c_bn : (2, 1024, 16, 14, 14) => res4_1_branch2a : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.590888 52234 misc.py:142] res4_1_branch2a : (2, 256, 16, 14, 14) => res4_1_branch2a_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.595599 52234 misc.py:142] res4_1_branch2a_bn : (2, 256, 16, 14, 14) => res4_1_branch2a_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:08.599977 52234 misc.py:142] res4_1_branch2a_bn : (2, 256, 16, 14, 14) => res4_1_branch2b : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.604516 52234 misc.py:142] res4_1_branch2b : (2, 256, 16, 14, 14) => res4_1_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.609165 52234 misc.py:142] res4_1_branch2b_bn : (2, 256, 16, 14, 14) => res4_1_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:08.621599 52234 misc.py:142] res4_1_branch2b_bn : (2, 256, 16, 14, 14) => res4_1_branch2c : (2, 1024, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.640321 52234 misc.py:142] res4_1_branch2c : (2, 1024, 16, 14, 14) => res4_1_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.657666 52234 misc.py:142] res4_1_branch2c_bn : (2, 1024, 16, 14, 14) => res4_1_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Sum) I0830 19:11:08.666399 52234 misc.py:142] res4_0_branch2c_bn : (2, 1024, 16, 14, 14) => res4_1_branch2c_bn : (2, 1024, 16, 14, 14) ------| I0830 19:11:08.683502 52234 misc.py:142] res4_1_branch2c_bn : (2, 1024, 16, 14, 14) => res4_1_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Relu) I0830 19:11:08.699078 52234 misc.py:142] res4_1_branch2c_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_1_theta : (2, 512, 3136) ------- (op: Conv) I0830 19:11:08.712101 52234 misc.py:142] res4_1_branch2c_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_1_pool : (2, 1024, 16, 7, 7) ------- (op: MaxPool) I0830 19:11:08.716834 52234 misc.py:142] nonlocal_conv4_1_pool : (2, 1024, 16, 7, 7) => nonlocal_conv4_1_phi : (2, 512, 784) ------- (op: Conv) I0830 19:11:08.720691 52234 misc.py:142] nonlocal_conv4_1_pool : (2, 1024, 16, 7, 7) => nonlocal_conv4_1_g : (2, 512, 784) ------- (op: Conv) I0830 19:11:08.730067 52234 misc.py:142] nonlocal_conv4_1_theta : (2, 512, 3136) => nonlocal_conv4_1_theta : (2, 512, 3136) ------- (op: Reshape) I0830 19:11:08.732799 52234 misc.py:142] nonlocal_conv4_1_phi : (2, 512, 784) => nonlocal_conv4_1_phi : (2, 512, 784) ------- (op: Reshape) I0830 19:11:08.735507 52234 misc.py:142] nonlocal_conv4_1_g : (2, 512, 784) => nonlocal_conv4_1_g : (2, 512, 784) ------- (op: Reshape) I0830 19:11:08.746192 52234 misc.py:142] nonlocal_conv4_1_theta : (2, 512, 3136) => nonlocal_conv4_1_affinity : (2, 3136, 784) ------- (op: BatchMatMul) I0830 19:11:08.747816 52234 misc.py:142] nonlocal_conv4_1_phi : (2, 512, 784) => nonlocal_conv4_1_affinity : (2, 3136, 784) ------| I0830 19:11:08.760063 52234 misc.py:142] nonlocal_conv4_1_affinity : (2, 3136, 784) => nonlocal_conv4_1_affinity : (2, 3136, 784) ------- (op: Scale) I0830 19:11:08.772195 52234 misc.py:142] nonlocal_conv4_1_affinity : (2, 3136, 784) => nonlocal_conv4_1_affinity_prob : (2, 3136, 784) ------- (op: Softmax) I0830 19:11:08.777863 52234 misc.py:142] nonlocal_conv4_1_g : (2, 512, 784) => nonlocal_conv4_1_y : (2, 512, 16, 14, 14) ------- (op: BatchMatMul) I0830 19:11:08.784243 52234 misc.py:142] nonlocal_conv4_1_affinity_prob: (2, 3136, 784) => nonlocal_conv4_1_y : (2, 512, 16, 14, 14) ------| I0830 19:11:08.792351 52234 misc.py:142] nonlocal_conv4_1_y : (2, 512, 16, 14, 14) => nonlocal_conv4_1_y : (2, 512, 16, 14, 14) ------- (op: Reshape) I0830 19:11:08.792767 52234 misc.py:142] nonlocal_conv4_1_theta_shape5d: (5,) => nonlocal_conv4_1_y : (2, 512, 16, 14, 14) ------| I0830 19:11:08.808907 52234 misc.py:142] nonlocal_conv4_1_y : (2, 512, 16, 14, 14) => nonlocal_conv4_1_out : (2, 1024, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.828573 52234 misc.py:142] nonlocal_conv4_1_out : (2, 1024, 16, 14, 14) => nonlocal_conv4_1_bn : (2, 1024, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.845968 52234 misc.py:142] res4_1_branch2c_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_1_sum : (2, 1024, 16, 14, 14) ------- (op: Sum) I0830 19:11:08.854705 52234 misc.py:142] nonlocal_conv4_1_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_1_sum : (2, 1024, 16, 14, 14) ------| I0830 19:11:08.867127 52234 misc.py:142] nonlocal_conv4_1_sum : (2, 1024, 16, 14, 14) => res4_2_branch2a : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.872379 52234 misc.py:142] res4_2_branch2a : (2, 256, 16, 14, 14) => res4_2_branch2a_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.877111 52234 misc.py:142] res4_2_branch2a_bn : (2, 256, 16, 14, 14) => res4_2_branch2a_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:08.881484 52234 misc.py:142] res4_2_branch2a_bn : (2, 256, 16, 14, 14) => res4_2_branch2b : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.886158 52234 misc.py:142] res4_2_branch2b : (2, 256, 16, 14, 14) => res4_2_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.890923 52234 misc.py:142] res4_2_branch2b_bn : (2, 256, 16, 14, 14) => res4_2_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:08.903889 52234 misc.py:142] res4_2_branch2b_bn : (2, 256, 16, 14, 14) => res4_2_branch2c : (2, 1024, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.922696 52234 misc.py:142] res4_2_branch2c : (2, 1024, 16, 14, 14) => res4_2_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.940157 52234 misc.py:142] res4_2_branch2c_bn : (2, 1024, 16, 14, 14) => res4_2_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Sum) I0830 19:11:08.948946 52234 misc.py:142] nonlocal_conv4_1_sum : (2, 1024, 16, 14, 14) => res4_2_branch2c_bn : (2, 1024, 16, 14, 14) ------| I0830 19:11:08.965914 52234 misc.py:142] res4_2_branch2c_bn : (2, 1024, 16, 14, 14) => res4_2_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Relu) I0830 19:11:08.978352 52234 misc.py:142] res4_2_branch2c_bn : (2, 1024, 16, 14, 14) => res4_3_branch2a : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.983613 52234 misc.py:142] res4_3_branch2a : (2, 256, 16, 14, 14) => res4_3_branch2a_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:08.988337 52234 misc.py:142] res4_3_branch2a_bn : (2, 256, 16, 14, 14) => res4_3_branch2a_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:08.992795 52234 misc.py:142] res4_3_branch2a_bn : (2, 256, 16, 14, 14) => res4_3_branch2b : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:08.997354 52234 misc.py:142] res4_3_branch2b : (2, 256, 16, 14, 14) => res4_3_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.002098 52234 misc.py:142] res4_3_branch2b_bn : (2, 256, 16, 14, 14) => res4_3_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.014465 52234 misc.py:142] res4_3_branch2b_bn : (2, 256, 16, 14, 14) => res4_3_branch2c : (2, 1024, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.033056 52234 misc.py:142] res4_3_branch2c : (2, 1024, 16, 14, 14) => res4_3_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.050420 52234 misc.py:142] res4_3_branch2c_bn : (2, 1024, 16, 14, 14) => res4_3_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Sum) I0830 19:11:09.059144 52234 misc.py:142] res4_2_branch2c_bn : (2, 1024, 16, 14, 14) => res4_3_branch2c_bn : (2, 1024, 16, 14, 14) ------| I0830 19:11:09.076195 52234 misc.py:142] res4_3_branch2c_bn : (2, 1024, 16, 14, 14) => res4_3_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.094782 52234 misc.py:142] res4_3_branch2c_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_3_theta : (2, 512, 3136) ------- (op: Conv) I0830 19:11:09.109225 52234 misc.py:142] res4_3_branch2c_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_3_pool : (2, 1024, 16, 7, 7) ------- (op: MaxPool) I0830 19:11:09.113986 52234 misc.py:142] nonlocal_conv4_3_pool : (2, 1024, 16, 7, 7) => nonlocal_conv4_3_phi : (2, 512, 784) ------- (op: Conv) I0830 19:11:09.117907 52234 misc.py:142] nonlocal_conv4_3_pool : (2, 1024, 16, 7, 7) => nonlocal_conv4_3_g : (2, 512, 784) ------- (op: Conv) I0830 19:11:09.127316 52234 misc.py:142] nonlocal_conv4_3_theta : (2, 512, 3136) => nonlocal_conv4_3_theta : (2, 512, 3136) ------- (op: Reshape) I0830 19:11:09.130073 52234 misc.py:142] nonlocal_conv4_3_phi : (2, 512, 784) => nonlocal_conv4_3_phi : (2, 512, 784) ------- (op: Reshape) I0830 19:11:09.132754 52234 misc.py:142] nonlocal_conv4_3_g : (2, 512, 784) => nonlocal_conv4_3_g : (2, 512, 784) ------- (op: Reshape) I0830 19:11:09.143459 52234 misc.py:142] nonlocal_conv4_3_theta : (2, 512, 3136) => nonlocal_conv4_3_affinity : (2, 3136, 784) ------- (op: BatchMatMul) I0830 19:11:09.145080 52234 misc.py:142] nonlocal_conv4_3_phi : (2, 512, 784) => nonlocal_conv4_3_affinity : (2, 3136, 784) ------| I0830 19:11:09.157320 52234 misc.py:142] nonlocal_conv4_3_affinity : (2, 3136, 784) => nonlocal_conv4_3_affinity : (2, 3136, 784) ------- (op: Scale) I0830 19:11:09.169403 52234 misc.py:142] nonlocal_conv4_3_affinity : (2, 3136, 784) => nonlocal_conv4_3_affinity_prob : (2, 3136, 784) ------- (op: Softmax) I0830 19:11:09.175276 52234 misc.py:142] nonlocal_conv4_3_g : (2, 512, 784) => nonlocal_conv4_3_y : (2, 512, 16, 14, 14) ------- (op: BatchMatMul) I0830 19:11:09.181761 52234 misc.py:142] nonlocal_conv4_3_affinity_prob: (2, 3136, 784) => nonlocal_conv4_3_y : (2, 512, 16, 14, 14) ------| I0830 19:11:09.189806 52234 misc.py:142] nonlocal_conv4_3_y : (2, 512, 16, 14, 14) => nonlocal_conv4_3_y : (2, 512, 16, 14, 14) ------- (op: Reshape) I0830 19:11:09.190212 52234 misc.py:142] nonlocal_conv4_3_theta_shape5d: (5,) => nonlocal_conv4_3_y : (2, 512, 16, 14, 14) ------| I0830 19:11:09.206352 52234 misc.py:142] nonlocal_conv4_3_y : (2, 512, 16, 14, 14) => nonlocal_conv4_3_out : (2, 1024, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.226121 52234 misc.py:142] nonlocal_conv4_3_out : (2, 1024, 16, 14, 14) => nonlocal_conv4_3_bn : (2, 1024, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.243560 52234 misc.py:142] res4_3_branch2c_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_3_sum : (2, 1024, 16, 14, 14) ------- (op: Sum) I0830 19:11:09.252310 52234 misc.py:142] nonlocal_conv4_3_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_3_sum : (2, 1024, 16, 14, 14) ------| I0830 19:11:09.264743 52234 misc.py:142] nonlocal_conv4_3_sum : (2, 1024, 16, 14, 14) => res4_4_branch2a : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.269983 52234 misc.py:142] res4_4_branch2a : (2, 256, 16, 14, 14) => res4_4_branch2a_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.274848 52234 misc.py:142] res4_4_branch2a_bn : (2, 256, 16, 14, 14) => res4_4_branch2a_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.279321 52234 misc.py:142] res4_4_branch2a_bn : (2, 256, 16, 14, 14) => res4_4_branch2b : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.283916 52234 misc.py:142] res4_4_branch2b : (2, 256, 16, 14, 14) => res4_4_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.288640 52234 misc.py:142] res4_4_branch2b_bn : (2, 256, 16, 14, 14) => res4_4_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.301098 52234 misc.py:142] res4_4_branch2b_bn : (2, 256, 16, 14, 14) => res4_4_branch2c : (2, 1024, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.319808 52234 misc.py:142] res4_4_branch2c : (2, 1024, 16, 14, 14) => res4_4_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.337377 52234 misc.py:142] res4_4_branch2c_bn : (2, 1024, 16, 14, 14) => res4_4_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Sum) I0830 19:11:09.346149 52234 misc.py:142] nonlocal_conv4_3_sum : (2, 1024, 16, 14, 14) => res4_4_branch2c_bn : (2, 1024, 16, 14, 14) ------| I0830 19:11:09.363138 52234 misc.py:142] res4_4_branch2c_bn : (2, 1024, 16, 14, 14) => res4_4_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.375593 52234 misc.py:142] res4_4_branch2c_bn : (2, 1024, 16, 14, 14) => res4_5_branch2a : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.380904 52234 misc.py:142] res4_5_branch2a : (2, 256, 16, 14, 14) => res4_5_branch2a_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.385716 52234 misc.py:142] res4_5_branch2a_bn : (2, 256, 16, 14, 14) => res4_5_branch2a_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.390139 52234 misc.py:142] res4_5_branch2a_bn : (2, 256, 16, 14, 14) => res4_5_branch2b : (2, 256, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.394820 52234 misc.py:142] res4_5_branch2b : (2, 256, 16, 14, 14) => res4_5_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.399566 52234 misc.py:142] res4_5_branch2b_bn : (2, 256, 16, 14, 14) => res4_5_branch2b_bn : (2, 256, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.411926 52234 misc.py:142] res4_5_branch2b_bn : (2, 256, 16, 14, 14) => res4_5_branch2c : (2, 1024, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.430654 52234 misc.py:142] res4_5_branch2c : (2, 1024, 16, 14, 14) => res4_5_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.448151 52234 misc.py:142] res4_5_branch2c_bn : (2, 1024, 16, 14, 14) => res4_5_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Sum) I0830 19:11:09.456882 52234 misc.py:142] res4_4_branch2c_bn : (2, 1024, 16, 14, 14) => res4_5_branch2c_bn : (2, 1024, 16, 14, 14) ------| I0830 19:11:09.473882 52234 misc.py:142] res4_5_branch2c_bn : (2, 1024, 16, 14, 14) => res4_5_branch2c_bn : (2, 1024, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.489494 52234 misc.py:142] res4_5_branch2c_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_5_theta : (2, 512, 3136) ------- (op: Conv) I0830 19:11:09.502657 52234 misc.py:142] res4_5_branch2c_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_5_pool : (2, 1024, 16, 7, 7) ------- (op: MaxPool) I0830 19:11:09.507427 52234 misc.py:142] nonlocal_conv4_5_pool : (2, 1024, 16, 7, 7) => nonlocal_conv4_5_phi : (2, 512, 784) ------- (op: Conv) I0830 19:11:09.511415 52234 misc.py:142] nonlocal_conv4_5_pool : (2, 1024, 16, 7, 7) => nonlocal_conv4_5_g : (2, 512, 784) ------- (op: Conv) I0830 19:11:09.520874 52234 misc.py:142] nonlocal_conv4_5_theta : (2, 512, 3136) => nonlocal_conv4_5_theta : (2, 512, 3136) ------- (op: Reshape) I0830 19:11:09.523617 52234 misc.py:142] nonlocal_conv4_5_phi : (2, 512, 784) => nonlocal_conv4_5_phi : (2, 512, 784) ------- (op: Reshape) I0830 19:11:09.526279 52234 misc.py:142] nonlocal_conv4_5_g : (2, 512, 784) => nonlocal_conv4_5_g : (2, 512, 784) ------- (op: Reshape) I0830 19:11:09.536962 52234 misc.py:142] nonlocal_conv4_5_theta : (2, 512, 3136) => nonlocal_conv4_5_affinity : (2, 3136, 784) ------- (op: BatchMatMul) I0830 19:11:09.538599 52234 misc.py:142] nonlocal_conv4_5_phi : (2, 512, 784) => nonlocal_conv4_5_affinity : (2, 3136, 784) ------| I0830 19:11:09.550837 52234 misc.py:142] nonlocal_conv4_5_affinity : (2, 3136, 784) => nonlocal_conv4_5_affinity : (2, 3136, 784) ------- (op: Scale) I0830 19:11:09.562905 52234 misc.py:142] nonlocal_conv4_5_affinity : (2, 3136, 784) => nonlocal_conv4_5_affinity_prob : (2, 3136, 784) ------- (op: Softmax) I0830 19:11:09.568588 52234 misc.py:142] nonlocal_conv4_5_g : (2, 512, 784) => nonlocal_conv4_5_y : (2, 512, 16, 14, 14) ------- (op: BatchMatMul) I0830 19:11:09.575026 52234 misc.py:142] nonlocal_conv4_5_affinity_prob: (2, 3136, 784) => nonlocal_conv4_5_y : (2, 512, 16, 14, 14) ------| I0830 19:11:09.583167 52234 misc.py:142] nonlocal_conv4_5_y : (2, 512, 16, 14, 14) => nonlocal_conv4_5_y : (2, 512, 16, 14, 14) ------- (op: Reshape) I0830 19:11:09.583580 52234 misc.py:142] nonlocal_conv4_5_theta_shape5d: (5,) => nonlocal_conv4_5_y : (2, 512, 16, 14, 14) ------| I0830 19:11:09.599668 52234 misc.py:142] nonlocal_conv4_5_y : (2, 512, 16, 14, 14) => nonlocal_conv4_5_out : (2, 1024, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.619527 52234 misc.py:142] nonlocal_conv4_5_out : (2, 1024, 16, 14, 14) => nonlocal_conv4_5_bn : (2, 1024, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.637033 52234 misc.py:142] res4_5_branch2c_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_5_sum : (2, 1024, 16, 14, 14) ------- (op: Sum) I0830 19:11:09.645759 52234 misc.py:142] nonlocal_conv4_5_bn : (2, 1024, 16, 14, 14) => nonlocal_conv4_5_sum : (2, 1024, 16, 14, 14) ------| I0830 19:11:09.661318 52234 misc.py:142] nonlocal_conv4_5_sum : (2, 1024, 16, 14, 14) => res5_0_branch2a : (2, 512, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.670754 52234 misc.py:142] res5_0_branch2a : (2, 512, 16, 14, 14) => res5_0_branch2a_bn : (2, 512, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.679487 52234 misc.py:142] res5_0_branch2a_bn : (2, 512, 16, 14, 14) => res5_0_branch2a_bn : (2, 512, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.687658 52234 misc.py:142] res5_0_branch2a_bn : (2, 512, 16, 14, 14) => res5_0_branch2b : (2, 512, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.695979 52234 misc.py:142] res5_0_branch2b : (2, 512, 16, 14, 14) => res5_0_branch2b_bn : (2, 512, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.704550 52234 misc.py:142] res5_0_branch2b_bn : (2, 512, 16, 14, 14) => res5_0_branch2b_bn : (2, 512, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.729733 52234 misc.py:142] res5_0_branch2b_bn : (2, 512, 16, 14, 14) => res5_0_branch2c : (2, 2048, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.770387 52234 misc.py:142] res5_0_branch2c : (2, 2048, 16, 14, 14) => res5_0_branch2c_bn : (2, 2048, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.801537 52234 misc.py:142] nonlocal_conv4_5_sum : (2, 1024, 16, 14, 14) => res5_0_branch1 : (2, 2048, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.842406 52234 misc.py:142] res5_0_branch1 : (2, 2048, 16, 14, 14) => res5_0_branch1_bn : (2, 2048, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.884166 52234 misc.py:142] res5_0_branch2c_bn : (2, 2048, 16, 14, 14) => res5_0_branch2c_bn : (2, 2048, 16, 14, 14) ------- (op: Sum) I0830 19:11:09.905071 52234 misc.py:142] res5_0_branch1_bn : (2, 2048, 16, 14, 14) => res5_0_branch2c_bn : (2, 2048, 16, 14, 14) ------| I0830 19:11:09.946340 52234 misc.py:142] res5_0_branch2c_bn : (2, 2048, 16, 14, 14) => res5_0_branch2c_bn : (2, 2048, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.971846 52234 misc.py:142] res5_0_branch2c_bn : (2, 2048, 16, 14, 14) => res5_1_branch2a : (2, 512, 16, 14, 14) ------- (op: Conv) I0830 19:11:09.981764 52234 misc.py:142] res5_1_branch2a : (2, 512, 16, 14, 14) => res5_1_branch2a_bn : (2, 512, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:09.990498 52234 misc.py:142] res5_1_branch2a_bn : (2, 512, 16, 14, 14) => res5_1_branch2a_bn : (2, 512, 16, 14, 14) ------- (op: Relu) I0830 19:11:09.998667 52234 misc.py:142] res5_1_branch2a_bn : (2, 512, 16, 14, 14) => res5_1_branch2b : (2, 512, 16, 14, 14) ------- (op: Conv) I0830 19:11:10.007133 52234 misc.py:142] res5_1_branch2b : (2, 512, 16, 14, 14) => res5_1_branch2b_bn : (2, 512, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:10.015702 52234 misc.py:142] res5_1_branch2b_bn : (2, 512, 16, 14, 14) => res5_1_branch2b_bn : (2, 512, 16, 14, 14) ------- (op: Relu) I0830 19:11:10.040975 52234 misc.py:142] res5_1_branch2b_bn : (2, 512, 16, 14, 14) => res5_1_branch2c : (2, 2048, 16, 14, 14) ------- (op: Conv) I0830 19:11:10.081971 52234 misc.py:142] res5_1_branch2c : (2, 2048, 16, 14, 14) => res5_1_branch2c_bn : (2, 2048, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:10.123790 52234 misc.py:142] res5_1_branch2c_bn : (2, 2048, 16, 14, 14) => res5_1_branch2c_bn : (2, 2048, 16, 14, 14) ------- (op: Sum) I0830 19:11:10.144726 52234 misc.py:142] res5_0_branch2c_bn : (2, 2048, 16, 14, 14) => res5_1_branch2c_bn : (2, 2048, 16, 14, 14) ------| I0830 19:11:10.186017 52234 misc.py:142] res5_1_branch2c_bn : (2, 2048, 16, 14, 14) => res5_1_branch2c_bn : (2, 2048, 16, 14, 14) ------- (op: Relu) I0830 19:11:10.211498 52234 misc.py:142] res5_1_branch2c_bn : (2, 2048, 16, 14, 14) => res5_2_branch2a : (2, 512, 16, 14, 14) ------- (op: Conv) I0830 19:11:10.221318 52234 misc.py:142] res5_2_branch2a : (2, 512, 16, 14, 14) => res5_2_branch2a_bn : (2, 512, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:10.229995 52234 misc.py:142] res5_2_branch2a_bn : (2, 512, 16, 14, 14) => res5_2_branch2a_bn : (2, 512, 16, 14, 14) ------- (op: Relu) I0830 19:11:10.238130 52234 misc.py:142] res5_2_branch2a_bn : (2, 512, 16, 14, 14) => res5_2_branch2b : (2, 512, 16, 14, 14) ------- (op: Conv) I0830 19:11:10.246541 52234 misc.py:142] res5_2_branch2b : (2, 512, 16, 14, 14) => res5_2_branch2b_bn : (2, 512, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:10.255095 52234 misc.py:142] res5_2_branch2b_bn : (2, 512, 16, 14, 14) => res5_2_branch2b_bn : (2, 512, 16, 14, 14) ------- (op: Relu) I0830 19:11:10.280448 52234 misc.py:142] res5_2_branch2b_bn : (2, 512, 16, 14, 14) => res5_2_branch2c : (2, 2048, 16, 14, 14) ------- (op: Conv) I0830 19:11:10.321563 52234 misc.py:142] res5_2_branch2c : (2, 2048, 16, 14, 14) => res5_2_branch2c_bn : (2, 2048, 16, 14, 14) ------- (op: AffineNd) I0830 19:11:10.363434 52234 misc.py:142] res5_2_branch2c_bn : (2, 2048, 16, 14, 14) => res5_2_branch2c_bn : (2, 2048, 16, 14, 14) ------- (op: Sum) I0830 19:11:10.384398 52234 misc.py:142] res5_1_branch2c_bn : (2, 2048, 16, 14, 14) => res5_2_branch2c_bn : (2, 2048, 16, 14, 14) ------| I0830 19:11:10.425740 52234 misc.py:142] res5_2_branch2c_bn : (2, 2048, 16, 14, 14) => res5_2_branch2c_bn : (2, 2048, 16, 14, 14) ------- (op: Relu) I0830 19:11:10.447900 52234 misc.py:142] res5_2_branch2c_bn : (2, 2048, 16, 14, 14) => blob_pooled : (2, 2048, 1, 14, 14) ------- (op: AveragePool) I0830 19:11:10.451148 52234 misc.py:142] blob_pooled : (2, 2048, 1, 14, 14) => blob_pooled_4d : (2, 2048, 14, 14) ------- (op: Squeeze) I0830 19:11:10.453512 52234 misc.py:142] blob_pooled_4d : (2, 2048, 14, 14) => roi_feat_3d : (6, 2048, 7, 7) ------- (op: RoIAlign) I0830 19:11:10.453953 52234 misc.py:142] proposals_train : (6, 5) => roi_feat_3d : (6, 2048, 7, 7) ------| I0830 19:11:10.455351 52234 misc.py:142] roi_feat_3d : (6, 2048, 7, 7) => roi_feat_1d : (6, 2048, 1, 1) ------- (op: MaxPool) I0830 19:11:10.455841 52234 misc.py:142] roi_feat_1d : (6, 2048, 1, 1) => box_pooled : (6, 2048, 1, 1, 1) ------- (op: Reshape) I0830 19:11:10.456351 52234 misc.py:142] box_pooled : (6, 2048, 1, 1, 1) => pool5 : (6, 2048, 1, 1, 1) ------- (op: Concat) I0830 19:11:10.456831 52234 misc.py:142] pool5 : (6, 2048, 1, 1, 1) => pool5_dropout : (6, 2048, 1, 1, 1) ------- (op: Dropout) I0830 19:11:10.457333 52234 misc.py:142] pool5_dropout : (6, 2048, 1, 1, 1) => pred : (6, 80) ------- (op: FC) I0830 19:11:10.458275 52234 misc.py:142] pred : (6, 80) => prob : (6, 80) ------- (op: Sigmoid) I0830 19:11:10.458874 52234 misc.py:142] pred : (6, 80) => loss : () ------- (op: SigmoidCrossEntropyLoss) I0830 19:11:10.459304 52234 misc.py:142] labels_train : (6, 80) => loss : () ------| I0830 19:11:10.515620 52234 misc.py:142] pred : (6, 80) => m13_shared : (2, 2048, 16, 14, 14) ------- (op: SigmoidCrossEntropyLossGradient) I0830 19:11:10.516119 52234 misc.py:142] labels_train : (6, 80) => m13_shared : (2, 2048, 16, 14, 14) ------| I0830 19:11:10.531191 52234 misc.py:142] __m4_shared : (2, 3, 32, 224, 224) => __m13_shared : (2, 2048, 16, 14, 14) ------| I0830 19:11:10.531517 52234 misc.py:147] End of model: CNN_1 Fri Aug 30 19:11:11 2019
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 390.30 Driver Version: 390.30 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla V100-PCIE... On | 00000000:04:00.0 Off | 0 | | N/A 36C P0 36W / 250W | 10491MiB / 16160MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla V100-PCIE... On | 00000000:05:00.0 Off | 0 | | N/A 37C P0 42W / 250W | 10405MiB / 16160MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 Tesla V100-PCIE... On | 00000000:08:00.0 Off | 0 | | N/A 37C P0 41W / 250W | 10399MiB / 16160MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 Tesla V100-PCIE... On | 00000000:09:00.0 Off | 0 | | N/A 38C P0 43W / 250W | 10409MiB / 16160MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 4 Tesla V100-PCIE... On | 00000000:85:00.0 Off | 0 | | N/A 37C P0 42W / 250W | 10397MiB / 16160MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 5 Tesla V100-PCIE... On | 00000000:86:00.0 Off | 0 | | N/A 36C P0 43W / 250W | 10403MiB / 16160MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 6 Tesla V100-PCIE... On | 00000000:89:00.0 Off | 0 | | N/A 37C P0 43W / 250W | 10401MiB / 16160MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 7 Tesla V100-PCIE... On | 00000000:8A:00.0 Off | 0 | | N/A 35C P0 41W / 250W | 10405MiB / 16160MiB | 0% Default | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 52234 C ...ei/anaconda3/envs/video-lfb/bin/python2 10480MiB | | 1 52234 C ...ei/anaconda3/envs/video-lfb/bin/python2 10394MiB | | 2 52234 C ...ei/anaconda3/envs/video-lfb/bin/python2 10388MiB | | 3 52234 C ...ei/anaconda3/envs/video-lfb/bin/python2 10398MiB | | 4 52234 C ...ei/anaconda3/envs/video-lfb/bin/python2 10386MiB | | 5 52234 C ...ei/anaconda3/envs/video-lfb/bin/python2 10392MiB | | 6 52234 C ...ei/anaconda3/envs/video-lfb/bin/python2 10390MiB | | 7 52234 C ...ei/anaconda3/envs/video-lfb/bin/python2 10394MiB | +-----------------------------------------------------------------------------+

[chaoyuaw]

One possibility is that it's I/O bound. This can happen, for example, when the frames are on a disk that's not fast enough. It might be also worth considering modifying number of threads and number of processes used in data loader to improve the I/O speed. https://github.com/facebookresearch/video-long-term-feature-banks/blob/master/lib/datasets/dataloader.py#L74

[Scofieldtangsiwei]

Thank you so much for your reply. I checked the speed of reading frames from the disk and it's about 0.03s per frame. Does it sound good? And do you have any suggestion about the number of workers and processes?

[chaoyuaw]

0.03s per frame seems slow. For a clip of 32 frame, it'll need ~1s, and for a batch of 16 clips, it takes 16 seconds per iteration.

A good number of workers/processes depends on the number of CPU cores you have, among other things. I recommend trying a few different values to see which one works the best.

[Scofieldtangsiwei]

Hi, thanks for your reply! I tried to increase the number of workers and processes, but unfortunately it didn't help. I tried to print out the time the program used for fetching a clip in lib/dataset/data_input.py.

def retry_load_images(image_paths, retry): time_start = time.time() for image_path in image_paths: imgs = [cv2.imread(image_path) for image_path in image_paths] time_end = time.time() print("reading time is {}, start time is {}, end time is {}".format(time_end - time_start, time_start, time_end)) print("number of paths is {}".format(len(image_paths))) print("\n")

And I found the fetchings were not parallel. There were 50 second interval between two fetchings. the following is the log output:

reading time is 1.37589097023, start time is 1567418960.04, end time is 1567418961.41 number of paths is 32 reading time is 1.21721506119, start time is 1567419012.92, end time is 1567419014.13 number of paths is 32 reading time is 1.53639793396, start time is 1567419069.67, end time is 1567419071.2 number of paths is 32 reading time is 6.43953704834, start time is 1567419134.82, end time is 1567419141.26 number of paths is 32 reading time is 1.85930204391, start time is 1567419189.83, end time is 1567419191.69 number of paths is 32 reading time is 2.36378502846, start time is 1567419246.6, end time is 1567419248.97 number of paths is 32 reading time is 1.83352994919, start time is 1567419316.73, end time is 1567419318.56 number of paths is 32 reading time is 1.69703602791, start time is 1567419374.78, end time is 1567419376.47 number of paths is 32 reading time is 3.92334008217, start time is 1567419424.55, end time is 1567419428.48 number of paths is 32 reading time is 1.39269399643, start time is 1567419482.3, end time is 1567419483.7 number of paths is 32 reading time is 0.661792993546, start time is 1567419545.74, end time is 1567419546.4 number of paths is 32 reading time is 1.7791519165, start time is 1567419599.76, end time is 1567419601.53 number of paths is 32 reading time is 5.34246492386, start time is 1567419652.86, end time is 1567419658.21 number of paths is 32 reading time is 1.22558903694, start time is 1567419718.47, end time is 1567419719.7 number of paths is 32 reading time is 6.62388300896, start time is 1567419776.29, end time is 1567419782.92 number of paths is 32

Does this look good to you? Thanks for your reply again!

[chaoyuaw]

Hi @Scofieldtangsiwei , thanks for providing more information!

This still looks slow to me. It's slower than GPU processing speed. (In our experiments, 8 GPUs process 16 clips in < 1 sec, and here your loader takes 1 sec to load one clip. ) What kind of disk are you using (e.g. flash? )?

[Scofieldtangsiwei]

Hi @chaoyuaw , thanks for your help! I have located the problem. It is caused by the IO limitation on the server. The image reading time on my server is twice as much as my local machine. Honestly, I don't know what kind of disk the server is using. What kind of CPU and disk did you use during training?

[chaoyuaw]

Hi @Scofieldtangsiwei , I used a flash disk in this project.

If it's IO bound in your case, I don't know how to mitigate this issue, unless you have a faster disk available.

[Scofieldtangsiwei]

Thank you so much for you reply! I will check if I can use a faster disk.

[banshee1]

Thank you so much for you reply! I will check if I can use a faster disk.

Hi, have you solved this problem?

[Scofieldtangsiwei]

@banshee1 Unfortunately, no. I can't find a good way to solve this problem except for changing disks.