/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.

zcl912 commented 3 years ago

hello, when i train the model on 4 GPU, i met the following error, if train it on 1 gpu, the error disappear:

[04/08 14:39:50 c2.utils.dump.events]: eta: 4:24:06 iter: 6960/22500 total_loss: 0.748 loss_cls: 0.333 loss_box_reg: 0.420 time: 1.0219 data_time: 0.6507 lr: 0.010000 max_mem: 5233M /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [121,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. ERROR [04/08 14:39:54 c2.engine.base_runner]: Exception during training: Traceback (most recent call last): File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 84, in train self.run_step() File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 185, in run_step loss_dict = self.model(data) File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(*input, kwargs) File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward output = self.module(*inputs[0], *kwargs[0]) File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(input, kwargs) File "../yolof_base/yolof.py", line 135, in forward pred_logits, pred_anchor_deltas) File "../yolof_base/yolof.py", line 216, in losses pred_class_logits[valid_idxs], RuntimeError: copy_if failed to synchronize: device-side assert triggered [04/08 14:39:54 c2.engine.hooks]: Overall training speed: 6961 iterations in 1:58:34 (1.0221 s / it) [04/08 14:39:54 c2.engine.hooks]: Total training time: 2:04:42 (0:06:08 on hooks) terminate called after throwing an instance of 'c10::Error' what(): CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:764)

chensnathan commented 3 years ago

Could you provide the full log file?

zcl912 commented 3 years ago

Could you provide the full log file?

yeah, the full log is as follw:

[04/08 19:21:13 c2.utils.dump.events]: eta: 5:54:29 iter: 1580/22500 total_loss: 0.915 loss_cls: 0.425 loss_box_reg: 0.483 time: 0.9901 data_time: 0.6297 lr: 0.026332 max_mem: 5223M /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [20,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. terminate called after throwing an instance of 'c10::Error' what(): CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:764) frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f1a20023193 in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libc10.so) frame #1: + 0x17f66 (0x7f1a20260f66 in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libc10_cuda.so) frame #2: + 0x19cbd (0x7f1a20262cbd in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libc10_cuda.so) frame #3: c10::TensorImpl::release_resources() + 0x4d (0x7f1a2001363d in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libc10.so) frame #4: + 0x67a902 (0x7f1a69402902 in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #5: + 0x67a9a6 (0x7f1a694029a6 in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libtorch_python.so) frame #6: + 0x19f636 (0x55ee290af636 in /home/env/python3.6env/bin/python3) frame #7: + 0xeef28 (0x55ee28ffef28 in /home/env/python3.6env/bin/python3) frame #8: + 0xeeb1f (0x55ee28ffeb1f in /home/env/python3.6env/bin/python3) frame #9: + 0xef0da (0x55ee28fff0da in /home/env/python3.6env/bin/python3) frame #10: + 0xef0da (0x55ee28fff0da in /home/env/python3.6env/bin/python3) frame #11: + 0xee917 (0x55ee28ffe917 in /home/env/python3.6env/bin/python3) frame #12: + 0xee7a7 (0x55ee28ffe7a7 in /home/env/python3.6env/bin/python3) frame #13: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3) frame #14: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3) frame #15: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3) frame #16: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3) frame #17: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3) frame #18: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3) frame #19: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3) frame #20: + 0xe1047 (0x55ee28ff1047 in /home/env/python3.6env/bin/python3) frame #21: + 0xe10b3 (0x55ee28ff10b3 in /home/env/python3.6env/bin/python3) frame #22: + 0xe1036 (0x55ee28ff1036 in /home/env/python3.6env/bin/python3) frame #23: + 0x1c8393 (0x55ee290d8393 in /home/env/python3.6env/bin/python3) frame #24: _PyEval_EvalFrameDefault + 0x4378 (0x55ee290d4d38 in /home/env/python3.6env/bin/python3) frame #25: + 0x19870b (0x55ee290a870b in /home/env/python3.6env/bin/python3) frame #26: + 0x19e755 (0x55ee290ae755 in /home/env/python3.6env/bin/python3) frame #27: _PyEval_EvalFrameDefault + 0x2fa (0x55ee290d0cba in /home/env/python3.6env/bin/python3) frame #28: + 0x19870b (0x55ee290a870b in /home/env/python3.6env/bin/python3) frame #29: + 0x19e755 (0x55ee290ae755 in /home/env/python3.6env/bin/python3) frame #30: _PyEval_EvalFrameDefault + 0x2fa (0x55ee290d0cba in /home/env/python3.6env/bin/python3) frame #31: + 0x197a94 (0x55ee290a7a94 in /home/env/python3.6env/bin/python3) frame #32: + 0x198941 (0x55ee290a8941 in /home/env/python3.6env/bin/python3) frame #33: + 0x19e755 (0x55ee290ae755 in /home/env/python3.6env/bin/python3) frame #34: _PyEval_EvalFrameDefault + 0x10ba (0x55ee290d1a7a in /home/env/python3.6env/bin/python3) frame #35: PyEval_EvalCodeEx + 0x329 (0x55ee290a9459 in /home/env/python3.6env/bin/python3) frame #36: PyEval_EvalCode + 0x1c (0x55ee290aa1ec in /home/env/python3.6env/bin/python3) frame #37: + 0x2149a4 (0x55ee291249a4 in /home/env/python3.6env/bin/python3) frame #38: PyRun_StringFlags + 0x7d (0x55ee29124a3d in /home/env/python3.6env/bin/python3) frame #39: PyRun_SimpleStringFlags + 0x3f (0x55ee29124a9f in /home/env/python3.6env/bin/python3) frame #40: Py_Main + 0x43b (0x55ee2912889b in /home/env/python3.6env/bin/python3) frame #41: main + 0xee (0x55ee28ff04be in /home/env/python3.6env/bin/python3) frame #42: __libc_start_main + 0xf0 (0x7f1a735fa840 in /lib/x86_64-linux-gnu/libc.so.6) frame #43: + 0x1c7773 (0x55ee290d7773 in /home/env/python3.6env/bin/python3)

len(valid_idxs):38000 pred_class_logits:torch.Size([38000, 80]) len(valid_idxs):42000 pred_class_logits:torch.Size([42000, 80]) len(valid_idxs):39000 pred_class_logits:torch.Size([39000, 80]) Traceback (most recent call last): File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/tools/train_net.py", line 114, in args=(args,), File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/launch.py", line 53, in launch daemon=False, File "/home/env/python3.6env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn while not spawn_context.join(): File "/home/env/python3.6env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception:

-- Process 2 terminated with the following error: Traceback (most recent call last): File "/home/env/python3.6env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, args) File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/launch.py", line 88, in _distributed_worker main_func(args) File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/tools/train_net.py", line 100, in main runner.train() File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/runner.py", line 270, in train super().train(self.start_iter, self.start_epoch, self.max_iter) File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 84, in train self.run_step() File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 185, in run_step loss_dict = self.model(data) File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(*input, kwargs) File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward output = self.module(*inputs[0], *kwargs[0]) File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(input, kwargs) File "../yolof_base/yolof.py", line 135, in forward pred_logits, pred_anchor_deltas) File "../yolof_base/yolof.py", line 218, in losses pred_class_logits[valid_idxs], RuntimeError: copy_if failed to synchronize: device-side assert triggered

chensnathan commented 3 years ago

Which model are you training with? and what command do you use?

It will be more clear if you post your training log file in the log directory.

zcl912 commented 3 years ago

okay, thanks for your reply. i just follw the commond: cd playground/detection/coco/yolof/yolof.res50.C5.1x pods_train --num-gpus 4

the log is:

[04/08 19:32:57] cvpods INFO: Rank of current process: 0. World size: 4 [04/08 19:32:57] cvpods INFO: Environment info:

sys.platform linux Python 3.6.5	Anaconda, Inc.	(default, Apr 29 2018, 16:14:56) [GCC 7.2.0] numpy 1.18.4 cvpods 0.1 @/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods cvpods compiler GCC 5.4 cvpods CUDA compiler 10.0 cvpods arch flags sm_75 cvpods_ENV_MODULE PyTorch 1.4.0+cu100 @/home/env/python3.6env/lib/python3.6/site-packages/torch PyTorch debug build False CUDA available True GPU 0,1,2,3 GeForce RTX 2080 Ti CUDA_HOME /usr/local/cuda-10.0 NVCC Cuda compilation tools, release 10.0, V10.0.130 Pillow 8.0.1 torchvision 0.5.0+cu100 @/home/env/python3.6env/lib/python3.6/site-packages/torchvision torchvision arch flags sm_35, sm_50, sm_60, sm_70, sm_75 cv2 4.5.1

PyTorch built with:

GCC 7.3
Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CUDA Runtime 10.0
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
CuDNN 7.6.3
Magma 2.5.1
Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

[04/08 19:32:57] cvpods INFO: Command line arguments: Namespace(dist_url='tcp://127.0.0.1:50155', eval_only=False, machine_rank=0, num_gpus=4, num_machines=1, opts=[], resume=False) [04/08 19:32:57] cvpods INFO: Running with full config: ╒═════════════════╤═══════════════════════════════════════════════════════════════════════════╕ │ config params │ values │ ╞═════════════════╪═══════════════════════════════════════════════════════════════════════════╡ │ MODEL │ {'ANCHOR_GENERATOR': {'ANGLES': [[-90, 0, 90]], │ │ │ 'ASPECT_RATIOS': [[1.0]], │ │ │ 'OFFSET': 0.0, │ │ │ 'SIZES': [[32, 64, 128, 256, 512]]}, │ │ │ 'AS_PRETRAIN': False, │ │ │ 'BACKBONE': {'FREEZE_AT': 2}, │ │ │ 'DEVICE': 'cuda', │ │ │ 'FPN': {'BLOCK_IN_FEATURES': 'p5', │ │ │ 'FUSE_TYPE': 'sum', │ │ │ 'IN_FEATURES': [], │ │ │ 'NORM': '', │ │ │ 'OUT_CHANNELS': 256}, │ │ │ 'KEYPOINT_ON': False, │ │ │ 'LOAD_PROPOSALS': False, │ │ │ 'MASK_ON': False, │ │ │ 'NMS_TYPE': 'normal', │ │ │ 'PIXEL_MEAN': [103.53, 116.28, 123.675], │ │ │ 'PIXEL_STD': [1.0, 1.0, 1.0], │ │ │ 'RESNETS': {'ACTIVATION': {'INPLACE': True, 'NAME': 'ReLU'}, │ │ │ 'DEEP_STEM': False, │ │ │ 'DEPTH': 50, │ │ │ 'NORM': 'FrozenBN', │ │ │ 'NUM_CLASSES': None, │ │ │ 'NUM_GROUPS': 1, │ │ │ 'OUT_FEATURES': ['res5'], │ │ │ 'RES2_OUT_CHANNELS': 256, │ │ │ 'RES5_DILATION': 1, │ │ │ 'STEM_OUT_CHANNELS': 64, │ │ │ 'STRIDE_IN_1X1': True, │ │ │ 'WIDTH_PER_GROUP': 64, │ │ │ 'ZERO_INIT_RESIDUAL': False}, │ │ │ 'WEIGHTS': 'detectron2://ImageNetPretrained/MSRA/R-50.pkl', │ │ │ 'YOLOF': {'ADD_CTR_CLAMP': True, │ │ │ 'BBOX_REG_WEIGHTS': [1.0, 1.0, 1.0, 1.0], │ │ │ 'CTR_CLAMP': 32, │ │ │ 'DECODER': {'ACTIVATION': 'ReLU', │ │ │ 'CLS_NUM_CONVS': 2, │ │ │ 'IN_CHANNELS': 512, │ │ │ 'NORM': 'BN', │ │ │ 'NUM_ANCHORS': 5, │ │ │ 'NUM_CLASSES': 80, │ │ │ 'PRIOR_PROB': 0.01, │ │ │ 'REG_NUM_CONVS': 4}, │ │ │ 'ENCODER': {'ACTIVATION': 'ReLU', │ │ │ 'BLOCK_DILATIONS': [2, 4, 6, 8], │ │ │ 'BLOCK_MID_CHANNELS': 128, │ │ │ 'IN_FEATURES': ['res5'], │ │ │ 'NORM': 'BN', │ │ │ 'NUM_CHANNELS': 512, │ │ │ 'NUM_RESIDUAL_BLOCKS': 4}, │ │ │ 'FOCAL_LOSS_ALPHA': 0.25, │ │ │ 'FOCAL_LOSS_GAMMA': 2.0, │ │ │ 'MATCHER_TOPK': 4, │ │ │ 'NEG_IGNORE_THRESHOLD': 0.7, │ │ │ 'NMS_THRESH_TEST': 0.6, │ │ │ 'POS_IGNORE_THRESHOLD': 0.15, │ │ │ 'SCORE_THRESH_TEST': 0.05, │ │ │ 'TOPK_CANDIDATES_TEST': 1000}} │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ INPUT │ {'AUG': {'TEST_PIPELINES': [('ResizeShortestEdge', │ │ │ {'max_size': 1333, │ │ │ 'sample_style': 'choice', │ │ │ 'short_edge_length': 800})], │ │ │ 'TRAIN_PIPELINES': [('ResizeShortestEdge', │ │ │ {'max_size': 1333, │ │ │ 'sample_style': 'choice', │ │ │ 'short_edge_length': (800,)}), │ │ │ ('RandomFlip', {}), │ │ │ ('RandomShift', {'max_shifts': 32})]}, │ │ │ 'FORMAT': 'BGR', │ │ │ 'MASK_FORMAT': 'polygon'} │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ DATASETS │ {'CUSTOM_TYPE': ['ConcatDataset', {}], │ │ │ 'PRECOMPUTED_PROPOSAL_TOPK_TEST': 1000, │ │ │ 'PRECOMPUTED_PROPOSAL_TOPK_TRAIN': 2000, │ │ │ 'PROPOSAL_FILES_TEST': [], │ │ │ 'PROPOSAL_FILES_TRAIN': [], │ │ │ 'TEST': ['coco_2017_val'], │ │ │ 'TRAIN': ['coco_2017_train']} │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ DATALOADER │ {'ASPECT_RATIO_GROUPING': True, │ │ │ 'FILTER_EMPTY_ANNOTATIONS': True, │ │ │ 'NUM_WORKERS': 0, │ │ │ 'REPEAT_THRESHOLD': 0.0, │ │ │ 'SAMPLER_TRAIN': 'DistributedGroupSampler'} │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ SOLVER │ {'BATCH_SUBDIVISIONS': 1, │ │ │ 'CHECKPOINT_PERIOD': 5000, │ │ │ 'CLIP_GRADIENTS': {'CLIP_TYPE': 'value', │ │ │ 'CLIP_VALUE': 1.0, │ │ │ 'ENABLED': False, │ │ │ 'NORM_TYPE': 2.0}, │ │ │ 'IMS_PER_BATCH': 32, │ │ │ 'IMS_PER_DEVICE': 8, │ │ │ 'LR_SCHEDULER': {'GAMMA': 0.1, │ │ │ 'MAX_EPOCH': None, │ │ │ 'MAX_ITER': 22500, │ │ │ 'NAME': 'WarmupMultiStepLR', │ │ │ 'STEPS': [15000, 20000], │ │ │ 'WARMUP_FACTOR': 0.00066667, │ │ │ 'WARMUP_ITERS': 3000, │ │ │ 'WARMUP_METHOD': 'linear'}, │ │ │ 'OPTIMIZER': {'BACKBONE_LR_FACTOR': 0.334, │ │ │ 'BASE_LR': 0.0005, │ │ │ 'BIAS_LR_FACTOR': 1.0, │ │ │ 'MOMENTUM': 0.9, │ │ │ 'NAME': 'D2SGD', │ │ │ 'WEIGHT_DECAY': 0.0001, │ │ │ 'WEIGHT_DECAY_NORM': 0.0}} │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ TEST │ {'AUG': {'ENABLED': False, │ │ │ 'EXTRA_SIZES': [], │ │ │ 'FLIP': True, │ │ │ 'MAX_SIZE': 4000, │ │ │ 'MIN_SIZES': [400, 500, 600, 700, 800, 900, 1000, 1100, 1200], │ │ │ 'SCALE_FILTER': False, │ │ │ 'SCALE_RANGES': []}, │ │ │ 'DETECTIONS_PER_IMAGE': 100, │ │ │ 'EVAL_PERIOD': 0, │ │ │ 'EXPECTED_RESULTS': [], │ │ │ 'KEYPOINT_OKS_SIGMAS': [], │ │ │ 'PRECISE_BN': {'ENABLED': False, 'NUM_ITER': 200}} │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ TRAINER │ {'FP16': {'ENABLED': False, 'OPTS': {'OPT_LEVEL': 'O1'}, 'TYPE': 'APEX'}, │ │ │ 'NAME': 'DefaultRunner', │ │ │ 'WINDOW_SIZE': 20} │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ OUTPUT_DIR │ './output' │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ SEED │ -1 │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ CUDNN_BENCHMARK │ False │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ VIS_PERIOD │ 0 │ ├─────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ GLOBAL │ {'DUMP_TEST': False, 'DUMP_TRAIN': True, 'HACK': 1.0} │ ╘═════════════════╧═══════════════════════════════════════════════════════════════════════════╛ [04/08 19:32:57] cvpods INFO: different config with base class: ╒═════════════════╤═════════════════════════════════════════════════════════════════════╕ │ config params │ values │ ╞═════════════════╪═════════════════════════════════════════════════════════════════════╡ │ MODEL │ {'ANCHOR_GENERATOR': {'ASPECT_RATIOS': [[1.0]]}, │ │ │ 'RESNETS': {'DEPTH': 50, 'OUT_FEATURES': ['res5']}, │ │ │ 'WEIGHTS': 'detectron2://ImageNetPretrained/MSRA/R-50.pkl', │ │ │ 'YOLOF': {'ADD_CTR_CLAMP': True, │ │ │ 'BBOX_REG_WEIGHTS': [1.0, 1.0, 1.0, 1.0], │ │ │ 'CTR_CLAMP': 32, │ │ │ 'DECODER': {'ACTIVATION': 'ReLU', │ │ │ 'CLS_NUM_CONVS': 2, │ │ │ 'IN_CHANNELS': 512, │ │ │ 'NORM': 'BN', │ │ │ 'NUM_ANCHORS': 5, │ │ │ 'NUM_CLASSES': 80, │ │ │ 'PRIOR_PROB': 0.01, │ │ │ 'REG_NUM_CONVS': 4}, │ │ │ 'ENCODER': {'ACTIVATION': 'ReLU', │ │ │ 'BLOCK_DILATIONS': [2, 4, 6, 8], │ │ │ 'BLOCK_MID_CHANNELS': 128, │ │ │ 'IN_FEATURES': ['res5'], │ │ │ 'NORM': 'BN', │ │ │ 'NUM_CHANNELS': 512, │ │ │ 'NUM_RESIDUAL_BLOCKS': 4}, │ │ │ 'FOCAL_LOSS_ALPHA': 0.25, │ │ │ 'FOCAL_LOSS_GAMMA': 2.0, │ │ │ 'MATCHER_TOPK': 4, │ │ │ 'NEG_IGNORE_THRESHOLD': 0.7, │ │ │ 'NMS_THRESH_TEST': 0.6, │ │ │ 'POS_IGNORE_THRESHOLD': 0.15, │ │ │ 'SCORE_THRESH_TEST': 0.05, │ │ │ 'TOPK_CANDIDATES_TEST': 1000}} │ ├─────────────────┼─────────────────────────────────────────────────────────────────────┤ │ INPUT │ {'AUG': {'TRAIN_PIPELINES': [('ResizeShortestEdge', │ │ │ {'max_size': 1333, │ │ │ 'sample_style': 'choice', │ │ │ 'short_edge_length': (800,)}), │ │ │ ('RandomFlip', {}), │ │ │ ('RandomShift', {'max_shifts': 32})]}} │ ├─────────────────┼─────────────────────────────────────────────────────────────────────┤ │ DATASETS │ {'TEST': ['coco_2017_val'], 'TRAIN': ['coco_2017_train']} │ ├─────────────────┼─────────────────────────────────────────────────────────────────────┤ │ DATALOADER │ {'NUM_WORKERS': 0} │ ├─────────────────┼─────────────────────────────────────────────────────────────────────┤ │ SOLVER │ {'IMS_PER_BATCH': 32, │ │ │ 'IMS_PER_DEVICE': 8, │ │ │ 'LR_SCHEDULER': {'MAX_ITER': 22500, │ │ │ 'STEPS': [15000, 20000], │ │ │ 'WARMUP_FACTOR': 0.00066667, │ │ │ 'WARMUP_ITERS': 3000}, │ │ │ 'OPTIMIZER': {'BACKBONE_LR_FACTOR': 0.334, 'BASE_LR': 0.0005}} │ ╘═════════════════╧═════════════════════════════════════════════════════════════════════╛ [04/08 19:32:57] c2.utils.env.env INFO: Using a generated random seed 57776492 [04/08 19:32:57] c2.data.build INFO: TransformGens used: [ResizeShortestEdge(short_edge_length=(800,), max_size=1333, sample_style='choice'), RandomFlip(), RandomShift(max_shifts=32)] in training [04/08 19:33:09] c2.data.datasets.coco INFO: Loading /media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/datasets/coco/annotations/instances_train2017.json takes 11.49 seconds. [04/08 19:33:10] c2.data.datasets.coco INFO: Loaded 118287 images in COCO format from /media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/datasets/coco/annotations/instances_train2017.json [04/08 19:33:16] c2.data.base_dataset INFO: Removed 1021 images with no usable annotations. 117266 images left. [04/08 19:33:18] c2.data.base_dataset INFO: Distribution of instances among all 80 categories: [36m	category	#instances	category	#instances	category	#instances
person	257253	bicycle	7056	car	43533
motorcycle	8654	airplane	5129	bus	6061
train	4570	truck	9970	boat	10576
traffic light	12842	fire hydrant	1865	stop sign	1983
parking meter	1283	bench	9820	bird	10542
cat	4766	dog	5500	horse	6567
sheep	9223	cow	8014	elephant	5484
bear	1294	zebra	5269	giraffe	5128
backpack	8714	umbrella	11265	handbag	12342
tie	6448	suitcase	6112	frisbee	2681
skis	6623	snowboard	2681	sports ball	6299
kite	8802	baseball bat	3273	baseball gl..	3747
skateboard	5536	surfboard	6095	tennis racket	4807
bottle	24070	wine glass	7839	cup	20574
fork	5474	knife	7760	spoon	6159
bowl	14323	banana	9195	apple	5776
sandwich	4356	orange	6302	broccoli	7261
carrot	7758	hot dog	2884	pizza	5807
donut	7005	cake	6296	chair	38073
couch	5779	potted plant	8631	bed	4192
dining table	15695	toilet	4149	tv	5803
laptop	4960	mouse	2261	remote	5700
keyboard	2854	cell phone	6422	microwave	1672
oven	3334	toaster	225	sink	5609
refrigerator	2634	book	24077	clock	6320
vase	6577	scissors	1464	teddy bear	4729
hair drier	198	toothbrush	1945
total	849949					[0m

[04/08 19:33:19] c2.data.build INFO: Using training sampler DistributedGroupSampler [04/08 19:33:19] cvpods INFO: Model: YOLOF( (backbone): ResNet( (stem): BasicStem( (conv1): Conv2d( 3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (activation): ReLU(inplace=True) (max_pool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False) ) (res2): Sequential( (0): BottleneckBlock( (shortcut): Conv2d( 64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (activation): ReLU(inplace=True) (conv1): Conv2d( 64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv2): Conv2d( 64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv3): Conv2d( 64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) ) (1): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv2): Conv2d( 64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv3): Conv2d( 64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) ) (2): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv2): Conv2d( 64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=64, eps=1e-05) ) (conv3): Conv2d( 64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) ) ) (res3): Sequential( (0): BottleneckBlock( (shortcut): Conv2d( 256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (activation): ReLU(inplace=True) (conv1): Conv2d( 256, 128, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv2): Conv2d( 128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv3): Conv2d( 128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) ) (1): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv2): Conv2d( 128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv3): Conv2d( 128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) ) (2): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv2): Conv2d( 128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv3): Conv2d( 128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) ) (3): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv2): Conv2d( 128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=128, eps=1e-05) ) (conv3): Conv2d( 128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) ) ) (res4): Sequential( (0): BottleneckBlock( (shortcut): Conv2d( 512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) (activation): ReLU(inplace=True) (conv1): Conv2d( 512, 256, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) (1): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) (2): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) (3): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) (4): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) (5): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv2): Conv2d( 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=256, eps=1e-05) ) (conv3): Conv2d( 256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05) ) ) ) (res5): Sequential( (0): BottleneckBlock( (shortcut): Conv2d( 1024, 2048, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05) ) (activation): ReLU(inplace=True) (conv1): Conv2d( 1024, 512, kernel_size=(1, 1), stride=(2, 2), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv2): Conv2d( 512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv3): Conv2d( 512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05) ) ) (1): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv2): Conv2d( 512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv3): Conv2d( 512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05) ) ) (2): BottleneckBlock( (activation): ReLU(inplace=True) (conv1): Conv2d( 2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv2): Conv2d( 512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=512, eps=1e-05) ) (conv3): Conv2d( 512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False (norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05) ) ) ) ) (encoder): DilatedEncoder( (lateral_conv): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1)) (lateral_norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (fpn_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (fpn_norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (dilated_encoder_blocks): Sequential( (0): Bottleneck( (conv1): Sequential( (0): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1)) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2)) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv3): Sequential( (0): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1)) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) ) (1): Bottleneck( (conv1): Sequential( (0): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1)) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4)) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv3): Sequential( (0): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1)) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) ) (2): Bottleneck( (conv1): Sequential( (0): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1)) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(6, 6), dilation=(6, 6)) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv3): Sequential( (0): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1)) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) ) (3): Bottleneck( (conv1): Sequential( (0): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1)) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(8, 8), dilation=(8, 8)) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv3): Sequential( (0): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1)) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) ) ) ) (decoder): Decoder( (cls_subnet): Sequential( (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU(inplace=True) ) (bbox_subnet): Sequential( (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU(inplace=True) (6): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (7): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (8): ReLU(inplace=True) (9): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (10): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (11): ReLU(inplace=True) ) (cls_score): Conv2d(512, 400, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bbox_pred): Conv2d(512, 20, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (object_pred): Conv2d(512, 5, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) ) (anchor_generator): DefaultAnchorGenerator( (cell_anchors): BufferList() ) (matcher): UniformMatcher() ) [04/08 19:33:21] c2.checkpoint.checkpoint INFO: Loading checkpoint from detectron2://ImageNetPretrained/MSRA/R-50.pkl [04/08 19:33:21] c2.utils.file.file_io INFO: URL https://dl.fbaipublicfiles.com/detectron2/ImageNetPretrained/MSRA/R-50.pkl cached in /home/.torch/cvpods_cache/detectron2/ImageNetPretrained/MSRA/R-50.pkl [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: Remapping C2 weights ...... [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv1.norm.bias loaded from res2_0_branch2a_bn_beta of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv1.norm.running_mean loaded from res2_0_branch2a_bn_running_mean of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv1.norm.running_var loaded from res2_0_branch2a_bn_running_var of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv1.norm.weight loaded from res2_0_branch2a_bn_gamma of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv1.weight loaded from res2_0_branch2a_w of shape (64, 64, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv2.norm.bias loaded from res2_0_branch2b_bn_beta of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv2.norm.running_mean loaded from res2_0_branch2b_bn_running_mean of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv2.norm.running_var loaded from res2_0_branch2b_bn_running_var of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv2.norm.weight loaded from res2_0_branch2b_bn_gamma of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv2.weight loaded from res2_0_branch2b_w of shape (64, 64, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv3.norm.bias loaded from res2_0_branch2c_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv3.norm.running_mean loaded from res2_0_branch2c_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv3.norm.running_var loaded from res2_0_branch2c_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv3.norm.weight loaded from res2_0_branch2c_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv3.weight loaded from res2_0_branch2c_w of shape (256, 64, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.shortcut.norm.bias loaded from res2_0_branch1_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.shortcut.norm.running_mean loaded from res2_0_branch1_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.shortcut.norm.running_var loaded from res2_0_branch1_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.shortcut.norm.weight loaded from res2_0_branch1_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.shortcut.weight loaded from res2_0_branch1_w of shape (256, 64, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv1.norm.bias loaded from res2_1_branch2a_bn_beta of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv1.norm.running_mean loaded from res2_1_branch2a_bn_running_mean of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv1.norm.running_var loaded from res2_1_branch2a_bn_running_var of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv1.norm.weight loaded from res2_1_branch2a_bn_gamma of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv1.weight loaded from res2_1_branch2a_w of shape (64, 256, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv2.norm.bias loaded from res2_1_branch2b_bn_beta of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv2.norm.running_mean loaded from res2_1_branch2b_bn_running_mean of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv2.norm.running_var loaded from res2_1_branch2b_bn_running_var of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv2.norm.weight loaded from res2_1_branch2b_bn_gamma of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv2.weight loaded from res2_1_branch2b_w of shape (64, 64, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv3.norm.bias loaded from res2_1_branch2c_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv3.norm.running_mean loaded from res2_1_branch2c_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv3.norm.running_var loaded from res2_1_branch2c_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv3.norm.weight loaded from res2_1_branch2c_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv3.weight loaded from res2_1_branch2c_w of shape (256, 64, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv1.norm.bias loaded from res2_2_branch2a_bn_beta of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv1.norm.running_mean loaded from res2_2_branch2a_bn_running_mean of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv1.norm.running_var loaded from res2_2_branch2a_bn_running_var of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv1.norm.weight loaded from res2_2_branch2a_bn_gamma of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv1.weight loaded from res2_2_branch2a_w of shape (64, 256, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv2.norm.bias loaded from res2_2_branch2b_bn_beta of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv2.norm.running_mean loaded from res2_2_branch2b_bn_running_mean of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv2.norm.running_var loaded from res2_2_branch2b_bn_running_var of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv2.norm.weight loaded from res2_2_branch2b_bn_gamma of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv2.weight loaded from res2_2_branch2b_w of shape (64, 64, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv3.norm.bias loaded from res2_2_branch2c_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv3.norm.running_mean loaded from res2_2_branch2c_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv3.norm.running_var loaded from res2_2_branch2c_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv3.norm.weight loaded from res2_2_branch2c_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv3.weight loaded from res2_2_branch2c_w of shape (256, 64, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv1.norm.bias loaded from res3_0_branch2a_bn_beta of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv1.norm.running_mean loaded from res3_0_branch2a_bn_running_mean of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv1.norm.running_var loaded from res3_0_branch2a_bn_running_var of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv1.norm.weight loaded from res3_0_branch2a_bn_gamma of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv1.weight loaded from res3_0_branch2a_w of shape (128, 256, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv2.norm.bias loaded from res3_0_branch2b_bn_beta of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv2.norm.running_mean loaded from res3_0_branch2b_bn_running_mean of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv2.norm.running_var loaded from res3_0_branch2b_bn_running_var of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv2.norm.weight loaded from res3_0_branch2b_bn_gamma of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv2.weight loaded from res3_0_branch2b_w of shape (128, 128, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv3.norm.bias loaded from res3_0_branch2c_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv3.norm.running_mean loaded from res3_0_branch2c_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv3.norm.running_var loaded from res3_0_branch2c_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv3.norm.weight loaded from res3_0_branch2c_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv3.weight loaded from res3_0_branch2c_w of shape (512, 128, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.shortcut.norm.bias loaded from res3_0_branch1_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.shortcut.norm.running_mean loaded from res3_0_branch1_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.shortcut.norm.running_var loaded from res3_0_branch1_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.shortcut.norm.weight loaded from res3_0_branch1_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.shortcut.weight loaded from res3_0_branch1_w of shape (512, 256, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv1.norm.bias loaded from res3_1_branch2a_bn_beta of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv1.norm.running_mean loaded from res3_1_branch2a_bn_running_mean of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv1.norm.running_var loaded from res3_1_branch2a_bn_running_var of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv1.norm.weight loaded from res3_1_branch2a_bn_gamma of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv1.weight loaded from res3_1_branch2a_w of shape (128, 512, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv2.norm.bias loaded from res3_1_branch2b_bn_beta of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv2.norm.running_mean loaded from res3_1_branch2b_bn_running_mean of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv2.norm.running_var loaded from res3_1_branch2b_bn_running_var of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv2.norm.weight loaded from res3_1_branch2b_bn_gamma of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv2.weight loaded from res3_1_branch2b_w of shape (128, 128, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv3.norm.bias loaded from res3_1_branch2c_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv3.norm.running_mean loaded from res3_1_branch2c_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv3.norm.running_var loaded from res3_1_branch2c_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv3.norm.weight loaded from res3_1_branch2c_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv3.weight loaded from res3_1_branch2c_w of shape (512, 128, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv1.norm.bias loaded from res3_2_branch2a_bn_beta of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv1.norm.running_mean loaded from res3_2_branch2a_bn_running_mean of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv1.norm.running_var loaded from res3_2_branch2a_bn_running_var of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv1.norm.weight loaded from res3_2_branch2a_bn_gamma of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv1.weight loaded from res3_2_branch2a_w of shape (128, 512, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv2.norm.bias loaded from res3_2_branch2b_bn_beta of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv2.norm.running_mean loaded from res3_2_branch2b_bn_running_mean of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv2.norm.running_var loaded from res3_2_branch2b_bn_running_var of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv2.norm.weight loaded from res3_2_branch2b_bn_gamma of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv2.weight loaded from res3_2_branch2b_w of shape (128, 128, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv3.norm.bias loaded from res3_2_branch2c_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv3.norm.running_mean loaded from res3_2_branch2c_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv3.norm.running_var loaded from res3_2_branch2c_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv3.norm.weight loaded from res3_2_branch2c_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv3.weight loaded from res3_2_branch2c_w of shape (512, 128, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv1.norm.bias loaded from res3_3_branch2a_bn_beta of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv1.norm.running_mean loaded from res3_3_branch2a_bn_running_mean of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv1.norm.running_var loaded from res3_3_branch2a_bn_running_var of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv1.norm.weight loaded from res3_3_branch2a_bn_gamma of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv1.weight loaded from res3_3_branch2a_w of shape (128, 512, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv2.norm.bias loaded from res3_3_branch2b_bn_beta of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv2.norm.running_mean loaded from res3_3_branch2b_bn_running_mean of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv2.norm.running_var loaded from res3_3_branch2b_bn_running_var of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv2.norm.weight loaded from res3_3_branch2b_bn_gamma of shape (128,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv2.weight loaded from res3_3_branch2b_w of shape (128, 128, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv3.norm.bias loaded from res3_3_branch2c_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv3.norm.running_mean loaded from res3_3_branch2c_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv3.norm.running_var loaded from res3_3_branch2c_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv3.norm.weight loaded from res3_3_branch2c_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv3.weight loaded from res3_3_branch2c_w of shape (512, 128, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv1.norm.bias loaded from res4_0_branch2a_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv1.norm.running_mean loaded from res4_0_branch2a_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv1.norm.running_var loaded from res4_0_branch2a_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv1.norm.weight loaded from res4_0_branch2a_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv1.weight loaded from res4_0_branch2a_w of shape (256, 512, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv2.norm.bias loaded from res4_0_branch2b_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv2.norm.running_mean loaded from res4_0_branch2b_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv2.norm.running_var loaded from res4_0_branch2b_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv2.norm.weight loaded from res4_0_branch2b_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv2.weight loaded from res4_0_branch2b_w of shape (256, 256, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv3.norm.bias loaded from res4_0_branch2c_bn_beta of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv3.norm.running_mean loaded from res4_0_branch2c_bn_running_mean of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv3.norm.running_var loaded from res4_0_branch2c_bn_running_var of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv3.norm.weight loaded from res4_0_branch2c_bn_gamma of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv3.weight loaded from res4_0_branch2c_w of shape (1024, 256, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.shortcut.norm.bias loaded from res4_0_branch1_bn_beta of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.shortcut.norm.running_mean loaded from res4_0_branch1_bn_running_mean of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.shortcut.norm.running_var loaded from res4_0_branch1_bn_running_var of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.shortcut.norm.weight loaded from res4_0_branch1_bn_gamma of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.shortcut.weight loaded from res4_0_branch1_w of shape (1024, 512, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv1.norm.bias loaded from res4_1_branch2a_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv1.norm.running_mean loaded from res4_1_branch2a_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv1.norm.running_var loaded from res4_1_branch2a_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv1.norm.weight loaded from res4_1_branch2a_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv1.weight loaded from res4_1_branch2a_w of shape (256, 1024, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv2.norm.bias loaded from res4_1_branch2b_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv2.norm.running_mean loaded from res4_1_branch2b_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv2.norm.running_var loaded from res4_1_branch2b_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv2.norm.weight loaded from res4_1_branch2b_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv2.weight loaded from res4_1_branch2b_w of shape (256, 256, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv3.norm.bias loaded from res4_1_branch2c_bn_beta of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv3.norm.running_mean loaded from res4_1_branch2c_bn_running_mean of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv3.norm.running_var loaded from res4_1_branch2c_bn_running_var of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv3.norm.weight loaded from res4_1_branch2c_bn_gamma of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv3.weight loaded from res4_1_branch2c_w of shape (1024, 256, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv1.norm.bias loaded from res4_2_branch2a_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv1.norm.running_mean loaded from res4_2_branch2a_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv1.norm.running_var loaded from res4_2_branch2a_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv1.norm.weight loaded from res4_2_branch2a_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv1.weight loaded from res4_2_branch2a_w of shape (256, 1024, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv2.norm.bias loaded from res4_2_branch2b_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv2.norm.running_mean loaded from res4_2_branch2b_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv2.norm.running_var loaded from res4_2_branch2b_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv2.norm.weight loaded from res4_2_branch2b_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv2.weight loaded from res4_2_branch2b_w of shape (256, 256, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv3.norm.bias loaded from res4_2_branch2c_bn_beta of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv3.norm.running_mean loaded from res4_2_branch2c_bn_running_mean of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv3.norm.running_var loaded from res4_2_branch2c_bn_running_var of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv3.norm.weight loaded from res4_2_branch2c_bn_gamma of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv3.weight loaded from res4_2_branch2c_w of shape (1024, 256, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv1.norm.bias loaded from res4_3_branch2a_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv1.norm.running_mean loaded from res4_3_branch2a_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv1.norm.running_var loaded from res4_3_branch2a_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv1.norm.weight loaded from res4_3_branch2a_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv1.weight loaded from res4_3_branch2a_w of shape (256, 1024, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv2.norm.bias loaded from res4_3_branch2b_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv2.norm.running_mean loaded from res4_3_branch2b_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv2.norm.running_var loaded from res4_3_branch2b_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv2.norm.weight loaded from res4_3_branch2b_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv2.weight loaded from res4_3_branch2b_w of shape (256, 256, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv3.norm.bias loaded from res4_3_branch2c_bn_beta of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv3.norm.running_mean loaded from res4_3_branch2c_bn_running_mean of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv3.norm.running_var loaded from res4_3_branch2c_bn_running_var of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv3.norm.weight loaded from res4_3_branch2c_bn_gamma of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv3.weight loaded from res4_3_branch2c_w of shape (1024, 256, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv1.norm.bias loaded from res4_4_branch2a_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv1.norm.running_mean loaded from res4_4_branch2a_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv1.norm.running_var loaded from res4_4_branch2a_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv1.norm.weight loaded from res4_4_branch2a_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv1.weight loaded from res4_4_branch2a_w of shape (256, 1024, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv2.norm.bias loaded from res4_4_branch2b_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv2.norm.running_mean loaded from res4_4_branch2b_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv2.norm.running_var loaded from res4_4_branch2b_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv2.norm.weight loaded from res4_4_branch2b_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv2.weight loaded from res4_4_branch2b_w of shape (256, 256, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv3.norm.bias loaded from res4_4_branch2c_bn_beta of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv3.norm.running_mean loaded from res4_4_branch2c_bn_running_mean of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv3.norm.running_var loaded from res4_4_branch2c_bn_running_var of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv3.norm.weight loaded from res4_4_branch2c_bn_gamma of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv3.weight loaded from res4_4_branch2c_w of shape (1024, 256, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv1.norm.bias loaded from res4_5_branch2a_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv1.norm.running_mean loaded from res4_5_branch2a_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv1.norm.running_var loaded from res4_5_branch2a_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv1.norm.weight loaded from res4_5_branch2a_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv1.weight loaded from res4_5_branch2a_w of shape (256, 1024, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv2.norm.bias loaded from res4_5_branch2b_bn_beta of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv2.norm.running_mean loaded from res4_5_branch2b_bn_running_mean of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv2.norm.running_var loaded from res4_5_branch2b_bn_running_var of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv2.norm.weight loaded from res4_5_branch2b_bn_gamma of shape (256,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv2.weight loaded from res4_5_branch2b_w of shape (256, 256, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv3.norm.bias loaded from res4_5_branch2c_bn_beta of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv3.norm.running_mean loaded from res4_5_branch2c_bn_running_mean of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv3.norm.running_var loaded from res4_5_branch2c_bn_running_var of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv3.norm.weight loaded from res4_5_branch2c_bn_gamma of shape (1024,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv3.weight loaded from res4_5_branch2c_w of shape (1024, 256, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv1.norm.bias loaded from res5_0_branch2a_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv1.norm.running_mean loaded from res5_0_branch2a_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv1.norm.running_var loaded from res5_0_branch2a_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv1.norm.weight loaded from res5_0_branch2a_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv1.weight loaded from res5_0_branch2a_w of shape (512, 1024, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv2.norm.bias loaded from res5_0_branch2b_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv2.norm.running_mean loaded from res5_0_branch2b_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv2.norm.running_var loaded from res5_0_branch2b_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv2.norm.weight loaded from res5_0_branch2b_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv2.weight loaded from res5_0_branch2b_w of shape (512, 512, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv3.norm.bias loaded from res5_0_branch2c_bn_beta of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv3.norm.running_mean loaded from res5_0_branch2c_bn_running_mean of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv3.norm.running_var loaded from res5_0_branch2c_bn_running_var of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv3.norm.weight loaded from res5_0_branch2c_bn_gamma of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv3.weight loaded from res5_0_branch2c_w of shape (2048, 512, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.shortcut.norm.bias loaded from res5_0_branch1_bn_beta of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.shortcut.norm.running_mean loaded from res5_0_branch1_bn_running_mean of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.shortcut.norm.running_var loaded from res5_0_branch1_bn_running_var of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.shortcut.norm.weight loaded from res5_0_branch1_bn_gamma of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.shortcut.weight loaded from res5_0_branch1_w of shape (2048, 1024, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv1.norm.bias loaded from res5_1_branch2a_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv1.norm.running_mean loaded from res5_1_branch2a_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv1.norm.running_var loaded from res5_1_branch2a_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv1.norm.weight loaded from res5_1_branch2a_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv1.weight loaded from res5_1_branch2a_w of shape (512, 2048, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv2.norm.bias loaded from res5_1_branch2b_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv2.norm.running_mean loaded from res5_1_branch2b_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv2.norm.running_var loaded from res5_1_branch2b_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv2.norm.weight loaded from res5_1_branch2b_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv2.weight loaded from res5_1_branch2b_w of shape (512, 512, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv3.norm.bias loaded from res5_1_branch2c_bn_beta of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv3.norm.running_mean loaded from res5_1_branch2c_bn_running_mean of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv3.norm.running_var loaded from res5_1_branch2c_bn_running_var of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv3.norm.weight loaded from res5_1_branch2c_bn_gamma of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv3.weight loaded from res5_1_branch2c_w of shape (2048, 512, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv1.norm.bias loaded from res5_2_branch2a_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv1.norm.running_mean loaded from res5_2_branch2a_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv1.norm.running_var loaded from res5_2_branch2a_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv1.norm.weight loaded from res5_2_branch2a_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv1.weight loaded from res5_2_branch2a_w of shape (512, 2048, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv2.norm.bias loaded from res5_2_branch2b_bn_beta of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv2.norm.running_mean loaded from res5_2_branch2b_bn_running_mean of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv2.norm.running_var loaded from res5_2_branch2b_bn_running_var of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv2.norm.weight loaded from res5_2_branch2b_bn_gamma of shape (512,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv2.weight loaded from res5_2_branch2b_w of shape (512, 512, 3, 3) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv3.norm.bias loaded from res5_2_branch2c_bn_beta of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv3.norm.running_mean loaded from res5_2_branch2c_bn_running_mean of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv3.norm.running_var loaded from res5_2_branch2c_bn_running_var of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv3.norm.weight loaded from res5_2_branch2c_bn_gamma of shape (2048,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv3.weight loaded from res5_2_branch2c_w of shape (2048, 512, 1, 1) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.stem.conv1.norm.bias loaded from res_conv1_bn_beta of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.stem.conv1.norm.running_mean loaded from res_conv1_bn_running_mean of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.stem.conv1.norm.running_var loaded from res_conv1_bn_running_var of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.stem.conv1.norm.weight loaded from res_conv1_bn_gamma of shape (64,) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.stem.conv1.weight loaded from conv1_w of shape (64, 3, 7, 7) [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: Some model parameters are not in the checkpoint: [34manchor_generator.cell_anchors.0[0m [34mdecoder.bbox_pred.{bias, weight}[0m [34mdecoder.bbox_subnet.0.{bias, weight}[0m [34mdecoder.bbox_subnet.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mdecoder.bbox_subnet.10.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mdecoder.bbox_subnet.3.{bias, weight}[0m [34mdecoder.bbox_subnet.4.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mdecoder.bbox_subnet.6.{bias, weight}[0m [34mdecoder.bbox_subnet.7.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mdecoder.bbox_subnet.9.{bias, weight}[0m [34mdecoder.cls_score.{bias, weight}[0m [34mdecoder.cls_subnet.0.{bias, weight}[0m [34mdecoder.cls_subnet.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mdecoder.cls_subnet.3.{bias, weight}[0m [34mdecoder.cls_subnet.4.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mdecoder.object_pred.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.0.conv1.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.0.conv1.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.0.conv2.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.0.conv2.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.0.conv3.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.0.conv3.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.1.conv1.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.1.conv1.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.1.conv2.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.1.conv2.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.1.conv3.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.1.conv3.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.2.conv1.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.2.conv1.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.2.conv2.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.2.conv2.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.2.conv3.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.2.conv3.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.3.conv1.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.3.conv1.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.3.conv2.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.3.conv2.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.dilated_encoder_blocks.3.conv3.0.{bias, weight}[0m [34mencoder.dilated_encoder_blocks.3.conv3.1.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.fpn_conv.{bias, weight}[0m [34mencoder.fpn_norm.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mencoder.lateral_conv.{bias, weight}[0m [34mencoder.lateral_norm.{bias, num_batches_tracked, running_mean, running_var, weight}[0m [34mpixel_mean[0m [34mpixel_std[0m [04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: The checkpoint contains parameters not used by the model: [35mfc1000_b[0m [35mfc1000_w[0m [35mconv1_b[0m [04/08 19:33:21] cvpods INFO: Running with full config: ╒════════════════════════╤═══════════════════════════════════════════════════════════════════════════╕ │ config params │ values │ ╞════════════════════════╪═══════════════════════════════════════════════════════════════════════════╡ │ MODEL │ {'ANCHOR_GENERATOR': {'ANGLES': [[-90, 0, 90]], │ │ │ 'ASPECT_RATIOS': [[1.0]], │ │ │ 'OFFSET': 0.0, │ │ │ 'SIZES': [[32, 64, 128, 256, 512]]}, │ │ │ 'AS_PRETRAIN': False, │ │ │ 'BACKBONE': {'FREEZE_AT': 2}, │ │ │ 'DEVICE': 'cuda', │ │ │ 'FPN': {'BLOCK_IN_FEATURES': 'p5', │ │ │ 'FUSE_TYPE': 'sum', │ │ │ 'IN_FEATURES': [], │ │ │ 'NORM': '', │ │ │ 'OUT_CHANNELS': 256}, │ │ │ 'KEYPOINT_ON': False, │ │ │ 'LOAD_PROPOSALS': False, │ │ │ 'MASK_ON': False, │ │ │ 'NMS_TYPE': 'normal', │ │ │ 'PIXEL_MEAN': [103.53, 116.28, 123.675], │ │ │ 'PIXEL_STD': [1.0, 1.0, 1.0], │ │ │ 'RESNETS': {'ACTIVATION': {'INPLACE': True, 'NAME': 'ReLU'}, │ │ │ 'DEEP_STEM': False, │ │ │ 'DEPTH': 50, │ │ │ 'NORM': 'FrozenBN', │ │ │ 'NUM_CLASSES': None, │ │ │ 'NUM_GROUPS': 1, │ │ │ 'OUT_FEATURES': ['res5'], │ │ │ 'RES2_OUT_CHANNELS': 256, │ │ │ 'RES5_DILATION': 1, │ │ │ 'STEM_OUT_CHANNELS': 64, │ │ │ 'STRIDE_IN_1X1': True, │ │ │ 'WIDTH_PER_GROUP': 64, │ │ │ 'ZERO_INIT_RESIDUAL': False}, │ │ │ 'WEIGHTS': 'detectron2://ImageNetPretrained/MSRA/R-50.pkl', │ │ │ 'YOLOF': {'ADD_CTR_CLAMP': True, │ │ │ 'BBOX_REG_WEIGHTS': [1.0, 1.0, 1.0, 1.0], │ │ │ 'CTR_CLAMP': 32, │ │ │ 'DECODER': {'ACTIVATION': 'ReLU', │ │ │ 'CLS_NUM_CONVS': 2, │ │ │ 'IN_CHANNELS': 512, │ │ │ 'NORM': 'BN', │ │ │ 'NUM_ANCHORS': 5, │ │ │ 'NUM_CLASSES': 80, │ │ │ 'PRIOR_PROB': 0.01, │ │ │ 'REG_NUM_CONVS': 4}, │ │ │ 'ENCODER': {'ACTIVATION': 'ReLU', │ │ │ 'BLOCK_DILATIONS': [2, 4, 6, 8], │ │ │ 'BLOCK_MID_CHANNELS': 128, │ │ │ 'IN_FEATURES': ['res5'], │ │ │ 'NORM': 'BN', │ │ │ 'NUM_CHANNELS': 512, │ │ │ 'NUM_RESIDUAL_BLOCKS': 4}, │ │ │ 'FOCAL_LOSS_ALPHA': 0.25, │ │ │ 'FOCAL_LOSS_GAMMA': 2.0, │ │ │ 'MATCHER_TOPK': 4, │ │ │ 'NEG_IGNORE_THRESHOLD': 0.7, │ │ │ 'NMS_THRESH_TEST': 0.6, │ │ │ 'POS_IGNORE_THRESHOLD': 0.15, │ │ │ 'SCORE_THRESH_TEST': 0.05, │ │ │ 'TOPK_CANDIDATES_TEST': 1000}} │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ INPUT │ {'AUG': {'TEST_PIPELINES': [('ResizeShortestEdge', │ │ │ {'max_size': 1333, │ │ │ 'sample_style': 'choice', │ │ │ 'short_edge_length': 800})], │ │ │ 'TRAIN_PIPELINES': [('ResizeShortestEdge', │ │ │ {'max_size': 1333, │ │ │ 'sample_style': 'choice', │ │ │ 'short_edge_length': (800,)}), │ │ │ ('RandomFlip', {}), │ │ │ ('RandomShift', {'max_shifts': 32})]}, │ │ │ 'FORMAT': 'BGR', │ │ │ 'MASK_FORMAT': 'polygon'} │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ DATASETS │ {'CUSTOM_TYPE': ['ConcatDataset', {}], │ │ │ 'PRECOMPUTED_PROPOSAL_TOPK_TEST': 1000, │ │ │ 'PRECOMPUTED_PROPOSAL_TOPK_TRAIN': 2000, │ │ │ 'PROPOSAL_FILES_TEST': [], │ │ │ 'PROPOSAL_FILES_TRAIN': [], │ │ │ 'TEST': ['coco_2017_val'], │ │ │ 'TRAIN': ['coco_2017_train']} │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ DATALOADER │ {'ASPECT_RATIO_GROUPING': True, │ │ │ 'FILTER_EMPTY_ANNOTATIONS': True, │ │ │ 'NUM_WORKERS': 0, │ │ │ 'REPEAT_THRESHOLD': 0.0, │ │ │ 'SAMPLER_TRAIN': 'DistributedGroupSampler'} │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ SOLVER │ {'BATCH_SUBDIVISIONS': 1, │ │ │ 'CHECKPOINT_PERIOD': 5000, │ │ │ 'CLIP_GRADIENTS': {'CLIP_TYPE': 'value', │ │ │ 'CLIP_VALUE': 1.0, │ │ │ 'ENABLED': False, │ │ │ 'NORM_TYPE': 2.0}, │ │ │ 'IMS_PER_BATCH': 32, │ │ │ 'IMS_PER_DEVICE': 8, │ │ │ 'LR_SCHEDULER': {'EPOCH_ITERS': -1, │ │ │ 'GAMMA': 0.1, │ │ │ 'MAX_EPOCH': None, │ │ │ 'MAX_ITER': 22500, │ │ │ 'NAME': 'WarmupMultiStepLR', │ │ │ 'STEPS': [15000, 20000], │ │ │ 'WARMUP_FACTOR': 0.00066667, │ │ │ 'WARMUP_ITERS': 3000, │ │ │ 'WARMUP_METHOD': 'linear'}, │ │ │ 'OPTIMIZER': {'BACKBONE_LR_FACTOR': 0.334, │ │ │ 'BASE_LR': 0.0005, │ │ │ 'BIAS_LR_FACTOR': 1.0, │ │ │ 'MOMENTUM': 0.9, │ │ │ 'NAME': 'D2SGD', │ │ │ 'WEIGHT_DECAY': 0.0001, │ │ │ 'WEIGHT_DECAY_NORM': 0.0}} │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ TEST │ {'AUG': {'ENABLED': False, │ │ │ 'EXTRA_SIZES': [], │ │ │ 'FLIP': True, │ │ │ 'MAX_SIZE': 4000, │ │ │ 'MIN_SIZES': [400, 500, 600, 700, 800, 900, 1000, 1100, 1200], │ │ │ 'SCALE_FILTER': False, │ │ │ 'SCALE_RANGES': []}, │ │ │ 'DETECTIONS_PER_IMAGE': 100, │ │ │ 'EVAL_PERIOD': 0, │ │ │ 'EXPECTED_RESULTS': [], │ │ │ 'KEYPOINT_OKS_SIGMAS': [], │ │ │ 'PRECISE_BN': {'ENABLED': False, 'NUM_ITER': 200}} │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ TRAINER │ {'FP16': {'ENABLED': False, 'OPTS': {'OPT_LEVEL': 'O1'}, 'TYPE': 'APEX'}, │ │ │ 'NAME': 'DefaultRunner', │ │ │ 'WINDOW_SIZE': 20} │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ OUTPUT_DIR │ './output' │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ SEED │ 57776492 │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ CUDNN_BENCHMARK │ False │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ VIS_PERIOD │ 0 │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ GLOBAL │ {'DUMP_TEST': False, 'DUMP_TRAIN': True, 'HACK': 1.0} │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ build_backbone │ <function build_backbone at 0x7f128cfb3488> │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ build_anchor_generator │ <function build_anchor_generator at 0x7f128cfb3510> │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ build_encoder │ <function build_encoder at 0x7f128cfb3598> │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤ │ build_decoder │ <function build_decoder at 0x7f128cfb3b70> │ ╘════════════════════════╧═══════════════════════════════════════════════════════════════════════════╛ [04/08 19:33:21] cvpods INFO: different config with base class: ╒════════════════════════╤═════════════════════════════════════════════════════════════════════╕ │ config params │ values │ ╞════════════════════════╪═════════════════════════════════════════════════════════════════════╡ │ MODEL │ {'ANCHOR_GENERATOR': {'ASPECT_RATIOS': [[1.0]]}, │ │ │ 'RESNETS': {'DEPTH': 50, 'OUT_FEATURES': ['res5']}, │ │ │ 'WEIGHTS': 'detectron2://ImageNetPretrained/MSRA/R-50.pkl', │ │ │ 'YOLOF': {'ADD_CTR_CLAMP': True, │ │ │ 'BBOX_REG_WEIGHTS': [1.0, 1.0, 1.0, 1.0], │ │ │ 'CTR_CLAMP': 32, │ │ │ 'DECODER': {'ACTIVATION': 'ReLU', │ │ │ 'CLS_NUM_CONVS': 2, │ │ │ 'IN_CHANNELS': 512, │ │ │ 'NORM': 'BN', │ │ │ 'NUM_ANCHORS': 5, │ │ │ 'NUM_CLASSES': 80, │ │ │ 'PRIOR_PROB': 0.01, │ │ │ 'REG_NUM_CONVS': 4}, │ │ │ 'ENCODER': {'ACTIVATION': 'ReLU', │ │ │ 'BLOCK_DILATIONS': [2, 4, 6, 8], │ │ │ 'BLOCK_MID_CHANNELS': 128, │ │ │ 'IN_FEATURES': ['res5'], │ │ │ 'NORM': 'BN', │ │ │ 'NUM_CHANNELS': 512, │ │ │ 'NUM_RESIDUAL_BLOCKS': 4}, │ │ │ 'FOCAL_LOSS_ALPHA': 0.25, │ │ │ 'FOCAL_LOSS_GAMMA': 2.0, │ │ │ 'MATCHER_TOPK': 4, │ │ │ 'NEG_IGNORE_THRESHOLD': 0.7, │ │ │ 'NMS_THRESH_TEST': 0.6, │ │ │ 'POS_IGNORE_THRESHOLD': 0.15, │ │ │ 'SCORE_THRESH_TEST': 0.05, │ │ │ 'TOPK_CANDIDATES_TEST': 1000}} │ ├────────────────────────┼─────────────────────────────────────────────────────────────────────┤ │ INPUT │ {'AUG': {'TRAIN_PIPELINES': [('ResizeShortestEdge', │ │ │ {'max_size': 1333, │ │ │ 'sample_style': 'choice', │ │ │ 'short_edge_length': (800,)}), │ │ │ ('RandomFlip', {}), │ │ │ ('RandomShift', {'max_shifts': 32})]}} │ ├────────────────────────┼─────────────────────────────────────────────────────────────────────┤ │ DATASETS │ {'TEST': ['coco_2017_val'], 'TRAIN': ['coco_2017_train']} │ ├────────────────────────┼─────────────────────────────────────────────────────────────────────┤ │ DATALOADER │ {'NUM_WORKERS': 0} │ ├────────────────────────┼─────────────────────────────────────────────────────────────────────┤ │ SOLVER │ {'IMS_PER_BATCH': 32, │ │ │ 'IMS_PER_DEVICE': 8, │ │ │ 'LR_SCHEDULER': {'EPOCH_ITERS': -1, │ │ │ 'MAX_ITER': 22500, │ │ │ 'STEPS': [15000, 20000], │ │ │ 'WARMUP_FACTOR': 0.00066667, │ │ │ 'WARMUP_ITERS': 3000}, │ │ │ 'OPTIMIZER': {'BACKBONE_LR_FACTOR': 0.334, 'BASE_LR': 0.0005}} │ ├────────────────────────┼─────────────────────────────────────────────────────────────────────┤ │ SEED │ 57776492 │ ├────────────────────────┼─────────────────────────────────────────────────────────────────────┤ │ build_backbone │ <function build_backbone at 0x7f128cfb3488> │ ├────────────────────────┼─────────────────────────────────────────────────────────────────────┤ │ build_anchor_generator │ <function build_anchor_generator at 0x7f128cfb3510> │ ├────────────────────────┼─────────────────────────────────────────────────────────────────────┤ │ build_encoder │ <function build_encoder at 0x7f128cfb3598> │ ├────────────────────────┼─────────────────────────────────────────────────────────────────────┤ │ build_decoder │ <function build_decoder at 0x7f128cfb3b70> │ ╘════════════════════════╧═════════════════════════════════════════════════════════════════════╛ [04/08 19:33:21] c2.engine.runner INFO: Starting training from iteration 0 [04/08 19:33:23] c2.utils.dump.events INFO: eta: N/A iter: 1/22500 total_loss: 2.216 loss_cls: 1.290 loss_box_reg: 0.926 data_time: 1.1023 lr: 0.000000 max_mem: 4760M [04/08 19:33:41] c2.utils.dump.events INFO: eta: 5:20:30 iter: 20/22500 total_loss: 2.150 loss_cls: 1.287 loss_box_reg: 0.864 time: 0.8982 data_time: 0.5389 lr: 0.000003 max_mem: 5186M [04/08 19:34:00] c2.utils.dump.events INFO: eta: 5:37:23 iter: 40/22500 total_loss: 2.165 loss_cls: 1.284 loss_box_reg: 0.882 time: 0.9086 data_time: 0.5233 lr: 0.000007 max_mem: 5190M [04/08 19:34:20] c2.utils.dump.events INFO: eta: 5:38:19 iter: 60/22500 total_loss: 2.116 loss_cls: 1.271 loss_box_reg: 0.851 time: 0.9096 data_time: 0.5163 lr: 0.000010 max_mem: 5190M [04/08 19:34:39] c2.utils.dump.events INFO: eta: 5:36:14 iter: 80/22500 total_loss: 2.102 loss_cls: 1.262 loss_box_reg: 0.842 time: 0.9070 data_time: 0.4970 lr: 0.000013 max_mem: 5200M [04/08 19:34:58] c2.utils.dump.events INFO: eta: 5:38:46 iter: 100/22500 total_loss: 2.100 loss_cls: 1.250 loss_box_reg: 0.854 time: 0.9112 data_time: 0.5313 lr: 0.000017 max_mem: 5200M [04/08 19:35:18] c2.utils.dump.events INFO: eta: 5:39:41 iter: 120/22500 total_loss: 2.056 loss_cls: 1.230 loss_box_reg: 0.846 time: 0.9126 data_time: 0.5210 lr: 0.000020 max_mem: 5200M [04/08 19:35:37] c2.utils.dump.events INFO: eta: 5:40:35 iter: 140/22500 total_loss: 2.055 loss_cls: 1.211 loss_box_reg: 0.842 time: 0.9142 data_time: 0.5304 lr: 0.000023 max_mem: 5200M [04/08 19:35:56] c2.utils.dump.events INFO: eta: 5:39:39 iter: 160/22500 total_loss: 1.986 loss_cls: 1.193 loss_box_reg: 0.799 time: 0.9133 data_time: 0.5155 lr: 0.000027 max_mem: 5200M [04/08 19:36:16] c2.utils.dump.events INFO: eta: 5:40:21 iter: 180/22500 total_loss: 1.994 loss_cls: 1.167 loss_box_reg: 0.819 time: 0.9144 data_time: 0.5241 lr: 0.000030 max_mem: 5200M [04/08 19:36:35] c2.utils.dump.events INFO: eta: 5:39:42 iter: 200/22500 total_loss: 1.966 loss_cls: 1.152 loss_box_reg: 0.808 time: 0.9146 data_time: 0.5258 lr: 0.000033 max_mem: 5203M [04/08 19:36:54] c2.utils.dump.events INFO: eta: 5:38:09 iter: 220/22500 total_loss: 1.916 loss_cls: 1.127 loss_box_reg: 0.789 time: 0.9121 data_time: 0.5016 lr: 0.000037 max_mem: 5203M [04/08 19:37:13] c2.utils.dump.events INFO: eta: 5:38:45 iter: 240/22500 total_loss: 1.911 loss_cls: 1.114 loss_box_reg: 0.798 time: 0.9122 data_time: 0.5127 lr: 0.000040 max_mem: 5203M [04/08 19:37:33] c2.utils.dump.events INFO: eta: 5:37:12 iter: 260/22500 total_loss: 1.869 loss_cls: 1.097 loss_box_reg: 0.779 time: 0.9119 data_time: 0.5214 lr: 0.000043 max_mem: 5203M [04/08 19:37:52] c2.utils.dump.events INFO: eta: 5:36:26 iter: 280/22500 total_loss: 1.874 loss_cls: 1.068 loss_box_reg: 0.788 time: 0.9105 data_time: 0.4854 lr: 0.000047 max_mem: 5203M [04/08 19:38:11] c2.utils.dump.events INFO: eta: 5:36:16 iter: 300/22500 total_loss: 1.859 loss_cls: 1.069 loss_box_reg: 0.794 time: 0.9115 data_time: 0.5283 lr: 0.000050 max_mem: 5203M [04/08 19:38:26] c2.engine.base_runner ERROR: Exception during training: Traceback (most recent call last): File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 84, in train self.run_step() File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 185, in run_step loss_dict = self.model(data) File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(*input, kwargs) File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward output = self.module(*inputs[0], *kwargs[0]) File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(input, kwargs) File "../yolof_base/yolof.py", line 135, in forward pred_logits, pred_anchor_deltas) File "../yolof_base/yolof.py", line 218, in losses pred_class_logits[valid_idxs], RuntimeError: copy_if failed to synchronize: device-side assert triggered [04/08 19:38:26] c2.engine.hooks INFO: Overall training speed: 312 iterations in 0:04:45 (0.9148 s / it) [04/08 19:38:26] c2.engine.hooks INFO: Total training time: 0:05:01 (0:00:16 on hooks)

chensnathan commented 3 years ago

It seems that you modify the BASE_LR. But it's weird. We have not encountered this error before. Could you try the standard 8 GPUs setting and train YOLOF again?

zcl912 commented 3 years ago

It seems that you modify the BASE_LR. But it's weird. We have not encountered this error before. Could you try the standard 8 GPUs setting and train YOLOF again?

thx, i will try it later

zcl912 commented 3 years ago

hello, the error is about the unusual value as follw:

chensnathan commented 3 years ago

It seems that there is somewhere the code performances ZeroDivision. Does this error occurs definitely or randomly?

zcl912 commented 3 years ago

It seems that there is somewhere the code performances ZeroDivision. Does this error occurs definitely or randomly?

it occurs randomly

Vericoware commented 3 years ago

I met the same problem? Have you fixed it?

megvii-model / YOLOF

/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed. #20