fatal: Not a git repository (or any parent up to mount point

lihuikenny commented 2 years ago

自己本地调试没有问题，提交集群训练的时候，会报这样的错误，但是找了很久没有找到问题。是代码中有某个地方用到了git，导致我上传集群git的过程中因为网络问题导致的错误？ 1、预训练模型是离线加载的，没有问题 2、环境都是离线安装的。

想请教一下，代码中是否有哪个地方用到了git这个操作？谢谢

MendelXu commented 2 years ago

这个是哪部分的提示啊，能贴一下完整日志么

MendelXu commented 2 years ago

如果是mmdetection collect env的时候提示，应该不影响训练？

lihuikenny commented 2 years ago

非常感谢您的回复， 1、首先，经过反复check，应该不是这个原因， 2、可能的原因是无标签数据是我根据自己的数据做的，可能有问题。我是对照这image_info_unlabeled2017.json格式自己做的，生产类似这样的

{"license": 1, "file_name": "varea_20200409424869ra00008979.jpg", "coco_url": "/data/datasets/varea_datasets_2020_2021/20_21_all_images/varea_20200409424869ra00008979.jpg", "height": 720, "width": 1280, "date_captured": "2021-11-22 16:10:19", "id": 1}, {"license": 1, "file_name": "varea_20201101184223ra00016020.jpg", "coco_url": "/data/datasets/varea_datasets_2020_2021/20_21_all_images/varea_20201101184223ra00016020.jpg", "height": 1080, "width": 1920, "date_captured": "2021-11-22 16:10:19", "id": 2}, {"license": 1, "file_name": "varea_20201229574793ra00001576.jpg", "coco_url": "/data/datasets/varea_datasets_2020_2021/20_21_all_images/varea_20201229574793ra00001576.jpg", "height": 1080, "width": 1920, "date_captured": "2021-11-22 16:10:19", "id": 3},

目前有点怀疑这里有问题，因为看了一下image_info_unlabeled2017.json的file_name和id是有对应关系的，可能coco是根据id去取file_name，而我自己制作的数据集没有这种对应关系，导致读取不到图片【这里希望得到您的解答，谢谢】

至于您tools/dataset/semi_coco.sh中调用了，semi_coco.py，其实里面应该是两部分，第一部分是将Coco有标签的数据按比例划分一部分出来作为无标签数据，第二部分，就是将image_info_unlabeled2017.json标注文件添加了空的标注信息和类别【有标注中提取的类别】。因为我是直接想加入真实的无标签数据进行，所以我应该只关注第二部分的制作应该问题

还请指点。谢谢

MendelXu commented 2 years ago

这个id只要自己分配，每个file_name对应一个id就可以了，可以直接用file_name排序后的index.

lihuikenny commented 2 years ago

2021-11-25 10:42:55,021 - mmdet.ssod - INFO - [<StreamHandler (INFO)>, <FileHandler ./work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k/20211125_104254.log (INFO)>] 2021-11-25 10:42:55,021 - mmdet.ssod - INFO - Environment info:

sys.platform: linux Python: 3.6.12 |Anaconda, Inc.| (default, Sep 8 2020, 23:10:56) [GCC 7.3.0] CUDA available: True GPU 0,1,2,3: Tesla V100-SXM2-32GB CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 10.1, V10.1.243 GCC: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609 PyTorch: 1.7.0 PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 10.2
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75
CuDNN 7.6.5
Magma 2.5.2
Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.8.0 OpenCV: 4.2.0 MMCV: 1.3.14 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 10.2 MMDetection: 2.17.0+

2021-11-25 10:42:58,562 - mmdet.ssod - INFO - Distributed training: True 2021-11-25 10:43:02,297 - mmdet.ssod - INFO - Config: model = dict( type='SoftTeacher', model=dict( type='FasterRCNN', backbone=dict( type='ResNet', depth=50, num_stages=4, out_indices=(0, 1, 2, 3), frozen_stages=1, norm_cfg=dict(type='BN', requires_grad=False), norm_eval=True, style='caffe', init_cfg=dict( type='Pretrained', checkpoint='resnet50_msra-5891d200.pth')), neck=dict( type='FPN', in_channels=[256, 512, 1024, 2048], out_channels=256, num_outs=5), rpn_head=dict( type='RPNHead', in_channels=256, feat_channels=256, anchor_generator=dict( type='AnchorGenerator', scales=[8], ratios=[0.25, 0.5, 1.0, 2.0, 4.0], strides=[4, 8, 16, 32, 64]), bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[1.0, 1.0, 1.0, 1.0]), loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0)), roi_head=dict( type='StandardRoIHead', bbox_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict( type='RoIAlign', output_size=7, sampling_ratio=0), out_channels=256, featmap_strides=[4, 8, 16, 32]), bbox_head=dict( type='Shared2FCBBoxHead', in_channels=256, fc_out_channels=1024, roi_feat_size=7, num_classes=1, bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[0.1, 0.1, 0.2, 0.2]), reg_class_agnostic=False, loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0))), train_cfg=dict( rpn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.7, neg_iou_thr=0.3, min_pos_iou=0.3, match_low_quality=True, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=256, pos_fraction=0.5, neg_pos_ub=-1, add_gt_as_proposals=False), allowed_border=-1, pos_weight=-1, debug=False), rpn_proposal=dict( nms_pre=2000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.5, neg_iou_thr=0.5, min_pos_iou=0.5, match_low_quality=False, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=512, pos_fraction=0.25, neg_pos_ub=-1, add_gt_as_proposals=True), pos_weight=-1, debug=False)), test_cfg=dict( rpn=dict( nms_pre=1000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( score_thr=0.05, nms=dict(type='nms', iou_threshold=0.5), max_per_img=100))), train_cfg=dict( use_teacher_proposal=False, pseudo_label_initial_score_thr=0.5, rpn_pseudo_threshold=0.9, cls_pseudo_threshold=0.9, reg_pseudo_threshold=0.02, jitter_times=10, jitter_scale=0.06, min_pseduo_box_size=0, unsup_weight=2.0), test_cfg=dict(inference_on='student')) dataset_type = 'CocoDataset' data_root = './' img_norm_cfg = dict( mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False) train_pipeline = [ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='sup'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag')) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ] data = dict( samples_per_gpu=8, workers_per_gpu=8, train=dict( type='SemiDataset', sup=dict( type='CocoDataset', ann_file= './new_ditu_train_v20200922.json', img_prefix='./train_imgs', pipeline=[ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='sup'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag')) ]), unsup=dict( type='CocoDataset', ann_file= './instances_unlabeled2017.json', img_prefix= './images', pipeline=[ dict(type='LoadImageFromFile'), dict(type='PseudoSamples', with_bbox=True), dict( type='MultiBranch', unsup_student=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='ShuffledSequential', transforms=[ dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]), dict( type='OneOf', transforms=[{ 'type': 'RandTranslate', 'x': (-0.1, 0.1) }, { 'type': 'RandTranslate', 'y': (-0.1, 0.1) }, { 'type': 'RandRotate', 'angle': (-30, 30) }, [{ 'type': 'RandShear', 'x': (-30, 30) }, { 'type': 'RandShear', 'y': (-30, 30) }]]) ]), dict( type='RandErase', n_iterations=(1, 5), size=[0, 0.2], squared=True) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_student'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ], unsup_teacher=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_teacher'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ]) ], filter_empty_gt=False)), val=dict( type='CocoDataset', ann_file= './new_ditu_val_v20200920.json', img_prefix='./val_imgs', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ]), test=dict( type='CocoDataset', ann_file= './new_ditu_val_v20200920.json', img_prefix='./val_imgs', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ]), sampler=dict( train=dict( type='SemiBalanceSampler', sample_ratio=[1, 1], by_prob=False, epoch_length=14460))) evaluation = dict(interval=4000, metric='bbox', type='SubModulesDistEvalHook') optimizer = dict(type='SGD', lr=0.001, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(grad_clip=None) lr_config = dict( policy='step', warmup='linear', warmup_iters=500, warmup_ratio=0.001, step=[480000, 640000]) runner = dict(type='IterBasedRunner', max_iters=720000) checkpoint_config = dict(interval=4000, by_epoch=False, max_keep_ckpts=20) log_config = dict( interval=50, hooks=[dict(type='TextLoggerHook', by_epoch=False)]) custom_hooks = [ dict(type='NumClassCheckHook'), dict(type='WeightSummary'), dict(type='MeanTeacher', momentum=0.999, interval=1, warm_up=0) ] dist_params = dict(backend='nccl') log_level = 'INFO' load_from = None resume_from = None workflow = [('train', 1)] mmdet_base = '../../thirdparty/mmdetection-2.17.0/configs/base' strong_pipeline = [ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='ShuffledSequential', transforms=[ dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]), dict( type='OneOf', transforms=[{ 'type': 'RandTranslate', 'x': (-0.1, 0.1) }, { 'type': 'RandTranslate', 'y': (-0.1, 0.1) }, { 'type': 'RandRotate', 'angle': (-30, 30) }, [{ 'type': 'RandShear', 'x': (-30, 30) }, { 'type': 'RandShear', 'y': (-30, 30) }]]) ]), dict( type='RandErase', n_iterations=(1, 5), size=[0, 0.2], squared=True) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_student'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ] weak_pipeline = [ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_teacher'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ] unsup_pipeline = [ dict(type='LoadImageFromFile'), dict(type='PseudoSamples', with_bbox=True), dict( type='MultiBranch', unsup_student=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='ShuffledSequential', transforms=[ dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]), dict( type='OneOf', transforms=[{ 'type': 'RandTranslate', 'x': (-0.1, 0.1) }, { 'type': 'RandTranslate', 'y': (-0.1, 0.1) }, { 'type': 'RandRotate', 'angle': (-30, 30) }, [{ 'type': 'RandShear', 'x': (-30, 30) }, { 'type': 'RandShear', 'y': (-30, 30) }]]) ]), dict( type='RandErase', n_iterations=(1, 5), size=[0, 0.2], squared=True) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_student'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ], unsup_teacher=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_teacher'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ]) ] fp16 = None work_dir = './work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k' cfg_name = 'soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k' gpu_ids = range(0, 4)

/thirdparty/mmdetection-2.17.0/mmdet/core/anchor/builder.py:17: UserWarning: build_anchor_generator would be deprecated soon, please use build_prior_generator 'build_anchor_generator would be deprecated soon, please use ' /thirdparty/mmdetection-2.17.0/mmdet/core/anchor/builder.py:17: UserWarning: build_anchor_generator would be deprecated soon, please use build_prior_generator 'build_anchor_generator would be deprecated soon, please use ' /thirdparty/mmdetection-2.17.0/mmdet/core/anchor/builder.py:17: UserWarning: build_anchor_generator would be deprecated soon, please use build_prior_generator 'build_anchor_generator would be deprecated soon, please use ' /root/paddlejob/workspace/env_run/thirdparty/mmdetection-2.17.0/mmdet/core/anchor/builder.py:17: UserWarning: build_anchor_generator would be deprecated soon, please use build_prior_generator 'build_anchor_generator would be deprecated soon, please use ' 2021-11-25 10:43:03,229 - mmdet.ssod - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'resnet50_msra-5891d200.pth'} 2021-11-25 10:43:03,229 - mmcv - INFO - load model from: resnet50_msra-5891d200.pth 2021-11-25 10:43:03,229 - mmcv - INFO - Use load_from_local loader 2021-11-25 10:43:03,356 - mmcv - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1.bias

2021-11-25 10:43:03,388 - mmdet.ssod - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'} 2021-11-25 10:43:03,419 - mmdet.ssod - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01} 2021-11-25 10:43:03,428 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}] 2021-11-25 10:43:03,815 - mmdet.ssod - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'resnet50_msra-5891d200.pth'} 2021-11-25 10:43:03,815 - mmcv - INFO - load model from: resnet50_msra-5891d200.pth 2021-11-25 10:43:03,816 - mmcv - INFO - Use load_from_local loader 2021-11-25 10:43:03,905 - mmcv - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1.bias

2021-11-25 10:43:03,937 - mmdet.ssod - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'} 2021-11-25 10:43:03,968 - mmdet.ssod - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01} 2021-11-25 10:43:03,977 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}] loading annotations into memory... loading annotations into memory... Done (t=0.06s) creating index... index created! Done (t=0.05s) creating index... index created! loading annotations into memory... loading annotations into memory... Done (t=0.03s) creating index... index created! loading annotations into memory... Done (t=0.05s) creating index... index created! Done (t=0.03s) creating index... index created! loading annotations into memory... loading annotations into memory... Done (t=0.03s) creating index... index created! Done (t=0.06s) creating index... index created! fatal: Not a git repository (or any parent up to mount point /root/paddlejob/workspace) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). self.flag: [1 1 1 ... 1 1 1] self.group_sizes: [ 941 18743] fatal: Not a git repository (or any parent up to mount point /root/paddlejob/workspace) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). self.flag: [1 1 1 ... 1 1 1] self.group_sizes: [ 941 18743] loading annotations into memory... Done (t=0.03s) creating index... index created! fatal: Not a git repository (or any parent up to mount point /workspace) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). self.flag: [1 1 1 ... 1 1 1] self.group_sizes: [ 941 18743] fatal: Not a git repository (or any parent up to mount point /workspace) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). self.flag: [1 1 1 ... 1 1 1] self.group_sizes: [ 941 18743] loading annotations into memory... loading annotations into memory... loading annotations into memory... loading annotations into memory... Done (t=0.05s) creating index... index created! Done (t=0.05s) creating index... Done (t=0.05s) creating index... Done (t=0.05s) creating index... index created! index created! index created! 2021-11-25 10:43:05,186 - mmdet.ssod - INFO - Start running, host: root@yq01-sys-hic-k8s-v100-box-a225-0055.yq01.baidu.com, work_dir: /work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k 2021-11-25 10:43:05,186 - mmdet.ssod - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) CheckpointHook
(NORMAL ) WeightSummary
(NORMAL ) MeanTeacher
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook

before_train_epoch: (VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) NumClassCheckHook
(LOW ) IterTimerHook
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook

before_train_iter: (VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) MeanTeacher
(LOW ) IterTimerHook
(80 ) SubModulesDistEvalHook

after_train_iter: (ABOVE_NORMAL) OptimizerHook
(NORMAL ) CheckpointHook
(NORMAL ) MeanTeacher
(LOW ) IterTimerHook
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook

after_train_epoch: (NORMAL ) CheckpointHook
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook

before_val_epoch: (NORMAL ) NumClassCheckHook
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook

before_val_iter: (LOW ) IterTimerHook

after_val_iter: (LOW ) IterTimerHook

after_val_epoch: (VERY_LOW ) TextLoggerHook

2021-11-25 10:43:05,187 - mmdet.ssod 2021-11-25 10:43:05,280 - mmdet.ssod +------------------------------------------ | Name +------------------------------------------ | teacher.backbone.conv1.weight | teacher.backbone.bn1.weight | teacher.backbone.bn1.bias | teacher.backbone.layer1.0.conv1.weight | teacher.backbone.layer1.0.bn1.weight | teacher.backbone.layer1.0.bn1.bias | teacher.backbone.layer1.0.conv2.weight | teacher.backbone.layer1.0.bn2.weight | teacher.backbone.layer1.0.bn2.bias | teacher.backbone.layer1.0.conv3.weight | teacher.backbone.layer1.0.bn3.weight | teacher.backbone.layer1.0.bn3.bias | teacher.backbone.layer1.0.downsample.0.weight | | teacher.backbone.layer1.0.downsample.1.weight | | teacher.backbone.layer1.0.downsample.1.bias | | teacher.backbone.layer1.1.conv1.weight | teacher.backbone.layer1.1.bn1.weight | teacher.backbone.layer1.1.bn1.bias | teacher.backbone.layer1.1.conv2.weight | teacher.backbone.layer1.1.bn2.weight | teacher.backbone.layer1.1.bn2.bias | teacher.backbone.layer1.1.conv3.weight | teacher.backbone.layer1.1.bn3.weight | teacher.backbone.layer1.1.bn3.bias | teacher.backbone.layer1.2.conv1.weight | teacher.backbone.layer1.2.bn1.weight | teacher.backbone.layer1.2.bn1.bias | teacher.backbone.layer1.2.conv2.weight | teacher.backbone.layer1.2.bn2.weight | teacher.backbone.layer1.2.bn2.bias | teacher.backbone.layer1.2.conv3.weight | teacher.backbone.layer1.2.bn3.weight | teacher.backbone.layer1.2.bn3.bias | teacher.backbone.layer2.0.conv1.weight | teacher.backbone.layer2.0.bn1.weight | teacher.backbone.layer2.0.bn1.bias | teacher.backbone.layer2.0.conv2.weight | teacher.backbone.layer2.0.bn2.weight | teacher.backbone.layer2.0.bn2.bias | teacher.backbone.layer2.0.conv3.weight | teacher.backbone.layer2.0.bn3.weight | teacher.backbone.layer2.0.bn3.bias | teacher.backbone.layer2.0.downsample.0.weight | | teacher.backbone.layer2.0.downsample.1.weight | | teacher.backbone.layer2.0.downsample.1.bias | | teacher.backbone.layer2.1.conv1.weight | teacher.backbone.layer2.1.bn1.weight | teacher.backbone.layer2.1.bn1.bias | teacher.backbone.layer2.1.conv2.weight | teacher.backbone.layer2.1.bn2.weight | teacher.backbone.layer2.1.bn2.bias | teacher.backbone.layer2.1.conv3.weight | teacher.backbone.layer2.1.bn3.weight | teacher.backbone.layer2.1.bn3.bias | teacher.backbone.layer2.2.conv1.weight | teacher.backbone.layer2.2.bn1.weight | teacher.backbone.layer2.2.bn1.bias | teacher.backbone.layer2.2.conv2.weight | teacher.backbone.layer2.2.bn2.weight | teacher.backbone.layer2.2.bn2.bias | teacher.backbone.layer2.2.conv3.weight | teacher.backbone.layer2.2.bn3.weight | teacher.backbone.layer2.2.bn3.bias | teacher.backbone.layer2.3.conv1.weight | teacher.backbone.layer2.3.bn1.weight | teacher.backbone.layer2.3.bn1.bias | teacher.backbone.layer2.3.conv2.weight | teacher.backbone.layer2.3.bn2.weight | teacher.backbone.layer2.3.bn2.bias | teacher.backbone.layer2.3.conv3.weight | teacher.backbone.layer2.3.bn3.weight | teacher.backbone.layer2.3.bn3.bias | teacher.backbone.layer3.0.conv1.weight | teacher.backbone.layer3.0.bn1.weight | teacher.backbone.layer3.0.bn1.bias | teacher.backbone.layer3.0.conv2.weight | teacher.backbone.layer3.0.bn2.weight | teacher.backbone.layer3.0.bn2.bias | teacher.backbone.layer3.0.conv3.weight | teacher.backbone.layer3.0.bn3.weight | teacher.backbone.layer3.0.bn3.bias | teacher.backbone.layer3.0.downsample.0.weight | | teacher.backbone.layer3.0.downsample.1.weight | | teacher.backbone.layer3.0.downsample.1.bias | | teacher.backbone.layer3.1.conv1.weight | teacher.backbone.layer3.1.bn1.weight | teacher.backbone.layer3.1.bn1.bias | teacher.backbone.layer3.1.conv2.weight | teacher.backbone.layer3.1.bn2.weight | teacher.backbone.layer3.1.bn2.bias | teacher.backbone.layer3.1.conv3.weight | teacher.backbone.layer3.1.bn3.weight | teacher.backbone.layer3.1.bn3.bias | teacher.backbone.layer3.2.conv1.weight | teacher.backbone.layer3.2.bn1.weight | teacher.backbone.layer3.2.bn1.bias | teacher.backbone.layer3.2.conv2.weight | teacher.backbone.layer3.2.bn2.weight | teacher.backbone.layer3.2.bn2.bias | teacher.backbone.layer3.2.conv3.weight | teacher.backbone.layer3.2.bn3.weight | teacher.backbone.layer3.2.bn3.bias | teacher.backbone.layer3.3.conv1.weight | teacher.backbone.layer3.3.bn1.weight | teacher.backbone.layer3.3.bn1.bias | teacher.backbone.layer3.3.conv2.weight | teacher.backbone.layer3.3.bn2.weight | teacher.backbone.layer3.3.bn2.bias | teacher.backbone.layer3.3.conv3.weight | teacher.backbone.layer3.3.bn3.weight | teacher.backbone.layer3.3.bn3.bias | teacher.backbone.layer3.4.conv1.weight | teacher.backbone.layer3.4.bn1.weight | teacher.backbone.layer3.4.bn1.bias | teacher.backbone.layer3.4.conv2.weight | teacher.backbone.layer3.4.bn2.weight | teacher.backbone.layer3.4.bn2.bias | teacher.backbone.layer3.4.conv3.weight | teacher.backbone.layer3.4.bn3.weight | teacher.backbone.layer3.4.bn3.bias | teacher.backbone.layer3.5.conv1.weight | teacher.backbone.layer3.5.bn1.weight | teacher.backbone.layer3.5.bn1.bias | teacher.backbone.layer3.5.conv2.weight | teacher.backbone.layer3.5.bn2.weight | teacher.backbone.layer3.5.bn2.bias | teacher.backbone.layer3.5.conv3.weight | teacher.backbone.layer3.5.bn3.weight | teacher.backbone.layer3.5.bn3.bias | teacher.backbone.layer4.0.conv1.weight | teacher.backbone.layer4.0.bn1.weight | teacher.backbone.layer4.0.bn1.bias | teacher.backbone.layer4.0.conv2.weight | teacher.backbone.layer4.0.bn2.weight | teacher.backbone.layer4.0.bn2.bias | teacher.backbone.layer4.0.conv3.weight | teacher.backbone.layer4.0.bn3.weight | teacher.backbone.layer4.0.bn3.bias | teacher.backbone.layer4.0.downsample.0.weight | | teacher.backbone.layer4.0.downsample.1.weight | | teacher.backbone.layer4.0.downsample.1.bias | | teacher.backbone.layer4.1.conv1.weight | teacher.backbone.layer4.1.bn1.weight | teacher.backbone.layer4.1.bn1.bias | teacher.backbone.layer4.1.conv2.weight | teacher.backbone.layer4.1.bn2.weight | teacher.backbone.layer4.1.bn2.bias | teacher.backbone.layer4.1.conv3.weight | teacher.backbone.layer4.1.bn3.weight | teacher.backbone.layer4.1.bn3.bias | teacher.backbone.layer4.2.conv1.weight | teacher.backbone.layer4.2.bn1.weight | teacher.backbone.layer4.2.bn1.bias | teacher.backbone.layer4.2.conv2.weight | teacher.backbone.layer4.2.bn2.weight | teacher.backbone.layer4.2.bn2.bias | teacher.backbone.layer4.2.conv3.weight | teacher.backbone.layer4.2.bn3.weight | teacher.backbone.layer4.2.bn3.bias | teacher.neck.lateral_convs.0.conv.weight | | teacher.neck.lateral_convs.0.conv.bias | teacher.neck.lateral_convs.1.conv.weight | | teacher.neck.lateral_convs.1.conv.bias | teacher.neck.lateral_convs.2.conv.weight | | teacher.neck.lateral_convs.2.conv.bias | teacher.neck.lateral_convs.3.conv.weight | | teacher.neck.lateral_convs.3.conv.bias | teacher.neck.fpn_convs.0.conv.weight | teacher.neck.fpn_convs.0.conv.bias | teacher.neck.fpn_convs.1.conv.weight | teacher.neck.fpn_convs.1.conv.bias | teacher.neck.fpn_convs.2.conv.weight | teacher.neck.fpn_convs.2.conv.bias | teacher.neck.fpn_convs.3.conv.weight | teacher.neck.fpn_convs.3.conv.bias | teacher.rpn_head.rpn_conv.weight | teacher.rpn_head.rpn_conv.bias | teacher.rpn_head.rpn_cls.weight | teacher.rpn_head.rpn_cls.bias | teacher.rpn_head.rpn_reg.weight | teacher.rpn_head.rpn_reg.bias | teacher.roi_head.bbox_head.fc_cls.weight | | teacher.roi_head.bbox_head.fc_cls.bias | teacher.roi_head.bbox_head.fc_reg.weight | | teacher.roi_head.bbox_head.fc_reg.bias | teacher.roi_head.bbox_head.shared_fcs.0.weight | | teacher.roi_head.bbox_head.shared_fcs.0.bias | | teacher.roi_head.bbox_head.shared_fcs.1.weight | | teacher.roi_head.bbox_head.shared_fcs.1.bias | | student.backbone.conv1.weight | student.backbone.bn1.weight | student.backbone.bn1.bias | student.backbone.layer1.0.conv1.weight | student.backbone.layer1.0.bn1.weight | student.backbone.layer1.0.bn1.bias | student.backbone.layer1.0.conv2.weight | student.backbone.layer1.0.bn2.weight | student.backbone.layer1.0.bn2.bias | student.backbone.layer1.0.conv3.weight | student.backbone.layer1.0.bn3.weight | student.backbone.layer1.0.bn3.bias | student.backbone.layer1.0.downsample.0.weight | | student.backbone.layer1.0.downsample.1.weight | | student.backbone.layer1.0.downsample.1.bias | | student.backbone.layer1.1.conv1.weight | student.backbone.layer1.1.bn1.weight | student.backbone.layer1.1.bn1.bias | student.backbone.layer1.1.conv2.weight | student.backbone.layer1.1.bn2.weight | student.backbone.layer1.1.bn2.bias | student.backbone.layer1.1.conv3.weight | student.backbone.layer1.1.bn3.weight | student.backbone.layer1.1.bn3.bias | student.backbone.layer1.2.conv1.weight | student.backbone.layer1.2.bn1.weight | student.backbone.layer1.2.bn1.bias | student.backbone.layer1.2.conv2.weight | student.backbone.layer1.2.bn2.weight | student.backbone.layer1.2.bn2.bias | student.backbone.layer1.2.conv3.weight | student.backbone.layer1.2.bn3.weight | student.backbone.layer1.2.bn3.bias | student.backbone.layer2.0.conv1.weight | student.backbone.layer2.0.bn1.weight | student.backbone.layer2.0.bn1.bias | student.backbone.layer2.0.conv2.weight | student.backbone.layer2.0.bn2.weight | student.backbone.layer2.0.bn2.bias | student.backbone.layer2.0.conv3.weight | student.backbone.layer2.0.bn3.weight | student.backbone.layer2.0.bn3.bias | student.backbone.layer2.0.downsample.0.weight | | student.backbone.layer2.0.downsample.1.weight | | student.backbone.layer2.0.downsample.1.bias | | student.backbone.layer2.1.conv1.weight | student.backbone.layer2.1.bn1.weight | student.backbone.layer2.1.bn1.bias | student.backbone.layer2.1.conv2.weight | student.backbone.layer2.1.bn2.weight | student.backbone.layer2.1.bn2.bias | student.backbone.layer2.1.conv3.weight | student.backbone.layer2.1.bn3.weight | student.backbone.layer2.1.bn3.bias | student.backbone.layer2.2.conv1.weight | student.backbone.layer2.2.bn1.weight | student.backbone.layer2.2.bn1.bias | student.backbone.layer2.2.conv2.weight | student.backbone.layer2.2.bn2.weight | student.backbone.layer2.2.bn2.bias | student.backbone.layer2.2.conv3.weight | student.backbone.layer2.2.bn3.weight | student.backbone.layer2.2.bn3.bias | student.backbone.layer2.3.conv1.weight | student.backbone.layer2.3.bn1.weight | student.backbone.layer2.3.bn1.bias | student.backbone.layer2.3.conv2.weight | student.backbone.layer2.3.bn2.weight | student.backbone.layer2.3.bn2.bias | student.backbone.layer2.3.conv3.weight | student.backbone.layer2.3.bn3.weight | student.backbone.layer2.3.bn3.bias | student.backbone.layer3.0.conv1.weight | student.backbone.layer3.0.bn1.weight | student.backbone.layer3.0.bn1.bias | student.backbone.layer3.0.conv2.weight | student.backbone.layer3.0.bn2.weight | student.backbone.layer3.0.bn2.bias | student.backbone.layer3.0.conv3.weight | student.backbone.layer3.0.bn3.weight | student.backbone.layer3.0.bn3.bias | student.backbone.layer3.0.downsample.0.weight | | student.backbone.layer3.0.downsample.1.weight | | student.backbone.layer3.0.downsample.1.bias | | student.backbone.layer3.1.conv1.weight | student.backbone.layer3.1.bn1.weight | student.backbone.layer3.1.bn1.bias | student.backbone.layer3.1.conv2.weight | student.backbone.layer3.1.bn2.weight | student.backbone.layer3.1.bn2.bias | student.backbone.layer3.1.conv3.weight | student.backbone.layer3.1.bn3.weight | student.backbone.layer3.1.bn3.bias | student.backbone.layer3.2.conv1.weight | student.backbone.layer3.2.bn1.weight | student.backbone.layer3.2.bn1.bias | student.backbone.layer3.2.conv2.weight | student.backbone.layer3.2.bn2.weight | student.backbone.layer3.2.bn2.bias | student.backbone.layer3.2.conv3.weight | student.backbone.layer3.2.bn3.weight | student.backbone.layer3.2.bn3.bias | student.backbone.layer3.3.conv1.weight | student.backbone.layer3.3.bn1.weight | student.backbone.layer3.3.bn1.bias | student.backbone.layer3.3.conv2.weight | student.backbone.layer3.3.bn2.weight | student.backbone.layer3.3.bn2.bias | student.backbone.layer3.3.conv3.weight | student.backbone.layer3.3.bn3.weight | student.backbone.layer3.3.bn3.bias | student.backbone.layer3.4.conv1.weight | student.backbone.layer3.4.bn1.weight | student.backbone.layer3.4.bn1.bias | student.backbone.layer3.4.conv2.weight | student.backbone.layer3.4.bn2.weight | student.backbone.layer3.4.bn2.bias | student.backbone.layer3.4.conv3.weight | student.backbone.layer3.4.bn3.weight | student.backbone.layer3.4.bn3.bias | student.backbone.layer3.5.conv1.weight | student.backbone.layer3.5.bn1.weight | student.backbone.layer3.5.bn1.bias | student.backbone.layer3.5.conv2.weight | student.backbone.layer3.5.bn2.weight | student.backbone.layer3.5.bn2.bias | student.backbone.layer3.5.conv3.weight | student.backbone.layer3.5.bn3.weight | student.backbone.layer3.5.bn3.bias | student.backbone.layer4.0.conv1.weight | student.backbone.layer4.0.bn1.weight | student.backbone.layer4.0.bn1.bias | student.backbone.layer4.0.conv2.weight | student.backbone.layer4.0.bn2.weight | student.backbone.layer4.0.bn2.bias | student.backbone.layer4.0.conv3.weight | student.backbone.layer4.0.bn3.weight | student.backbone.layer4.0.bn3.bias | student.backbone.layer4.0.downsample.0.weight | | student.backbone.layer4.0.downsample.1.weight | | student.backbone.layer4.0.downsample.1.bias | | student.backbone.layer4.1.conv1.weight | student.backbone.layer4.1.bn1.weight | student.backbone.layer4.1.bn1.bias | student.backbone.layer4.1.conv2.weight | student.backbone.layer4.1.bn2.weight | student.backbone.layer4.1.bn2.bias | student.backbone.layer4.1.conv3.weight | student.backbone.layer4.1.bn3.weight | student.backbone.layer4.1.bn3.bias | student.backbone.layer4.2.conv1.weight | student.backbone.layer4.2.bn1.weight | student.backbone.layer4.2.bn1.bias | student.backbone.layer4.2.conv2.weight | student.backbone.layer4.2.bn2.weight | student.backbone.layer4.2.bn2.bias | student.backbone.layer4.2.conv3.weight | student.backbone.layer4.2.bn3.weight | student.backbone.layer4.2.bn3.bias | student.neck.lateral_convs.0.conv.weight | | student.neck.lateral_convs.0.conv.bias | student.neck.lateral_convs.1.conv.weight | | student.neck.lateral_convs.1.conv.bias | student.neck.lateral_convs.2.conv.weight | | student.neck.lateral_convs.2.conv.bias | student.neck.lateral_convs.3.conv.weight | | student.neck.lateral_convs.3.conv.bias | student.neck.fpn_convs.0.conv.weight | student.neck.fpn_convs.0.conv.bias | student.neck.fpn_convs.1.conv.weight | student.neck.fpn_convs.1.conv.bias | student.neck.fpn_convs.2.conv.weight | student.neck.fpn_convs.2.conv.bias | student.neck.fpn_convs.3.conv.weight | student.neck.fpn_convs.3.conv.bias | student.rpn_head.rpn_conv.weight | student.rpn_head.rpn_conv.bias | student.rpn_head.rpn_cls.weight | student.rpn_head.rpn_cls.bias | student.rpn_head.rpn_reg.weight | student.rpn_head.rpn_reg.bias | student.roi_head.bbox_head.fc_cls.weight | | student.roi_head.bbox_head.fc_cls.bias | student.roi_head.bbox_head.fc_reg.weight | | student.roi_head.bbox_head.fc_reg.bias | student.roi_head.bbox_head.shared_fcs.0.weight | | student.roi_head.bbox_head.shared_fcs.0.bias | | student.roi_head.bbox_head.shared_fcs.1.weight | | student.roi_head.bbox_head.shared_fcs.1.bias | +------------------------------------------ self.by_prob: False indices: 14460, self: 14460 self.epoch_length: [691, 13769], self.samples_per_gpu: self.by_prob: False indices: 14460, self: 14460 self.epoch_length: [691, 13769], self.samples_per_gpu: self.by_prob: False indices: 14460, self: 14460 self.epoch_length: [691, 13769], self.samples_per_gpu: self.by_prob: False indices: 14460, self: 14460 self.epoch_length: [691, 13769], self.samples_per_g: - INFO - workflow: [('train', 1)], max: 720000 iters - INFO - ------+-----------+---------------+-----------------------+-------+--------+ | Optimized | Shape | Value Scale [Min,Max] | Lr | Wd | ------+-----------+---------------+-----------------------+-------+--------+ | N | 64X3X7X7 | Min:-0.671 Max:0.704 | 0.001 | 0.0001 | | N | 64 | Min:0.513 Max:2.669 | 0.001 | 0.0001 | | N | 64 | Min:-2.654 Max:6.354 | 0.001 | 0.0001 | | N | 64X64X1X1 | Min:-0.717 Max:0.392 | 0.001 | 0.0001 | | N | 64 | Min:0.509 Max:2.066 | 0.001 | 0.0001 | | N | 64 | Min:-2.411 Max:3.608 | 0.001 | 0.0001 | | N | 64X64X3X3 | Min:-0.390 Max:0.364 | 0.001 | 0.0001 | | N | 64 | Min:0.420 Max:2.530 | 0.001 | 0.0001 | | N | 64 | Min:-2.286 Max:5.913 | 0.001 | 0.0001 | | N | 256X64X1X1 | Min:-0.397 Max:0.348 | 0.001 | 0.0001 | | N | 256 | Min:0.011 Max:2.820 | 0.001 | 0.0001 | | N | 256 | Min:-1.126 Max:1.522 | 0.001 | 0.0001 | N | 256X64X1X1 | Min:-0.772 Max:0.900 | 0.001 | 0.0001 | N | 256 | Min:0.004 Max:3.064 | 0.001 | 0.0001 | N | 256 | Min:-1.126 Max:1.522 | 0.001 | 0.0001 | | N | 64X256X1X1 | Min:-0.297 Max:0.220 | 0.001 | 0.0001 | | N | 64 | Min:0.746 Max:1.949 | 0.001 | 0.0001 | | N | 64 | Min:-1.688 Max:1.578 | 0.001 | 0.0001 | | N | 64X64X3X3 | Min:-0.240 Max:0.318 | 0.001 | 0.0001 | | N | 64 | Min:0.621 Max:1.618 | 0.001 | 0.0001 | | N | 64 | Min:-2.003 Max:2.398 | 0.001 | 0.0001 | | N | 256X64X1X1 | Min:-0.240 Max:0.280 | 0.001 | 0.0001 | | N | 256 | Min:-0.017 Max:2.130 | 0.001 | 0.0001 | | N | 256 | Min:-1.711 Max:1.291 | 0.001 | 0.0001 | | N | 64X256X1X1 | Min:-0.210 Max:0.264 | 0.001 | 0.0001 | | N | 64 | Min:0.574 Max:1.688 | 0.001 | 0.0001 | | N | 64 | Min:-1.876 Max:1.090 | 0.001 | 0.0001 | | N | 64X64X3X3 | Min:-0.218 Max:0.201 | 0.001 | 0.0001 | | N | 64 | Min:0.757 Max:1.649 | 0.001 | 0.0001 | | N | 64 | Min:-2.221 Max:1.878 | 0.001 | 0.0001 | | N | 256X64X1X1 | Min:-0.275 Max:0.350 | 0.001 | 0.0001 | | N | 256 | Min:-0.058 Max:2.154 | 0.001 | 0.0001 | | N | 256 | Min:-1.570 Max:1.535 | 0.001 | 0.0001 | | N | 128X256X1X1 | Min:-0.334 Max:0.300 | 0.001 | 0.0001 | | N | 128 | Min:0.610 Max:1.642 | 0.001 | 0.0001 | | N | 128 | Min:-1.579 Max:1.449 | 0.001 | 0.0001 | | N | 128X128X3X3 | Min:-0.384 Max:0.377 | 0.001 | 0.0001 | | N | 128 | Min:0.605 Max:1.622 | 0.001 | 0.0001 | | N | 128 | Min:-2.768 Max:1.747 | 0.001 | 0.0001 | | N | 512X128X1X1 | Min:-0.374 Max:0.434 | 0.001 | 0.0001 | | N | 512 | Min:-0.007 Max:2.730 | 0.001 | 0.0001 | | N | 512 | Min:-1.545 Max:1.256 | 0.001 | 0.0001 | N | 512X256X1X1 | Min:-0.466 Max:0.642 | 0.001 | 0.0001 | N | 512 | Min:0.006 Max:2.552 | 0.001 | 0.0001 | N | 512 | Min:-1.545 Max:1.256 | 0.001 | 0.0001 | | N | 128X512X1X1 | Min:-0.162 Max:0.195 | 0.001 | 0.0001 | | N | 128 | Min:0.578 Max:1.429 | 0.001 | 0.0001 | | N | 128 | Min:-4.348 Max:0.588 | 0.001 | 0.0001 | | N | 128X128X3X3 | Min:-0.176 Max:0.177 | 0.001 | 0.0001 | | N | 128 | Min:0.511 Max:1.794 | 0.001 | 0.0001 | | N | 128 | Min:-3.825 Max:1.343 | 0.001 | 0.0001 | | N | 512X128X1X1 | Min:-0.344 Max:0.336 | 0.001 | 0.0001 | | N | 512 | Min:-0.072 Max:2.122 | 0.001 | 0.0001 | | N | 512 | Min:-1.502 Max:1.166 | 0.001 | 0.0001 | | N | 128X512X1X1 | Min:-0.330 Max:0.369 | 0.001 | 0.0001 | | N | 128 | Min:0.406 Max:1.696 | 0.001 | 0.0001 | | N | 128 | Min:-2.696 Max:1.944 | 0.001 | 0.0001 | | N | 128X128X3X3 | Min:-0.326 Max:0.374 | 0.001 | 0.0001 | | N | 128 | Min:0.460 Max:2.179 | 0.001 | 0.0001 | | N | 128 | Min:-1.587 Max:0.589 | 0.001 | 0.0001 | | N | 512X128X1X1 | Min:-0.288 Max:0.232 | 0.001 | 0.0001 | | N | 512 | Min:-0.006 Max:3.043 | 0.001 | 0.0001 | | N | 512 | Min:-2.369 Max:0.440 | 0.001 | 0.0001 | | N | 128X512X1X1 | Min:-0.298 Max:0.346 | 0.001 | 0.0001 | | N | 128 | Min:0.736 Max:2.394 | 0.001 | 0.0001 | | N | 128 | Min:-2.643 Max:0.756 | 0.001 | 0.0001 | | N | 128X128X3X3 | Min:-0.272 Max:0.208 | 0.001 | 0.0001 | | N | 128 | Min:0.682 Max:1.694 | 0.001 | 0.0001 | | N | 128 | Min:-1.365 Max:1.599 | 0.001 | 0.0001 | | N | 512X128X1X1 | Min:-0.279 Max:0.281 | 0.001 | 0.0001 | | N | 512 | Min:-0.009 Max:1.721 | 0.001 | 0.0001 | | N | 512 | Min:-1.897 Max:1.182 | 0.001 | 0.0001 | | N | 256X512X1X1 | Min:-0.230 Max:0.341 | 0.001 | 0.0001 | | N | 256 | Min:0.621 Max:1.636 | 0.001 | 0.0001 | | N | 256 | Min:-1.420 Max:0.917 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.267 Max:0.179 | 0.001 | 0.0001 | | N | 256 | Min:0.585 Max:1.749 | 0.001 | 0.0001 | | N | 256 | Min:-1.837 Max:1.398 | 0.001 | 0.0001 | | N | 1024X256X1X1 | Min:-0.333 Max:0.384 | 0.001 | 0.0001 | | N | 1024 | Min:0.071 Max:2.367 | 0.001 | 0.0001 | | N | 1024 | Min:-0.938 Max:0.887 | 0.001 | 0.0001 | N | 1024X512X1X1 | Min:-0.333 Max:0.421 | 0.001 | 0.0001 | N | 1024 | Min:0.034 Max:2.779 | 0.001 | 0.0001 | N | 1024 | Min:-0.938 Max:0.887 | 0.001 | 0.0001 | | N | 256X1024X1X1 | Min:-0.197 Max:0.236 | 0.001 | 0.0001 | | N | 256 | Min:0.566 Max:1.743 | 0.001 | 0.0001 | | N | 256 | Min:-2.703 Max:1.042 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.436 Max:0.196 | 0.001 | 0.0001 | | N | 256 | Min:0.515 Max:2.301 | 0.001 | 0.0001 | | N | 256 | Min:-2.548 Max:1.856 | 0.001 | 0.0001 | | N | 1024X256X1X1 | Min:-0.438 Max:0.295 | 0.001 | 0.0001 | | N | 1024 | Min:0.055 Max:1.943 | 0.001 | 0.0001 | | N | 1024 | Min:-1.647 Max:1.016 | 0.001 | 0.0001 | | N | 256X1024X1X1 | Min:-0.387 Max:0.337 | 0.001 | 0.0001 | | N | 256 | Min:0.463 Max:1.886 | 0.001 | 0.0001 | | N | 256 | Min:-2.399 Max:0.488 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.165 Max:0.258 | 0.001 | 0.0001 | | N | 256 | Min:0.555 Max:1.901 | 0.001 | 0.0001 | | N | 256 | Min:-1.655 Max:0.704 | 0.001 | 0.0001 | | N | 1024X256X1X1 | Min:-0.290 Max:0.261 | 0.001 | 0.0001 | | N | 1024 | Min:0.049 Max:1.450 | 0.001 | 0.0001 | | N | 1024 | Min:-1.201 Max:0.587 | 0.001 | 0.0001 | | N | 256X1024X1X1 | Min:-0.194 Max:0.295 | 0.001 | 0.0001 | | N | 256 | Min:0.442 Max:1.353 | 0.001 | 0.0001 | | N | 256 | Min:-2.322 Max:0.509 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.201 Max:0.176 | 0.001 | 0.0001 | | N | 256 | Min:0.529 Max:1.939 | 0.001 | 0.0001 | | N | 256 | Min:-1.610 Max:0.776 | 0.001 | 0.0001 | | N | 1024X256X1X1 | Min:-0.205 Max:0.239 | 0.001 | 0.0001 | | N | 1024 | Min:-0.037 Max:1.646 | 0.001 | 0.0001 | | N | 1024 | Min:-1.484 Max:0.344 | 0.001 | 0.0001 | | N | 256X1024X1X1 | Min:-0.226 Max:0.306 | 0.001 | 0.0001 | | N | 256 | Min:0.438 Max:1.446 | 0.001 | 0.0001 | | N | 256 | Min:-2.511 Max:0.557 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.147 Max:0.223 | 0.001 | 0.0001 | | N | 256 | Min:0.651 Max:1.858 | 0.001 | 0.0001 | | N | 256 | Min:-1.588 Max:0.661 | 0.001 | 0.0001 | | N | 1024X256X1X1 | Min:-0.178 Max:0.265 | 0.001 | 0.0001 | | N | 1024 | Min:-0.001 Max:1.501 | 0.001 | 0.0001 | | N | 1024 | Min:-1.108 Max:0.639 | 0.001 | 0.0001 | | N | 256X1024X1X1 | Min:-0.153 Max:0.330 | 0.001 | 0.0001 | | N | 256 | Min:0.425 Max:1.547 | 0.001 | 0.0001 | | N | 256 | Min:-1.972 Max:0.823 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.293 Max:0.276 | 0.001 | 0.0001 | | N | 256 | Min:0.650 Max:2.942 | 0.001 | 0.0001 | | N | 256 | Min:-1.093 Max:0.771 | 0.001 | 0.0001 | | N | 1024X256X1X1 | Min:-0.232 Max:0.294 | 0.001 | 0.0001 | | N | 1024 | Min:0.004 Max:1.984 | 0.001 | 0.0001 | | N | 1024 | Min:-1.636 Max:1.250 | 0.001 | 0.0001 | | N | 512X1024X1X1 | Min:-0.184 Max:0.331 | 0.001 | 0.0001 | | N | 512 | Min:0.535 Max:1.594 | 0.001 | 0.0001 | | N | 512 | Min:-1.756 Max:0.288 | 0.001 | 0.0001 | | N | 512X512X3X3 | Min:-0.175 Max:0.272 | 0.001 | 0.0001 | | N | 512 | Min:0.456 Max:1.542 | 0.001 | 0.0001 | | N | 512 | Min:-1.820 Max:0.839 | 0.001 | 0.0001 | | N | 2048X512X1X1 | Min:-0.332 Max:0.432 | 0.001 | 0.0001 | | N | 2048 | Min:0.888 Max:3.492 | 0.001 | 0.0001 | | N | 2048 | Min:-1.810 Max:0.980 | 0.001 | 0.0001 | N | 2048X1024X1X1 | Min:-0.622 Max:0.465 | 0.001 | 0.0001 | N | 2048 | Min:0.261 Max:4.575 | 0.001 | 0.0001 | N | 2048 | Min:-1.810 Max:0.980 | 0.001 | 0.0001 | | N | 512X2048X1X1 | Min:-0.316 Max:0.577 | 0.001 | 0.0001 | | N | 512 | Min:0.398 Max:1.429 | 0.001 | 0.0001 | | N | 512 | Min:-1.380 Max:0.428 | 0.001 | 0.0001 | | N | 512X512X3X3 | Min:-0.217 Max:0.284 | 0.001 | 0.0001 | | N | 512 | Min:0.349 Max:1.550 | 0.001 | 0.0001 | | N | 512 | Min:-1.867 Max:0.880 | 0.001 | 0.0001 | | N | 2048X512X1X1 | Min:-0.200 Max:0.277 | 0.001 | 0.0001 | | N | 2048 | Min:0.574 Max:2.847 | 0.001 | 0.0001 | | N | 2048 | Min:-2.638 Max:0.544 | 0.001 | 0.0001 | | N | 512X2048X1X1 | Min:-0.289 Max:0.514 | 0.001 | 0.0001 | | N | 512 | Min:0.366 Max:1.249 | 0.001 | 0.0001 | | N | 512 | Min:-1.664 Max:0.753 | 0.001 | 0.0001 | | N | 512X512X3X3 | Min:-0.142 Max:0.144 | 0.001 | 0.0001 | | N | 512 | Min:0.516 Max:1.335 | 0.001 | 0.0001 | | N | 512 | Min:-1.871 Max:1.181 | 0.001 | 0.0001 | | N | 2048X512X1X1 | Min:-0.135 Max:0.300 | 0.001 | 0.0001 | | N | 2048 | Min:0.435 Max:3.073 | 0.001 | 0.0001 | | N | 2048 | Min:-3.885 Max:-0.249 | 0.001 | 0.0001 | N | 256X256X1X1 | Min:-0.108 Max:0.108 | 0.001 | 0.0001 | | N | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | N | 256X512X1X1 | Min:-0.088 Max:0.088 | 0.001 | 0.0001 | | N | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | N | 256X1024X1X1 | Min:-0.068 Max:0.068 | 0.001 | 0.0001 | | N | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | N | 256X2048X1X1 | Min:-0.051 Max:0.051 | 0.001 | 0.0001 | | N | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.001 | 0.0001 | | N | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.001 | 0.0001 | | N | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.001 | 0.0001 | | N | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.001 | 0.0001 | | N | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | N | 256X256X3X3 | Min:-0.047 Max:0.046 | 0.001 | 0.0001 | | N | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | N | 5X256X1X1 | Min:-0.030 Max:0.030 | 0.001 | 0.0001 | | N | 5 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | N | 20X256X1X1 | Min:-0.037 Max:0.033 | 0.001 | 0.0001 | | N | 20 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | N | 2X1024 | Min:-0.138 Max:0.143 | 0.001 | 0.0001 | | N | 2 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | N | 4X1024 | Min:-0.165 Max:0.143 | 0.001 | 0.0001 | | N | 4 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | N | 1024X12544 | Min:-0.068 Max:0.061 | 0.001 | 0.0001 | N | 1024 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | N | 1024X1024 | Min:-0.155 Max:0.154 | 0.001 | 0.0001 | N | 1024 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | N | 64X3X7X7 | Min:-0.671 Max:0.704 | 0.001 | 0.0001 | | N | 64 | Min:0.513 Max:2.669 | 0.001 | 0.0001 | | N | 64 | Min:-2.654 Max:6.354 | 0.001 | 0.0001 | | N | 64X64X1X1 | Min:-0.717 Max:0.392 | 0.001 | 0.0001 | | N | 64 | Min:0.509 Max:2.066 | 0.001 | 0.0001 | | N | 64 | Min:-2.411 Max:3.608 | 0.001 | 0.0001 | | N | 64X64X3X3 | Min:-0.390 Max:0.364 | 0.001 | 0.0001 | | N | 64 | Min:0.420 Max:2.530 | 0.001 | 0.0001 | | N | 64 | Min:-2.286 Max:5.913 | 0.001 | 0.0001 | | N | 256X64X1X1 | Min:-0.397 Max:0.348 | 0.001 | 0.0001 | | N | 256 | Min:0.011 Max:2.820 | 0.001 | 0.0001 | | N | 256 | Min:-1.126 Max:1.522 | 0.001 | 0.0001 | N | 256X64X1X1 | Min:-0.772 Max:0.900 | 0.001 | 0.0001 | N | 256 | Min:0.004 Max:3.064 | 0.001 | 0.0001 | N | 256 | Min:-1.126 Max:1.522 | 0.001 | 0.0001 | | N | 64X256X1X1 | Min:-0.297 Max:0.220 | 0.001 | 0.0001 | | N | 64 | Min:0.746 Max:1.949 | 0.001 | 0.0001 | | N | 64 | Min:-1.688 Max:1.578 | 0.001 | 0.0001 | | N | 64X64X3X3 | Min:-0.240 Max:0.318 | 0.001 | 0.0001 | | N | 64 | Min:0.621 Max:1.618 | 0.001 | 0.0001 | | N | 64 | Min:-2.003 Max:2.398 | 0.001 | 0.0001 | | N | 256X64X1X1 | Min:-0.240 Max:0.280 | 0.001 | 0.0001 | | N | 256 | Min:-0.017 Max:2.130 | 0.001 | 0.0001 | | N | 256 | Min:-1.711 Max:1.291 | 0.001 | 0.0001 | | N | 64X256X1X1 | Min:-0.210 Max:0.264 | 0.001 | 0.0001 | | N | 64 | Min:0.574 Max:1.688 | 0.001 | 0.0001 | | N | 64 | Min:-1.876 Max:1.090 | 0.001 | 0.0001 | | N | 64X64X3X3 | Min:-0.218 Max:0.201 | 0.001 | 0.0001 | | N | 64 | Min:0.757 Max:1.649 | 0.001 | 0.0001 | | N | 64 | Min:-2.221 Max:1.878 | 0.001 | 0.0001 | | N | 256X64X1X1 | Min:-0.275 Max:0.350 | 0.001 | 0.0001 | | N | 256 | Min:-0.058 Max:2.154 | 0.001 | 0.0001 | | N | 256 | Min:-1.570 Max:1.535 | 0.001 | 0.0001 | | Y | 128X256X1X1 | Min:-0.334 Max:0.300 | 0.001 | 0.0001 | | N | 128 | Min:0.610 Max:1.642 | 0.001 | 0.0001 | | N | 128 | Min:-1.579 Max:1.449 | 0.001 | 0.0001 | | Y | 128X128X3X3 | Min:-0.384 Max:0.377 | 0.001 | 0.0001 | | N | 128 | Min:0.605 Max:1.622 | 0.001 | 0.0001 | | N | 128 | Min:-2.768 Max:1.747 | 0.001 | 0.0001 | | Y | 512X128X1X1 | Min:-0.374 Max:0.434 | 0.001 | 0.0001 | | N | 512 | Min:-0.007 Max:2.730 | 0.001 | 0.0001 | | N | 512 | Min:-1.545 Max:1.256 | 0.001 | 0.0001 | Y | 512X256X1X1 | Min:-0.466 Max:0.642 | 0.001 | 0.0001 | N | 512 | Min:0.006 Max:2.552 | 0.001 | 0.0001 | N | 512 | Min:-1.545 Max:1.256 | 0.001 | 0.0001 | | Y | 128X512X1X1 | Min:-0.162 Max:0.195 | 0.001 | 0.0001 | | N | 128 | Min:0.578 Max:1.429 | 0.001 | 0.0001 | | N | 128 | Min:-4.348 Max:0.588 | 0.001 | 0.0001 | | Y | 128X128X3X3 | Min:-0.176 Max:0.177 | 0.001 | 0.0001 | | N | 128 | Min:0.511 Max:1.794 | 0.001 | 0.0001 | | N | 128 | Min:-3.825 Max:1.343 | 0.001 | 0.0001 | | Y | 512X128X1X1 | Min:-0.344 Max:0.336 | 0.001 | 0.0001 | | N | 512 | Min:-0.072 Max:2.122 | 0.001 | 0.0001 | | N | 512 | Min:-1.502 Max:1.166 | 0.001 | 0.0001 | | Y | 128X512X1X1 | Min:-0.330 Max:0.369 | 0.001 | 0.0001 | | N | 128 | Min:0.406 Max:1.696 | 0.001 | 0.0001 | | N | 128 | Min:-2.696 Max:1.944 | 0.001 | 0.0001 | | Y | 128X128X3X3 | Min:-0.326 Max:0.374 | 0.001 | 0.0001 | | N | 128 | Min:0.460 Max:2.179 | 0.001 | 0.0001 | | N | 128 | Min:-1.587 Max:0.589 | 0.001 | 0.0001 | | Y | 512X128X1X1 | Min:-0.288 Max:0.232 | 0.001 | 0.0001 | | N | 512 | Min:-0.006 Max:3.043 | 0.001 | 0.0001 | | N | 512 | Min:-2.369 Max:0.440 | 0.001 | 0.0001 | | Y | 128X512X1X1 | Min:-0.298 Max:0.346 | 0.001 | 0.0001 | | N | 128 | Min:0.736 Max:2.394 | 0.001 | 0.0001 | | N | 128 | Min:-2.643 Max:0.756 | 0.001 | 0.0001 | | Y | 128X128X3X3 | Min:-0.272 Max:0.208 | 0.001 | 0.0001 | | N | 128 | Min:0.682 Max:1.694 | 0.001 | 0.0001 | | N | 128 | Min:-1.365 Max:1.599 | 0.001 | 0.0001 | | Y | 512X128X1X1 | Min:-0.279 Max:0.281 | 0.001 | 0.0001 | | N | 512 | Min:-0.009 Max:1.721 | 0.001 | 0.0001 | | N | 512 | Min:-1.897 Max:1.182 | 0.001 | 0.0001 | | Y | 256X512X1X1 | Min:-0.230 Max:0.341 | 0.001 | 0.0001 | | N | 256 | Min:0.621 Max:1.636 | 0.001 | 0.0001 | | N | 256 | Min:-1.420 Max:0.917 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.267 Max:0.179 | 0.001 | 0.0001 | | N | 256 | Min:0.585 Max:1.749 | 0.001 | 0.0001 | | N | 256 | Min:-1.837 Max:1.398 | 0.001 | 0.0001 | | Y | 1024X256X1X1 | Min:-0.333 Max:0.384 | 0.001 | 0.0001 | | N | 1024 | Min:0.071 Max:2.367 | 0.001 | 0.0001 | | N | 1024 | Min:-0.938 Max:0.887 | 0.001 | 0.0001 | Y | 1024X512X1X1 | Min:-0.333 Max:0.421 | 0.001 | 0.0001 | N | 1024 | Min:0.034 Max:2.779 | 0.001 | 0.0001 | N | 1024 | Min:-0.938 Max:0.887 | 0.001 | 0.0001 | | Y | 256X1024X1X1 | Min:-0.197 Max:0.236 | 0.001 | 0.0001 | | N | 256 | Min:0.566 Max:1.743 | 0.001 | 0.0001 | | N | 256 | Min:-2.703 Max:1.042 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.436 Max:0.196 | 0.001 | 0.0001 | | N | 256 | Min:0.515 Max:2.301 | 0.001 | 0.0001 | | N | 256 | Min:-2.548 Max:1.856 | 0.001 | 0.0001 | | Y | 1024X256X1X1 | Min:-0.438 Max:0.295 | 0.001 | 0.0001 | | N | 1024 | Min:0.055 Max:1.943 | 0.001 | 0.0001 | | N | 1024 | Min:-1.647 Max:1.016 | 0.001 | 0.0001 | | Y | 256X1024X1X1 | Min:-0.387 Max:0.337 | 0.001 | 0.0001 | | N | 256 | Min:0.463 Max:1.886 | 0.001 | 0.0001 | | N | 256 | Min:-2.399 Max:0.488 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.165 Max:0.258 | 0.001 | 0.0001 | | N | 256 | Min:0.555 Max:1.901 | 0.001 | 0.0001 | | N | 256 | Min:-1.655 Max:0.704 | 0.001 | 0.0001 | | Y | 1024X256X1X1 | Min:-0.290 Max:0.261 | 0.001 | 0.0001 | | N | 1024 | Min:0.049 Max:1.450 | 0.001 | 0.0001 | | N | 1024 | Min:-1.201 Max:0.587 | 0.001 | 0.0001 | | Y | 256X1024X1X1 | Min:-0.194 Max:0.295 | 0.001 | 0.0001 | | N | 256 | Min:0.442 Max:1.353 | 0.001 | 0.0001 | | N | 256 | Min:-2.322 Max:0.509 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.201 Max:0.176 | 0.001 | 0.0001 | | N | 256 | Min:0.529 Max:1.939 | 0.001 | 0.0001 | | N | 256 | Min:-1.610 Max:0.776 | 0.001 | 0.0001 | | Y | 1024X256X1X1 | Min:-0.205 Max:0.239 | 0.001 | 0.0001 | | N | 1024 | Min:-0.037 Max:1.646 | 0.001 | 0.0001 | | N | 1024 | Min:-1.484 Max:0.344 | 0.001 | 0.0001 | | Y | 256X1024X1X1 | Min:-0.226 Max:0.306 | 0.001 | 0.0001 | | N | 256 | Min:0.438 Max:1.446 | 0.001 | 0.0001 | | N | 256 | Min:-2.511 Max:0.557 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.147 Max:0.223 | 0.001 | 0.0001 | | N | 256 | Min:0.651 Max:1.858 | 0.001 | 0.0001 | | N | 256 | Min:-1.588 Max:0.661 | 0.001 | 0.0001 | | Y | 1024X256X1X1 | Min:-0.178 Max:0.265 | 0.001 | 0.0001 | | N | 1024 | Min:-0.001 Max:1.501 | 0.001 | 0.0001 | | N | 1024 | Min:-1.108 Max:0.639 | 0.001 | 0.0001 | | Y | 256X1024X1X1 | Min:-0.153 Max:0.330 | 0.001 | 0.0001 | | N | 256 | Min:0.425 Max:1.547 | 0.001 | 0.0001 | | N | 256 | Min:-1.972 Max:0.823 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.293 Max:0.276 | 0.001 | 0.0001 | | N | 256 | Min:0.650 Max:2.942 | 0.001 | 0.0001 | | N | 256 | Min:-1.093 Max:0.771 | 0.001 | 0.0001 | | Y | 1024X256X1X1 | Min:-0.232 Max:0.294 | 0.001 | 0.0001 | | N | 1024 | Min:0.004 Max:1.984 | 0.001 | 0.0001 | | N | 1024 | Min:-1.636 Max:1.250 | 0.001 | 0.0001 | | Y | 512X1024X1X1 | Min:-0.184 Max:0.331 | 0.001 | 0.0001 | | N | 512 | Min:0.535 Max:1.594 | 0.001 | 0.0001 | | N | 512 | Min:-1.756 Max:0.288 | 0.001 | 0.0001 | | Y | 512X512X3X3 | Min:-0.175 Max:0.272 | 0.001 | 0.0001 | | N | 512 | Min:0.456 Max:1.542 | 0.001 | 0.0001 | | N | 512 | Min:-1.820 Max:0.839 | 0.001 | 0.0001 | | Y | 2048X512X1X1 | Min:-0.332 Max:0.432 | 0.001 | 0.0001 | | N | 2048 | Min:0.888 Max:3.492 | 0.001 | 0.0001 | | N | 2048 | Min:-1.810 Max:0.980 | 0.001 | 0.0001 | Y | 2048X1024X1X1 | Min:-0.622 Max:0.465 | 0.001 | 0.0001 | N | 2048 | Min:0.261 Max:4.575 | 0.001 | 0.0001 | N | 2048 | Min:-1.810 Max:0.980 | 0.001 | 0.0001 | | Y | 512X2048X1X1 | Min:-0.316 Max:0.577 | 0.001 | 0.0001 | | N | 512 | Min:0.398 Max:1.429 | 0.001 | 0.0001 | | N | 512 | Min:-1.380 Max:0.428 | 0.001 | 0.0001 | | Y | 512X512X3X3 | Min:-0.217 Max:0.284 | 0.001 | 0.0001 | | N | 512 | Min:0.349 Max:1.550 | 0.001 | 0.0001 | | N | 512 | Min:-1.867 Max:0.880 | 0.001 | 0.0001 | | Y | 2048X512X1X1 | Min:-0.200 Max:0.277 | 0.001 | 0.0001 | | N | 2048 | Min:0.574 Max:2.847 | 0.001 | 0.0001 | | N | 2048 | Min:-2.638 Max:0.544 | 0.001 | 0.0001 | | Y | 512X2048X1X1 | Min:-0.289 Max:0.514 | 0.001 | 0.0001 | | N | 512 | Min:0.366 Max:1.249 | 0.001 | 0.0001 | | N | 512 | Min:-1.664 Max:0.753 | 0.001 | 0.0001 | | Y | 512X512X3X3 | Min:-0.142 Max:0.144 | 0.001 | 0.0001 | | N | 512 | Min:0.516 Max:1.335 | 0.001 | 0.0001 | | N | 512 | Min:-1.871 Max:1.181 | 0.001 | 0.0001 | | Y | 2048X512X1X1 | Min:-0.135 Max:0.300 | 0.001 | 0.0001 | | N | 2048 | Min:0.435 Max:3.073 | 0.001 | 0.0001 | | N | 2048 | Min:-3.885 Max:-0.249 | 0.001 | 0.0001 | Y | 256X256X1X1 | Min:-0.108 Max:0.108 | 0.001 | 0.0001 | | Y | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | Y | 256X512X1X1 | Min:-0.088 Max:0.088 | 0.001 | 0.0001 | | Y | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | Y | 256X1024X1X1 | Min:-0.068 Max:0.068 | 0.001 | 0.0001 | | Y | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | Y | 256X2048X1X1 | Min:-0.051 Max:0.051 | 0.001 | 0.0001 | | Y | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.001 | 0.0001 | | Y | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.001 | 0.0001 | | Y | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.001 | 0.0001 | | Y | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.036 Max:0.036 | 0.001 | 0.0001 | | Y | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | Y | 256X256X3X3 | Min:-0.047 Max:0.044 | 0.001 | 0.0001 | | Y | 256 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | Y | 5X256X1X1 | Min:-0.033 Max:0.041 | 0.001 | 0.0001 | | Y | 5 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | | Y | 20X256X1X1 | Min:-0.034 Max:0.035 | 0.001 | 0.0001 | | Y | 20 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | Y | 2X1024 | Min:-0.150 Max:0.157 | 0.001 | 0.0001 | | Y | 2 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | Y | 4X1024 | Min:-0.166 Max:0.148 | 0.001 | 0.0001 | | Y | 4 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | Y | 1024X12544 | Min:-0.066 Max:0.064 | 0.001 | 0.0001 | Y | 1024 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | Y | 1024X1024 | Min:-0.147 Max:0.153 | 0.001 | 0.0001 | Y | 1024 | Min:0.000 Max:0.000 | 0.001 | 0.0001 | ------+-----------+---------------+-----------------------+-------+--------+ 8 8 8 8

1、我按照您说的file_name对应到id,，还是卡住了， 2、以上我的配置文件和log，部分路径因为隐私问题，我修改了一下log的一些路径，但是不影响您review log, 现在的情况是迭代12个iteration后就会卡住了，感觉是有些图像读不起进去还是其他什么原因。我如果把无标签的数据集和路径都换成和有标签数据完全一样，是没问题的。所以感觉是无标签数据制作有问题。

"2021-11-25 17:02:36", "id": 9856}, {"license": 1, "file_name": "000000009953.jpg", "coco_url": "/data/datasets/varea_datasets_2020_2021/20_21_all_images_rename/000000009953.jpg", "height": 1080, "width": 1920, "date_captured": "2021-11-25 17:02:36", "id": 9953}, {"license": 1, "file_name": "000000001809.jpg", "coco_url": "/data/datasets/varea_datasets_2020_2021/20_21_all_images_rename/000000001809.jpg", "height": 720, "width": 1280, "date_captured": "2021-11-25 17:02:36", "id": 1809}, {"license": 1, "file_name": "000000008735.jpg", "coco_url": "/data/datasets/varea_datasets_2020_2021/20_21_all_images_rename/000000008735.jpg", "height": 1080, "width": 1920, "date_captured": "2021-11-25 17:02:36", "id": 8735}, {"license": 1, "file_name": "000000003184.jpg", "coco_url": "/data/datasets/varea_datasets_2020_2021/20_21_all_images_rename/000000003184.jpg", "height": 720, "width": 1280, "date_captured": "2021-11-25 17:02:36", "id": 3184}, {"license": 1, "file_name": "000000008634.jpg", "coco_url": "/data/datasets/varea_datasets_2020_2021/20_21_all_images_rename/000000008634.jpg", "height": 1080, "width": 1920, "date_captured": "2021-11-25 17:02:36", "id": 8634},

这是我改过的无标签数据json文件。接着也是执行tools下的脚本。

还请指点。谢谢

lihuikenny commented 2 years ago

我尝试了各种版本的pytorch和cuda。单卡运行好像没问题，多卡运行就会卡住 @MendelXu

lihuikenny commented 2 years ago

您解决了这个问题么，如果解决了，请告知一下，谢谢

发件人: HannahHan1119 @.> 发送时间: 2021年11月26日 15:35 收件人: microsoft/SoftTeacher @.> 抄送: lihuikenny @.>; Author @.> 主题: Re: [microsoft/SoftTeacher] fatal: Not a git repository (or any parent up to mount point (Issue #118)

I have same issue. It seems no error with a gpu. However, when I use two gpus, it raise error like here #121https://github.com/microsoft/SoftTeacher/issues/121 .

― You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/SoftTeacher/issues/118#issuecomment-979753231, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALD4VKOK5OGWQXL2EZBBAQTUN42ERANCNFSM5IXOWKAA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

HannahHan1119 commented 2 years ago

I still raise error when 10 or more iteration. I also got error with only one gpu.

microsoft / SoftTeacher

fatal: Not a git repository (or any parent up to mount point #118

2021-11-25 10:42:55,021 - mmdet.ssod - INFO - [<StreamHandler (INFO)>, <FileHandler ./work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k/20211125_104254.log (INFO)>] 2021-11-25 10:42:55,021 - mmdet.ssod - INFO - Environment info:

TorchVision: 0.8.0 OpenCV: 4.2.0 MMCV: 1.3.14 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 10.2 MMDetection: 2.17.0+