assert len(cls_scores) == len(score_factors) AssertionError

leijie0106 commented 3 years ago

when I train the model using"bash tools/dist_train_partially,sh semi 1 10 2", there is a problem. Can you help me？ Traceback (most recent call last): File "tools/train.py", line 200, in main() File "tools/train.py", line 195, in main meta=meta, File "./ssod/apis/train.py", line 206, in train_detector runner.run(data_loaders, cfg.workflow) File "/home/myuser/.conda/envs/lj36/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 133, in run iter_runner(iter_loaders[i], kwargs) File "/home/myuser/.conda/envs/lj36/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 60, in train outputs = self.model.train_step(data_batch, self.optimizer, kwargs) File "/home/myuser/.conda/envs/lj36/lib/python3.6/site-packages/mmcv/parallel/distributed.py", line 53, in train_step output = self.module.train_step(inputs[0], kwargs[0]) File "/home/myuser/lj/mmdetection/mmdet/models/detectors/base.py", line 238, in train_step losses = self(data) File "/home/myuser/.conda/envs/lj36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(input, *kwargs) File "/home/myuser/.conda/envs/lj36/lib/python3.6/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func output = old_func(new_args, new_kwargs) File "/home/myuser/lj/mmdetection/mmdet/models/detectors/base.py", line 172, in forward return self.forward_train(img, img_metas, kwargs) File "./ssod/models/soft_teacher.py", line 50, in forward_train data_groups["unsup_teacher"], data_groups["unsup_student"] File "./ssod/models/soft_teacher.py", line 73, in foward_unsup_train else None, File "./ssod/models/soft_teacher.py", line 357, in extract_teacher_info rpn_out, img_metas, cfg=proposal_cfg File "/home/myuser/.conda/envs/lj36/lib/python3.6/site-packages/mmcv/runner/fp16_utils.py", line 214, in new_func output = old_func(new_args, **new_kwargs) File "/home/myuser/lj/mmdetection/mmdet/models/dense_heads/base_dense_head.py", line 72, in get_bboxes assert len(cls_scores) == len(score_factors) AssertionError

MendelXu commented 3 years ago

Could you post you the full log here?

sarmientoj24 commented 3 years ago

also getting the same error on the new update

sarmientoj24 commented 3 years ago

actually this is what i am getting

  File "/home/user/anaconda3/envs/pave/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 944, in __init__
    self._reset(loader, first_iter=True)
  File "/home/user/anaconda3/envs/pave/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 975, in _reset
    self._try_put_index()
  File "/home/user/anaconda3/envs/pave/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1209, in _try_put_index
    index = self._next_index()
  File "/home/user/anaconda3/envs/pave/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 512, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/home/user/anaconda3/envs/pave/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 226, in __iter__
    for idx in self.sampler:
  File "/home/user/pave/semi_supervised_pavement_detector/ssod/datasets/samplers/semi_sampler.py", line 185, in __iter__
    assert len(indices) == len(self)
AssertionError

Using cuda 10.2, pytorch 1.9, mmdet 2.16

leijie0106 commented 3 years ago

When I use mmdet 2.17, the problem is solved.

MendelXu commented 3 years ago

actually this is what i am getting

  File "/home/user/anaconda3/envs/pave/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 944, in __init__
    self._reset(loader, first_iter=True)
  File "/home/user/anaconda3/envs/pave/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 975, in _reset
    self._try_put_index()
  File "/home/user/anaconda3/envs/pave/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1209, in _try_put_index
    index = self._next_index()
  File "/home/user/anaconda3/envs/pave/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 512, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/home/user/anaconda3/envs/pave/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 226, in __iter__
    for idx in self.sampler:
  File "/home/user/pave/semi_supervised_pavement_detector/ssod/datasets/samplers/semi_sampler.py", line 185, in __iter__
    assert len(indices) == len(self)
AssertionError

Using cuda 10.2, pytorch 1.9, mmdet 2.16

Did you solve it? If not, may you share your config file here?

chenxinfeng4 commented 3 years ago

When I use mmdet 2.17, the problem is solved.

It also works for me to fix the same bug. Thank you.

lihuikenny commented 3 years ago

@sarmientoj24 @MendelXu ,我也是这个错误，我按照 full labeled data 进行训练的也是报了一样的错误我输出了indices和self 的长度，确实是不相等，但不知道啥原因， indices: 0, self: 14660 Traceback (most recent call last): File "tools/train.py", line 198, in main() File "tools/train.py", line 193, in main meta=meta,

assert len(indices) == len(self)

AssertionError 你们解决这个问题了吗

lihuikenny commented 3 years ago

When I use mmdet 2.17, the problem is solved. @leijie0106 请问你是怎么解决的，不管我怎么设置都会有下面的错误换了2.17也有，by_prob设置成False也是一样的错误 assert len(indices) == len(self) AssertionError self.by_prob: True indices: 0, self: 29320

JLC7 commented 2 years ago

I get the same problem but using mmdet 2.17 still doesn't work for me.

smntjugithub commented 2 years ago

我也遇到了这个问题怎么解决啊？

smntjugithub commented 2 years ago

这个问题很显著，亟待解决啊

smntjugithub commented 2 years ago

(Soft) smn@smn-Z87X-HD3:/media/smn/Elements SE/SoftTeacher-main$ bash tools/dist_train_partially.sh semi 2 10 1

TYPE=semi
FOLD=2
PERCENT=10
GPUS=1
PORT=29500 ++ dirname tools/dist_train_partially.sh
PYTHONPATH=tools/..:
[[ semi == \b\a\s\e\l\i\n\e ]] ++ dirname tools/dist_train_partially.sh
python -m torch.distributed.launch --nproc_per_node=1 --master_port=29500 tools/train.py configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k.py --launcher pytorch --cfg-options fold=2 percent=10 The module torch.distributed.launch is deprecated and going to be removed in future.Migrate to torch.distributed.run WARNING:torch.distributed.run:--use_env is deprecated and will be removed in future releases. Please read local_rank from os.environ('LOCAL_RANK') instead. INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs: entrypoint : tools/train.py min_nodes : 1 max_nodes : 1 nproc_per_node : 1 run_id : none rdzv_backend : static rdzv_endpoint : 127.0.0.1:29500 rdzv_configs : {'rank': 0, 'timeout': 900} max_restarts : 3 monitor_interval : 5 log_dir : None metrics_cfg : {}

INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_f7ekhqpw/none_1gfe0zmq INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group /home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/distributed/elastic/utils/store.py:53: FutureWarning: This is an experimental API and will be changed in future. "This is an experimental API and will be changed in future.", FutureWarning INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=0 master_addr=127.0.0.1 master_port=29500 group_rank=0 group_world_size=1 local_ranks=[0] role_ranks=[0] global_ranks=[0] role_world_sizes=[1] global_world_sizes=[1]

INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_f7ekhqpw/none_1gfe0zmq/attempt_0/0/error.json fatal: Not a git repository (or any parent up to mount point /media/smn/Elements SE) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). 2021-12-28 01:59:12,256 - mmdet.ssod - INFO - [<StreamHandler (INFO)>, <FileHandler /media/smn/Elements SE/SoftTeacher-main/work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k/10/2/20211228_015912.log (INFO)>] 2021-12-28 01:59:12,256 - mmdet.ssod - INFO - Environment info:

sys.platform: linux Python: 3.6.13 |Anaconda, Inc.| (default, Jun 4 2021, 14:25:59) [GCC 7.5.0] CUDA available: True GPU 0: NVIDIA GeForce GTX 1080 CUDA_HOME: :/usr/local/cuda-10.2 GCC: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609 PyTorch: 1.9.0+cu102 PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 10.2
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
CuDNN 7.6.5
Magma 2.5.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.10.0+cu102 OpenCV: 4.5.4 MMCV: 1.3.9 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 10.2 MMDetection: 2.17.0+

2021-12-28 01:59:14,716 - mmdet.ssod - INFO - Distributed training: True 2021-12-28 01:59:17,167 - mmdet.ssod - INFO - Config: model = dict( type='SoftTeacher', model=dict( type='FasterRCNN', backbone=dict( type='ResNet', depth=50, num_stages=4, out_indices=(0, 1, 2, 3), frozen_stages=1, norm_cfg=dict(type='BN', requires_grad=False), norm_eval=True, style='caffe', init_cfg=dict( type='Pretrained', checkpoint='open-mmlab://detectron2/resnet50_caffe')), neck=dict( type='FPN', in_channels=[256, 512, 1024, 2048], out_channels=256, num_outs=5), rpn_head=dict( type='RPNHead', in_channels=256, feat_channels=256, anchor_generator=dict( type='AnchorGenerator', scales=[8], ratios=[0.5, 1.0, 2.0], strides=[4, 8, 16, 32, 64]), bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[1.0, 1.0, 1.0, 1.0]), loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0)), roi_head=dict( type='StandardRoIHead', bbox_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict( type='RoIAlign', output_size=7, sampling_ratio=0), out_channels=256, featmap_strides=[4, 8, 16, 32]), bbox_head=dict( type='Shared2FCBBoxHead', in_channels=256, fc_out_channels=1024, roi_feat_size=7, num_classes=80, bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[0.1, 0.1, 0.2, 0.2]), reg_class_agnostic=False, loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0))), train_cfg=dict( rpn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.7, neg_iou_thr=0.3, min_pos_iou=0.3, match_low_quality=True, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=256, pos_fraction=0.5, neg_pos_ub=-1, add_gt_as_proposals=False), allowed_border=-1, pos_weight=-1, debug=False), rpn_proposal=dict( nms_pre=2000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.5, neg_iou_thr=0.5, min_pos_iou=0.5, match_low_quality=False, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=512, pos_fraction=0.25, neg_pos_ub=-1, add_gt_as_proposals=True), pos_weight=-1, debug=False)), test_cfg=dict( rpn=dict( nms_pre=1000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( score_thr=0.05, nms=dict(type='nms', iou_threshold=0.5), max_per_img=100))), train_cfg=dict( use_teacher_proposal=False, pseudo_label_initial_score_thr=0.5, rpn_pseudo_threshold=0.9, cls_pseudo_threshold=0.9, reg_pseudo_threshold=0.02, jitter_times=10, jitter_scale=0.06, min_pseduo_box_size=0, unsup_weight=4.0), test_cfg=dict(inference_on='student')) dataset_type = 'CocoDataset' data_root = 'data/coco/' img_norm_cfg = dict( mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False) train_pipeline = [ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='sup'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag')) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ] data = dict( samples_per_gpu=4, workers_per_gpu=4, train=dict( type='SemiDataset', sup=dict( type='CocoDataset', ann_file= 'data/coco/annotations/semi_supervised/instances_train2017.2@10.json', img_prefix='data/coco/train2017/', pipeline=[ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='sup'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag')) ]), unsup=dict( type='CocoDataset', ann_file= 'data/coco/annotations/semi_supervised/instances_train2017.2@10-unlabeled.json', img_prefix='data/coco/train2017/', pipeline=[ dict(type='LoadImageFromFile'), dict(type='PseudoSamples', with_bbox=True), dict( type='MultiBranch', unsup_student=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='ShuffledSequential', transforms=[ dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]), dict( type='OneOf', transforms=[{ 'type': 'RandTranslate', 'x': (-0.1, 0.1) }, { 'type': 'RandTranslate', 'y': (-0.1, 0.1) }, { 'type': 'RandRotate', 'angle': (-30, 30) }, [{ 'type': 'RandShear', 'x': (-30, 30) }, { 'type': 'RandShear', 'y': (-30, 30) }]]) ]), dict( type='RandErase', n_iterations=(1, 5), size=[0, 0.2], squared=True) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_student'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ], unsup_teacher=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_teacher'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ]) ], filter_empty_gt=False)), val=dict( type='CocoDataset', ann_file='data/coco/annotations/instances_val2017.json', img_prefix='data/coco/val2017/', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ]), test=dict( type='CocoDataset', ann_file='data/coco/annotations/instances_val2017.json', img_prefix='data/coco/val2017/', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ]), sampler=dict( train=dict( type='SemiBalanceSampler', sample_ratio=[1, 4], by_prob=True, epoch_length=7330))) evaluation = dict(interval=4000, metric='bbox', type='SubModulesDistEvalHook') optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(grad_clip=None) lr_config = dict( policy='step', warmup='linear', warmup_iters=500, warmup_ratio=0.001, step=[120000, 160000]) runner = dict(type='IterBasedRunner', max_iters=180000) checkpoint_config = dict(interval=4000, by_epoch=False, max_keep_ckpts=20) log_config = dict( interval=50, hooks=[ dict(type='TextLoggerHook'), dict( type='WandbLoggerHook', init_kwargs=dict( project='pre_release', name='soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k', config=dict( fold=2, percent=10, work_dirs='work_dirs/${cfg_name}/${percent}/${fold}', total_step=180000)), by_epoch=False) ]) custom_hooks = [ dict(type='NumClassCheckHook'), dict(type='WeightSummary'), dict(type='MeanTeacher', momentum=0.999, interval=1, warm_up=0) ] dist_params = dict(backend='nccl') log_level = 'INFO' load_from = None resume_from = None workflow = [('train', 1)] mmdet_base = '../../thirdparty/mmdetection/configs/base' strong_pipeline = [ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='ShuffledSequential', transforms=[ dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]), dict( type='OneOf', transforms=[{ 'type': 'RandTranslate', 'x': (-0.1, 0.1) }, { 'type': 'RandTranslate', 'y': (-0.1, 0.1) }, { 'type': 'RandRotate', 'angle': (-30, 30) }, [{ 'type': 'RandShear', 'x': (-30, 30) }, { 'type': 'RandShear', 'y': (-30, 30) }]]) ]), dict( type='RandErase', n_iterations=(1, 5), size=[0, 0.2], squared=True) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_student'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ] weak_pipeline = [ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_teacher'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ] unsup_pipeline = [ dict(type='LoadImageFromFile'), dict(type='PseudoSamples', with_bbox=True), dict( type='MultiBranch', unsup_student=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='ShuffledSequential', transforms=[ dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]), dict( type='OneOf', transforms=[{ 'type': 'RandTranslate', 'x': (-0.1, 0.1) }, { 'type': 'RandTranslate', 'y': (-0.1, 0.1) }, { 'type': 'RandRotate', 'angle': (-30, 30) }, [{ 'type': 'RandShear', 'x': (-30, 30) }, { 'type': 'RandShear', 'y': (-30, 30) }]]) ]), dict( type='RandErase', n_iterations=(1, 5), size=[0, 0.2], squared=True) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_student'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ], unsup_teacher=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_teacher'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ]) ] fp16 = dict(loss_scale='dynamic') fold = 2 percent = 10 work_dir = 'work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k/10/2' cfg_name = 'soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k' gpu_ids = range(0, 1)

/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/mmdet/core/anchor/builder.py:17: UserWarning: build_anchor_generator would be deprecated soon, please use build_prior_generator 'build_anchor_generator would be deprecated soon, please use ' 2021-12-28 01:59:17,819 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe 2021-12-28 01:59:17,819 - mmcv - INFO - Use load_from_openmmlab loader 2021-12-28 01:59:17,876 - mmcv - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1.bias

2021-12-28 01:59:18,055 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe 2021-12-28 01:59:18,056 - mmcv - INFO - Use load_from_openmmlab loader 2021-12-28 01:59:18,103 - mmcv - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1.bias

loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.12s) creating index... index created! fatal: Not a git repository (or any parent up to mount point /media/smn/Elements SE) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). loading annotations into memory... Done (t=0.02s) creating index... index created! 2021-12-28 01:59:20,405 - mmdet.ssod - INFO - Start running, host: smn@smn-Z87X-HD3, work_dir: /media/smn/Elements SE/SoftTeacher-main/work_dirs/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k/10/2 2021-12-28 01:59:20,406 - mmdet.ssod - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) StepLrUpdaterHook
(ABOVE_NORMAL) Fp16OptimizerHook
(NORMAL ) CheckpointHook
(NORMAL ) WeightSummary
(NORMAL ) MeanTeacher
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook

before_train_epoch: (VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) IterTimerHook
(NORMAL ) NumClassCheckHook
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook

before_train_iter: (VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) IterTimerHook
(NORMAL ) MeanTeacher
(80 ) SubModulesDistEvalHook

after_train_iter: (ABOVE_NORMAL) Fp16OptimizerHook
(NORMAL ) CheckpointHook
(NORMAL ) IterTimerHook
(NORMAL ) MeanTeacher
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook

after_train_epoch: (NORMAL ) CheckpointHook
(80 ) SubModulesDistEvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook

before_val_epoch: (NORMAL ) IterTimerHook
(NORMAL ) NumClassCheckHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook

before_val_iter: (NORMAL ) IterTimerHook

after_val_iter: (NORMAL ) IterTimerHook

after_val_epoch: (VERY_LOW ) TextLoggerHook
(VERY_LOW ) WandbLoggerHook

after_run: (VERY_LOW ) WandbLoggerHook

Traceback (most recent call last): File "tools/train.py", line 198, in main() File "tools/train.py", line 193, in main meta=meta, File "/media/smn/Elements SE/SoftTeacher-main/ssod/apis/train.py", line 206, in train_detector runner.run(data_loaders, cfg.workflow) File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 117, in run iter_loaders = [IterLoader(x) for x in data_loaders] File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 117, in iter_loaders = [IterLoader(x) for x in data_loaders] File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 23, in init self.iter_loader = iter(self._dataloader) File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 359, in iter return self._get_iterator() File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 305, in _get_iterator return _MultiProcessingDataLoaderIter(self) File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 944, in init self._reset(loader, first_iter=True) File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 975, in _reset self._try_put_index() File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1209, in _try_put_index index = self._next_index() File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 512, in _next_index return next(self._sampler_iter) # may raise StopIteration File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 226, in iter for idx in self.sampler: File "/media/smn/Elements SE/SoftTeacher-main/ssod/datasets/samplers/semi_sampler.py", line 187, in iter assert len(indices) == len(self) AssertionError

wandb: Waiting for W&B process to finish, PID 26081 wandb: Program failed with code 1. Press ctrl-c to abort syncing. wandb:
wandb: Find user logs for this run at: /media/smn/Elements SE/SoftTeacher-main/wandb/run-20211228_015920-4c0xw02u/logs/debug.log wandb: Find internal logs for this run at: /media/smn/Elements SE/SoftTeacher-main/wandb/run-20211228_015920-4c0xw02u/logs/debug-internal.log wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: wandb: Synced soft_teacher_faster_rcnn_r50_caffe_fpn_coco_180k: https://wandb.ai/shimengnan/pre_release/runs/4c0xw02u ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 25824) of binary: /home/smn/anaconda3/envs/Soft/bin/python ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 3/3 attempts left; will restart worker group INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=1 master_addr=127.0.0.1 master_port=29500 group_rank=0 group_world_size=1 local_ranks=[0] role_ranks=[0] global_ranks=[0] role_world_sizes=[1] globalworld

smntjugithub commented 2 years ago

Traceback (most recent call last): File "tools/train.py", line 198, in main() File "tools/train.py", line 193, in main meta=meta, File "/media/smn/Elements SE/SoftTeacher-main/ssod/apis/train.py", line 206, in train_detector runner.run(data_loaders, cfg.workflow) File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 117, in run iter_loaders = [IterLoader(x) for x in data_loaders] File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 117, in iter_loaders = [IterLoader(x) for x in data_loaders] File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 23, in init self.iter_loader = iter(self._dataloader) File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 359, in iter return self._get_iterator() File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 305, in _get_iterator return _MultiProcessingDataLoaderIter(self) File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 944, in init self._reset(loader, first_iter=True) File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 975, in _reset self._try_put_index() File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1209, in _try_put_index index = self._next_index() File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 512, in _next_index return next(self._sampler_iter) # may raise StopIteration File "/home/smn/anaconda3/envs/Soft/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 226, in iter for idx in self.sampler: File "/media/smn/Elements SE/SoftTeacher-main/ssod/datasets/samplers/semi_sampler.py", line 187, in iter assert len(indices) == len(self) AssertionError

GoodNightIsabelle commented 2 years ago

请问还有后续吗？我也遇到了类似的问题

microsoft / SoftTeacher

assert len(cls_scores) == len(score_factors) AssertionError #83

TorchVision: 0.10.0+cu102 OpenCV: 4.5.4 MMCV: 1.3.9 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 10.2 MMDetection: 2.17.0+