chenzx921020 commented 2 years ago

Describe the bug

A clear and concise description of what the bug is.

When I running detnas by step 3 : use evolution search do training in a det supernet ，it appeared a distributed trainning problem

To Reproduce

The command you executed.

Traceback (most recent call last):
  File "./tools/mmdet/search_mmdet.py", line 197, in <module>
    main()
  File "./tools/mmdet/search_mmdet.py", line 193, in main
    searcher.search()
  File "/home/chenzhixuan/project/mmrazor/mmrazor/core/searcher/evolution_search.py", line 145, in search
    broadcast_candidate_pool)
  File "/home/chenzhixuan/project/mmrazor/mmrazor/core/utils/broadcast.py", line 38, in broadcast_object_list
    dist.broadcast(dir_tensor, src)
  File "/home/chenzhixuan/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1075, in broadcast
    default_pg = _get_default_group()
  File "/home/chenzhixuan/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 358, in _get_default_group
    raise RuntimeError("Default process group has not been initialized, "
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

humu789 commented 2 years ago

You can try to add the parameter: --launcher in your running command to perform init_dist

humu789 commented 2 years ago

Can you tell me how much GPUs to use in your experiments

xingyueye commented 2 years ago

@humu789 Hi, I encountered this problem as well with single GPU. It seems because that torch.distributed is called when not been initialized in "mmrazor/mmrazor/core/utils/broadcast.py" dist.broadcast(dir_tensor, src) And the initialization process is ignored in "./tools/mmdet/search_mmdet.py"(and other search scripts)

    if args.launcher == 'none':
        distributed = False
    else:
        distributed = True
        init_dist(args.launcher, **cfg.dist_params)

humu789 commented 2 years ago

@xingyueye Yes, we are trying to solve it better. You can quick run your experiment by referring to it.

You can try to add the parameter: --launcher in your running command to perform init_dist

daitranskku commented 2 years ago

I had the same issue with two TeslaV100. I tried —launch, but I still get the same error. Is anyone resolving these issues? Thanks

humu789 commented 2 years ago

@daitranskku Can you provide details of the command to perform your experiment?

daitranskku commented 2 years ago

@humu789 I tried step 3 with the command below. Also, I added --launcher pytorch, and it produced "KeyError: 'RANK"

Step 3

python tools/mmdet/search_mmdet.py \
configs/step3.py \
work_dirs/step2/latest.pth

022-02-16 09:25:59,017 - mmdet - INFO - Environment info:

sys.platform: linux Python: 3.7.11 (default, Jul 27 2021, 14:32:16) [GCC 7.5.0] CUDA available: True GPU 0,1: NVIDIA Tesla V100-PCIE-32GB CUDA_HOME: /usr/local/cuda NVCC: Build cuda_11.3.r11.3/compiler.29920130_0 GCC: gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4) PyTorch: 1.10.1+cu113 PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX512
CUDA Runtime 11.3
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
CuDNN 8.2
Magma 2.5.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, TorchVision: 0.11.2+cu113 OpenCV: 4.5.5 MMCV: 1.4.3 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 11.3 MMDetection: 2.20.0+0dd407a

2022-02-16 09:25:59,018 - mmdet - INFO - Distributed training: False 2022-02-16 09:26:00,414 - mmdet - INFO - Config: dataset_type = 'COCODataset' data_root = 'data/coco/' img_norm_cfg = dict( mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True) train_pipeline = [ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict(type='Resize', img_scale=(1333, 800), keep_ratio=True), dict(type='RandomFlip', flip_ratio=0.5), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ] data = dict( samples_per_gpu=5, workers_per_gpu=10, train=dict( type='CocoDataset', ann_file= '.json', img_prefix= '', pipeline=[ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict(type='Resize', img_scale=(1333, 800), keep_ratio=True), dict(type='RandomFlip', flip_ratio=0.5), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']) ], classes=('15', '10', '16', '18', '13', '09', '12', '17', '11', '19', '14')), val=dict( type='CocoDataset', ann_file= '.json', img_prefix= '/', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ], classes=('15', '10', '16', '18', '13', '09', '12', '17', '11', '19', '14')), test=dict( type='CocoDataset', ann_file= '.json', img_prefix= '/', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ], classes=('15', '10', '16', '18', '13', '09', '12', '17', '11', '19', '14'), test_mode=True)) evaluation = dict(interval=1, metric='bbox') optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(grad_clip=None) lr_config = dict( policy='step', warmup='linear', warmup_iters=500, warmup_ratio=0.001, step=[8, 11]) runner = dict(type='EpochBasedRunner', max_epochs=12) checkpoint_config = dict(interval=1) log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')]) custom_hooks = [dict(type='NumClassCheckHook')] dist_params = dict(backend='nccl') log_level = 'INFO' load_from = None resume_from = None workflow = [('train', 1)] norm_cfg = dict(type='BN', requires_grad=True) model = dict( type='mmdet.FasterRCNN', backbone=dict( type='mmcls.SearchableShuffleNetV2', norm_cfg=dict(type='BN', requires_grad=True), out_indices=(0, 1, 2, 3), widen_factor=1.0, with_last_layer=False), neck=dict( type='FPN', norm_cfg=dict(type='BN', requires_grad=True), in_channels=[64, 160, 320, 640], out_channels=256, num_outs=5), rpn_head=dict( type='RPNHead', in_channels=256, feat_channels=256, anchor_generator=dict( type='AnchorGenerator', scales=[8], ratios=[0.5, 1.0, 2.0], strides=[4, 8, 16, 32, 64]), bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[1.0, 1.0, 1.0, 1.0]), loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0)), roi_head=dict( type='StandardRoIHead', bbox_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0), out_channels=256, featmap_strides=[4, 8, 16, 32]), bbox_head=dict( type='Shared4Conv1FCBBoxHead', norm_cfg=dict(type='BN', requires_grad=True), in_channels=256, fc_out_channels=1024, roi_feat_size=7, num_classes=80, bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[0.1, 0.1, 0.2, 0.2]), reg_class_agnostic=False, loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0))), train_cfg=dict( rpn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.7, neg_iou_thr=0.3, min_pos_iou=0.3, match_low_quality=True, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=256, pos_fraction=0.5, neg_pos_ub=-1, add_gt_as_proposals=False), allowed_border=-1, pos_weight=-1, debug=False), rpn_proposal=dict( nms_pre=2000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.5, neg_iou_thr=0.5, min_pos_iou=0.5, match_low_quality=False, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=512, pos_fraction=0.25, neg_pos_ub=-1, add_gt_as_proposals=True), pos_weight=-1, debug=False)), test_cfg=dict( rpn=dict( nms_pre=1000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( score_thr=0.05, nms=dict(type='nms', iou_threshold=0.5), max_per_img=100))) mutator = dict( type='OneShotMutator', placeholder_mapping=dict( all_blocks=dict( type='OneShotOP', choices=dict( shuffle_3x3=dict( type='ShuffleBlock', norm_cfg=dict(type='BN', requires_grad=True), kernel_size=3), shuffle_5x5=dict( type='ShuffleBlock', norm_cfg=dict(type='BN', requires_grad=True), kernel_size=5), shuffle_7x7=dict( type='ShuffleBlock', norm_cfg=dict(type='BN', requires_grad=True), kernel_size=7), shuffle_xception=dict( type='ShuffleXception', norm_cfg=dict(type='BN', requires_grad=True)))))) algorithm = dict( type='DetNAS', architecture=dict( type='MMDetArchitecture', model=dict( type='mmdet.FasterRCNN', backbone=dict( type='mmcls.SearchableShuffleNetV2', norm_cfg=dict(type='BN', requires_grad=True), out_indices=(0, 1, 2, 3), widen_factor=1.0, with_last_layer=False), neck=dict( type='FPN', norm_cfg=dict(type='BN', requires_grad=True), in_channels=[64, 160, 320, 640], out_channels=256, num_outs=5), rpn_head=dict( type='RPNHead', in_channels=256, feat_channels=256, anchor_generator=dict( type='AnchorGenerator', scales=[8], ratios=[0.5, 1.0, 2.0], strides=[4, 8, 16, 32, 64]), bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[1.0, 1.0, 1.0, 1.0]), loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0)), roi_head=dict( type='StandardRoIHead', bbox_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict( type='RoIAlign', output_size=7, sampling_ratio=0), out_channels=256, featmap_strides=[4, 8, 16, 32]), bbox_head=dict( type='Shared4Conv1FCBBoxHead', norm_cfg=dict(type='BN', requires_grad=True), in_channels=256, fc_out_channels=1024, roi_feat_size=7, num_classes=80, bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[0.1, 0.1, 0.2, 0.2]), reg_class_agnostic=False, loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0))), train_cfg=dict( rpn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.7, neg_iou_thr=0.3, min_pos_iou=0.3, match_low_quality=True, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=256, pos_fraction=0.5, neg_pos_ub=-1, add_gt_as_proposals=False), allowed_border=-1, pos_weight=-1, debug=False), rpn_proposal=dict( nms_pre=2000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.5, neg_iou_thr=0.5, min_pos_iou=0.5, match_low_quality=False, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=512, pos_fraction=0.25, neg_pos_ub=-1, add_gt_as_proposals=True), pos_weight=-1, debug=False)), test_cfg=dict( rpn=dict( nms_pre=1000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( score_thr=0.05, nms=dict(type='nms', iou_threshold=0.5), max_per_img=100)), pretrained=None)), mutator=dict( type='OneShotMutator', placeholder_mapping=dict( all_blocks=dict( type='OneShotOP', choices=dict( shuffle_3x3=dict( type='ShuffleBlock', norm_cfg=dict(type='BN', requires_grad=True), kernel_size=3), shuffle_5x5=dict( type='ShuffleBlock', norm_cfg=dict(type='BN', requires_grad=True), kernel_size=5), shuffle_7x7=dict( type='ShuffleBlock', norm_cfg=dict(type='BN', requires_grad=True), kernel_size=7), shuffle_xception=dict( type='ShuffleXception', norm_cfg=dict(type='BN', requires_grad=True)))))), pruner=None, distiller=None, retraining=False) find_unused_parameters = True classes = ('15', '10', '16', '18', '13', '09', '12', '17', '11', '19', '14') searcher = dict( type='EvolutionSearcher', metrics='bbox', score_key='bbox_mAP', constraints=dict(flops=300000000.0), candidate_pool_size=50, candidate_top_k=10, max_epoch=20, num_mutation=20, num_crossover=20) work_dir = 'work_dirs/step3'

2022-02-16 09:26:10,328 - mmdet - INFO - build search... 2022-02-16 09:26:10,329 - mmdet - INFO - start search... 2022-02-16 09:26:10,329 - mmdet - INFO - Experiment setting: 2022-02-16 09:26:10,329 - mmdet - INFO - candidate_pool_size: 50 2022-02-16 09:26:10,329 - mmdet - INFO - candidate_top_k: 10 2022-02-16 09:26:10,329 - mmdet - INFO - num_crossover: 20 2022-02-16 09:26:10,329 - mmdet - INFO - num_mutation: 20 2022-02-16 09:26:10,329 - mmdet - INFO - mutate_prob: 0.1 2022-02-16 09:26:10,329 - mmdet - INFO - max_epoch: 20 2022-02-16 09:26:10,329 - mmdet - INFO - score_key: bbox_mAP 2022-02-16 09:26:10,329 - mmdet - INFO - constraints: {'flops': 300000000.0} 2022-02-16 09:26:10,329 - mmdet - INFO - ####################################################################################################

humu789 commented 2 years ago

@daitranskku distributed is False in your log, which means that init_dist is not been executed in search_mmdet.py. You can try to check whther --launcher is used correctly.

daitranskku commented 2 years ago

Thanks @humu789 . I added -m torch.distributed.launch and its working.

open-mmlab / mmrazor

Default process group has not been initialized, please make sure to call init_process_group #42

Describe the bug

To Reproduce

Step 3

022-02-16 09:25:59,017 - mmdet - INFO - Environment info: