[Bug] use dist_test.sh to eval the same model,get different results every time

[Ryan111111]

Branch

master branch (0.24 or other 0.x version)

Describe the bug

use dist_test.sh to eval the same model,get different results every time！！！

./tools/dist_test.sh configs/violence/violence_cls35_resnet50.py /workspace/xdcv-classification/work_dirs/violence_cls35/epoch_2.pth 4

Problems is data distribution for distributed testing.

Environment

./tools/dist_test.sh configs/violence/violence_cls35_resnet50.py /workspace/xdcv-classification/work_dirs/violence_cls35/epoch_2.pth 4

Other information

1.no modifications on the code or config 2.Problems is data distribution for distributed testing.

[Ezra-Yu]

Can you show your config? Is there any randomness when testing？

[Ryan111111]

1.No changes in code 2.configs is standard

_base_ = [
    '../_base_/default_runtime.py'
]

model = dict(
    type='ImageClassifier',
    backbone=dict(
        type='ResNet',
        depth=50,
        num_stages=4,
        out_indices=(3, ),
        style='pytorch',
        init_cfg=dict(
            type='Pretrained',
            checkpoint='/workspace/xdcv-classification/work_dirs/epoch_140.pth',
            prefix='backbone',
        )),
    neck=dict(type='GlobalAveragePooling'),
    head=dict(
        type='LinearClsHead',
        num_classes=35,
        in_channels=2048,
        loss=dict(type='CrossEntropyLoss', loss_weight=1.0),
        topk=(1, 5),
    ))

img_scale = (288, 288)
dataset_type = 'FileDataset'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='Pad', pad_to_square=True),
    dict(type ='Resize', size=img_scale, backend='pillow'),
    dict(type='RandomFlip', flip_prob=0.5, direction='horizontal'),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='ToTensor', keys=['gt_label']),
    dict(type='Collect', keys=['img', 'gt_label'])
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='Pad', pad_to_square=True),
    dict(type ='Resize', size=img_scale, backend='pillow'),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='Collect', keys=['img'])
]
data = dict(
    samples_per_gpu=64,
    workers_per_gpu=2,
    train=dict(
        type='FileDataset',
        data_prefix='',
        ann_file='/workspace/xdcv-classification/configs/violence/violence_cls35_train.txt',
        pipeline=train_pipeline,
        ),
    val=dict(
        type='FileDataset',
        data_prefix='',
        ann_file='/workspace/xdcv-classification/configs/violence/violence_cls35_test.txt',
        pipeline=test_pipeline,
        ),
    test=dict(
        type='FileDataset',
        data_prefix='',
        ann_file='/workspace/xdcv-classification/configs/violence/violence_cls35_test.txt',
        pipeline=test_pipeline,
        classes='/workspace/xdcv-classification/configs/label_info/violence_cls35.json'))

evaluation = dict(interval=1, metric='accuracy')

work_dir = './work_dirs/violence_cls35'
# optimizer
optimizer = dict(type='SGD', lr=0.03, momentum=0.9, weight_decay=5e-5)
optimizer_config = dict(grad_clip=None)
# learning policy
lr_config = dict(
    policy='CosineAnnealing',
    min_lr=0,
    warmup='linear',
    warmup_iters=5,
    warmup_ratio=0.1,
    warmup_by_epoch=True)
runner = dict(type='EpochBasedRunner', max_epochs=100)

[Ryan111111]

but i use single test，python ./tools/test.py ....,get same results every time. only use dist_test.sh will get different results every time. you can run your dist_test.sh to Check if the same problem exists.