./tools/dist_test.sh evaluation value is all zero

[fathur-rs]

I am performing object detection using Cascade R-CNN. I have already trained the model, but when I try to evaluate it using test data, the result is consistently zero.

My code: !./tools/dist_test.sh configs/cascade_rcnn/cascade-rcnn_x101-64x4d_fpn_1x_coco.py checkpoints/cascade_rcnn_x101_64x4d_fpn_1x_coco_20200515_075702-43ce6a30.pth 1

My configs:

config_resep_cascade_rcnn = """
# Inherit and overwrite part of the config based on this config
_base_ = './cascade-rcnn_r50_fpn_1x_coco.py'

data_root = 'data/RESEP-2-P-S-10/' # dataset root

train_batch_size_per_gpu = 4
train_num_workers = 2

max_epochs = 50
stage2_num_epochs = 1
base_lr = 0.00008

metainfo = {
    'classes': ('PRESCRIPTIO', 'SIGNATURA', ),
    'palette': [
        (190, 77, 37),
        (37, 150, 190)
    ]
}

# dataloader
train_dataloader = dict(
    batch_size=train_batch_size_per_gpu,
    num_workers=train_num_workers,
    dataset=dict(
        data_root=data_root,
        metainfo=metainfo,
        data_prefix=dict(img='train/'),
        ann_file='train/_annotations.coco.json'))

val_dataloader = dict(
    dataset=dict(
        data_root=data_root,
        metainfo=metainfo,
        data_prefix=dict(img='valid/'),
        ann_file='valid/_annotations.coco.json'))

test_dataloader = dict(
    dataset=dict(
        data_root=data_root,
        metainfo=metainfo,
        data_prefix=dict(img='test/'),
        ann_file='test/_annotations.coco.json'))

val_evaluator = dict(ann_file=data_root + 'valid/_annotations.coco.json')

test_evaluator = dict(ann_file=data_root + 'test/_annotations.coco.json')

#######################################
classes = ('prescriptio', 'signatura', )
n_classes = len(classes)
model = dict(
    type='CascadeRCNN',
    backbone=dict(
        type='ResNeXt',
        depth=101,
        groups=64,
        base_width=4,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=1,
        norm_cfg=dict(type='BN', requires_grad=True),
        style='pytorch',
        init_cfg=dict(
            type='Pretrained', checkpoint='open-mmlab://resnext101_64x4d')),
    roi_head=dict(
        bbox_head=[
            dict(
                type='Shared2FCBBoxHead',
                in_channels=256,
                fc_out_channels=1024,
                roi_feat_size=7,
                num_classes=n_classes,
                bbox_coder=dict(
                    type='DeltaXYWHBBoxCoder',
                    target_means=[0.0, 0.0, 0.0, 0.0],
                    target_stds=[0.1, 0.1, 0.2, 0.2]),
                reg_class_agnostic=True,
                loss_cls=dict(
                    type='CrossEntropyLoss',
                    use_sigmoid=False,
                    loss_weight=1.0),
                loss_bbox=dict(type='SmoothL1Loss', beta=1.0,
                               loss_weight=1.0)),
            dict(
                type='Shared2FCBBoxHead',
                in_channels=256,
                fc_out_channels=1024,
                roi_feat_size=7,
                num_classes=n_classes,
                bbox_coder=dict(
                    type='DeltaXYWHBBoxCoder',
                    target_means=[0.0, 0.0, 0.0, 0.0],
                    target_stds=[0.05, 0.05, 0.1, 0.1]),
                reg_class_agnostic=True,
                loss_cls=dict(
                    type='CrossEntropyLoss',
                    use_sigmoid=False,
                    loss_weight=1.0),
                loss_bbox=dict(type='SmoothL1Loss', beta=1.0,
                               loss_weight=1.0)),
            dict(
                type='Shared2FCBBoxHead',
                in_channels=256,
                fc_out_channels=1024,
                roi_feat_size=7,
                num_classes=n_classes,
                bbox_coder=dict(
                    type='DeltaXYWHBBoxCoder',
                    target_means=[0.0, 0.0, 0.0, 0.0],
                    target_stds=[0.033, 0.033, 0.067, 0.067]),
                reg_class_agnostic=True,
                loss_cls=dict(
                    type='CrossEntropyLoss',
                    use_sigmoid=False,
                    loss_weight=1.0),
                loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0))
        ])
      )

#######################################

# learning rate
param_scheduler = [
    dict(
        type='LinearLR',
        start_factor=1.0e-5,
        by_epoch=False,
        begin=0,
        end=10),
    dict(
        # use cosine lr from 10 to 20 epoch
        type='CosineAnnealingLR',
        eta_min=base_lr * 0.05,
        begin=max_epochs // 2,
        end=max_epochs,
        T_max=max_epochs // 2,
        by_epoch=True,
        convert_to_iter_based=True),
]

train_pipeline_stage2 = [
    dict(type='LoadImageFromFile', backend_args=None),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='Resize', scale=(640, 800), keep_ratio=False),
    dict(type='PackDetInputs')
]

# optimizer
optim_wrapper = dict(
    _delete_=True,
    type='OptimWrapper',
    optimizer=dict(type='AdamW', lr=base_lr, weight_decay=0.05),
    paramwise_cfg=dict(
        norm_decay_mult=0, bias_decay_mult=0, bypass_duplicate=True))

default_hooks = dict(
    checkpoint=dict(
        interval=5,
        max_keep_ckpts=2,  # only keep latest 2 checkpoints
        save_best='auto'
    ),
    logger=dict(type='LoggerHook', interval=5))

custom_hooks = [
    dict(
        type='PipelineSwitchHook',
        switch_epoch=max_epochs - stage2_num_epochs,
        switch_pipeline=train_pipeline_stage2)
]

# load COCO pre-trained weight
load_from = './checkpoints/cascade_rcnn_x101_64x4d_fpn_1x_coco_20200515_075702-43ce6a30.pth'

train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=max_epochs, val_interval=1)
visualizer = dict(vis_backends=[dict(type='LocalVisBackend'),dict(type='TensorboardVisBackend')])
"""

with open('./configs/cascade_rcnn/cascade-rcnn_x101-64x4d_fpn_1x_coco.py', 'w') as f:
    f.write(config_resep_cascade_rcnn)

Output:

Accumulating evaluation results... DONE (t=0.29s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.011 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.011 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.011 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.017 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.002 05/05 10:00:09 - mmengine - INFO - bbox_mAP_copypaste: 0.000 0.000 0.000 0.000 0.000 0.000 05/05 10:00:09 - mmengine - INFO - Results has been saved to results.pkl. 05/05 10:00:09 - mmengine - INFO - Epoch(test) [68/68] coco/bbox_mAP: 0.0000 coco/bbox_mAP_50: 0.0000 coco/bbox_mAP_75: 0.0000 coco/bbox_mAP_s: 0.0000 coco/bbox_mAP_m: 0.0000 coco/bbox_mAP_l: 0.0000 data_time: 0.0036 time: 0.0511

any suggestions is really helpfull

[SongPool]

so how you slove this question?

[Mukil07]

Im also getting zero mAP for my custom config file. I wanted to reduce the size to half of the original size of the image. So i added this line in the test_pipeline

dict(type="Resize, scale=0.5, keep_ratio=True)

And it starts giving zero mAP after evaluation. Any suggestions would be helpful.

[dahhei]

During your training process, are you also getting 0s for your mAP scores? That's what's happening to me during my actual training

[fathur-rs]

so how you slove this question?

I solved this by changing my activation function to ReLU