search

open-mmlab / mmdetection

OpenMMLab Detection Toolbox and Benchmark
https://mmdetection.readthedocs.io
Apache License 2.0
29.55k stars 9.46k forks source link

Slow instance segmentation inference speed on RTMDet #10160

Open pd162 opened 1 year ago

pd162 commented 1 year ago

Checklist

  1. I have searched related issues but cannot get the expected help.
  2. The issue has not been fixed in the latest version.

Describe the issue

I tried to use RTMDet with segmentation head in scene text detection. Except a replacement of dataset, I didn't change any config. However, when I have trained a model and tried to analysis its inference speed, I found the speed of RTMDet only less than 10 fps, which is much lower than the report.

Reproduction

  1. What command or script did you run?
python tools.analysis_tools/benchmark.py configs/rtmdet/rtmdet_text_ic15.py --checkpoint work_dirs/rtmdet_text_ic15/epoch_100.pth --task inference
  1. What config dir you run?
_base_ = [
    '../_base_/default_runtime.py',
    '../_base_/schedules/schedule_1x.py',
    '../_base_/datasets/text_detection.py',
    './rtmdet_tta.py'
]

model = dict(
    type='RTMDet',
    data_preprocessor=dict(
        type='DetDataPreprocessor',
        mean=[103.53, 116.28, 123.675],
        std=[57.375, 57.12, 58.395],
        bgr_to_rgb=False,
        batch_augments=None),
    backbone=dict(
        type='CSPNeXt',
        arch='P5',
        expand_ratio=0.5,
        deepen_factor=1,
        widen_factor=1,
        channel_attention=True,
        norm_cfg=dict(type='BN'),
        act_cfg=dict(type='SiLU', inplace=True)),
    neck=dict(
        type='CSPNeXtPAFPN',
        in_channels=[256, 512, 1024],
        out_channels=256,
        num_csp_blocks=3,
        expand_ratio=0.5,
        norm_cfg=dict(type='SyncBN'),
        act_cfg=dict(type='SiLU', inplace=True)),
    bbox_head=dict(
        type='RTMDetInsSepBNHead',
        num_classes=80,
        in_channels=256,
        stacked_convs=2,
        share_conv=True,
        pred_kernel_size=1,
        feat_channels=256,
        act_cfg=dict(type='SiLU', inplace=True),
        norm_cfg=dict(type='BN', requires_grad=True),
        anchor_generator=dict(
            type='MlvlPointGenerator', offset=0, strides=[8, 16, 32]),
        bbox_coder=dict(type='DistancePointBBoxCoder'),
        loss_cls=dict(
            type='QualityFocalLoss',
            use_sigmoid=True,
            beta=2.0,
            loss_weight=1.0),
        loss_bbox=dict(type='GIoULoss', loss_weight=2.0),
        loss_mask=dict(
            type='DiceLoss', loss_weight=2.0, eps=5e-6, reduction='mean')),
    test_cfg=dict(
        nms_pre=1000,
        min_bbox_size=0,
        score_thr=0.05,
        nms=dict(type='nms', iou_threshold=0.6),
        max_per_img=100,
        mask_thr_binary=0.5),
    train_cfg=dict(
        assigner=dict(type='DynamicSoftLabelAssigner', topk=13),
        allowed_border=-1,
        pos_weight=-1,
        debug=False),
)

train_pipeline = [
    dict(type='LoadImageFromFile', backend_args={{_base_.backend_args}}),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='CachedMosaic', img_scale=(640, 640), pad_val=114.0),
    dict(
        type='RandomResize',
        scale=(1280, 1280),
        ratio_range=(0.1, 2.0),
        keep_ratio=True),
    dict(type='RandomCrop', crop_size=(640, 640)),
    dict(type='YOLOXHSVRandomAug'),
    dict(type='RandomFlip', prob=0.5),
    dict(type='Pad', size=(640, 640), pad_val=dict(img=(114, 114, 114))),
    dict(
        type='CachedMixUp',
        img_scale=(640, 640),
        ratio_range=(1.0, 1.0),
        max_cached_images=20,
        pad_val=(114, 114, 114)),
    dict(type='PackDetInputs')
]

train_pipeline_stage2 = [
    dict(type='LoadImageFromFile', backend_args={{_base_.backend_args}}),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(
        type='RandomResize',
        scale=(640, 640),
        ratio_range=(0.1, 2.0),
        keep_ratio=True),
    dict(type='RandomCrop', crop_size=(640, 640)),
    dict(type='YOLOXHSVRandomAug'),
    dict(type='RandomFlip', prob=0.5),
    dict(type='Pad', size=(640, 640), pad_val=dict(img=(114, 114, 114))),
    dict(type='PackDetInputs')
]

test_pipeline = [
    dict(type='LoadImageFromFile', backend_args={{_base_.backend_args}}),
    dict(type='Resize', scale=(640, 640), keep_ratio=True),
    dict(type='Pad', size=(640, 640), pad_val=dict(img=(114, 114, 114))),
    dict(
        type='PackDetInputs',
        meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
                   'scale_factor'))
]

train_dataloader = dict(
    batch_size=64,
    num_workers=10,
    batch_sampler=None,
    pin_memory=True,
    dataset=dict(pipeline=train_pipeline))
val_dataloader = dict(
    batch_size=5, num_workers=10, dataset=dict(pipeline=test_pipeline))
test_dataloader = val_dataloader

max_epochs = 100
stage2_num_epochs = 20
base_lr = 0.004
interval = 10

train_cfg = dict(
    max_epochs=max_epochs,
    val_interval=interval,
    dynamic_intervals=[(max_epochs - stage2_num_epochs, 1)])

val_evaluator = dict(proposal_nums=(100, 1, 10))
test_evaluator = val_evaluator

# optimizer
optim_wrapper = dict(
    # _delete_=True,
    type='OptimWrapper',
    optimizer=dict(type='AdamW', lr=base_lr, weight_decay=0.05),
    paramwise_cfg=dict(
        norm_decay_mult=0, bias_decay_mult=0, bypass_duplicate=True))

# learning rate
param_scheduler = [
    dict(
        type='LinearLR',
        start_factor=1.0e-5,
        by_epoch=False,
        begin=0,
        end=1000),
    dict(
        # use cosine lr from 150 to 300 epoch
        type='CosineAnnealingLR',
        eta_min=base_lr * 0.05,
        begin=max_epochs // 2,
        end=max_epochs,
        T_max=max_epochs // 2,
        by_epoch=True,
        convert_to_iter_based=True),
]

# hooks
default_hooks = dict(
    checkpoint=dict(
        interval=interval,
        max_keep_ckpts=3  # only keep latest 3 checkpoints
    ))
custom_hooks = [
    dict(
        type='EMAHook',
        ema_type='ExpMomentumEMA',
        momentum=0.0002,
        update_buffers=True,
        priority=49),
    dict(
        type='PipelineSwitchHook',
        switch_epoch=max_epochs - stage2_num_epochs,
        switch_pipeline=train_pipeline_stage2)
]

train_pipeline = [
    dict(type='LoadImageFromFile', backend_args={{_base_.backend_args}}),
    dict(
        type='LoadAnnotations',
        with_bbox=True,
        with_mask=True,
        poly2mask=False),
    dict(type='CachedMosaic', img_scale=(640, 640), pad_val=114.0),
    dict(
        type='RandomResize',
        scale=(1280, 1280),
        ratio_range=(0.1, 2.0),
        keep_ratio=True),
    dict(
        type='RandomCrop',
        crop_size=(640, 640),
        recompute_bbox=True,
        allow_negative_crop=True),
    dict(type='YOLOXHSVRandomAug'),
    dict(type='RandomFlip', prob=0.5),
    dict(type='Pad', size=(640, 640), pad_val=dict(img=(114, 114, 114))),
    dict(
        type='CachedMixUp',
        img_scale=(640, 640),
        ratio_range=(1.0, 1.0),
        max_cached_images=20,
        pad_val=(114, 114, 114)),
    dict(type='FilterAnnotations', min_gt_bbox_wh=(1, 1)),
    dict(type='PackDetInputs')
]

train_dataloader = dict(pin_memory=True, dataset=dict(pipeline=train_pipeline))

train_pipeline_stage2 = [
    dict(type='LoadImageFromFile', backend_args={{_base_.backend_args}}),
    dict(
        type='LoadAnnotations',
        with_bbox=True,
        with_mask=True,
        poly2mask=False),
    dict(
        type='RandomResize',
        scale=(640, 640),
        ratio_range=(0.1, 2.0),
        keep_ratio=True),
    dict(
        type='RandomCrop',
        crop_size=(640, 640),
        recompute_bbox=True,
        allow_negative_crop=True),
    dict(type='FilterAnnotations', min_gt_bbox_wh=(1, 1)),
    dict(type='YOLOXHSVRandomAug'),
    dict(type='RandomFlip', prob=0.5),
    dict(type='Pad', size=(640, 640), pad_val=dict(img=(114, 114, 114))),
    dict(type='PackDetInputs')
]
custom_hooks = [
    dict(
        type='EMAHook',
        ema_type='ExpMomentumEMA',
        momentum=0.0002,
        update_buffers=True,
        priority=49),
    dict(
        type='PipelineSwitchHook',
        switch_epoch=280,
        switch_pipeline=train_pipeline_stage2)
]

val_evaluator = dict(metric=['bbox', 'segm'])
test_evaluator = val_evaluator

  1. Did you make any modifications on the code or config? Did you understand what you have modified? Yes. I did.

  2. What dataset did you use? ICDAR 2015, a text detection dataset.

Environment

I guarantee my environment works.

Results

If applicable, paste the related results here, e.g., what you expect and what you get.

04/16 04:13:05 - mmengine - INFO - ==================================
04/16 04:13:05 - mmengine - INFO - Done image [50 /2000], fps: 3.9 img/s, times per image: 253.4 ms/img, cuda memory: 1310 MB
04/16 04:13:05 - mmengine - INFO - (GB) mem_used: 24.63 | uss: 2.70 | pss: 2.70 | total_proc: 1
04/16 04:13:09 - mmengine - INFO - ==================================
04/16 04:13:10 - mmengine - INFO - Done image [100/2000], fps: 6.7 img/s, times per image: 148.5 ms/img, cuda memory: 1212 MB
04/16 04:13:10 - mmengine - INFO - (GB) mem_used: 24.73 | uss: 2.70 | pss: 2.70 | total_proc: 1
04/16 04:13:13 - mmengine - INFO - ==================================
04/16 04:13:13 - mmengine - INFO - Done image [150/2000], fps: 9.0 img/s, times per image: 111.7 ms/img, cuda memory: 1310 MB
04/16 04:13:13 - mmengine - INFO - (GB) mem_used: 24.71 | uss: 2.70 | pss: 2.70 | total_proc: 1
04/16 04:13:16 - mmengine - INFO - ==================================
04/16 04:13:16 - mmengine - INFO - Done image [200/2000], fps: 10.6 img/s, times per image: 94.3 ms/img, cuda memory: 1310 MB
04/16 04:13:16 - mmengine - INFO - (GB) mem_used: 24.77 | uss: 2.70 | pss: 2.70 | total_proc: 1
04/16 04:13:19 - mmengine - INFO - ==================================
04/16 04:13:19 - mmengine - INFO - Done image [250/2000], fps: 11.9 img/s, times per image: 84.1 ms/img, cuda memory: 1310 MB
04/16 04:13:20 - mmengine - INFO - (GB) mem_used: 24.81 | uss: 2.70 | pss: 2.70 | total_proc: 1
04/16 04:13:23 - mmengine - INFO - ==================================
04/16 04:13:23 - mmengine - INFO - Done image [300/2000], fps: 12.8 img/s, times per image: 77.8 ms/img, cuda memory: 1310 MB
04/16 04:13:23 - mmengine - INFO - (GB) mem_used: 22.45 | uss: 2.70 | pss: 2.70 | total_proc: 1
04/16 04:13:49 - mmengine - INFO - ==================================
04/16 04:13:49 - mmengine - INFO - Done image [350/2000], fps: 8.9 img/s, times per image: 112.4 ms/img, cuda memory: 1310 MB
04/16 04:13:49 - mmengine - INFO - (GB) mem_used: 24.20 | uss: 2.70 | pss: 2.70 | total_proc: 1
04/16 04:14:16 - mmengine - INFO - ==================================
04/16 04:14:16 - mmengine - INFO - Done image [400/2000], fps: 7.3 img/s, times per image: 137.1 ms/img, cuda memory: 1310 MB
04/16 04:14:17 - mmengine - INFO - (GB) mem_used: 24.58 | uss: 2.70 | pss: 2.70 | total_proc: 1
04/16 04:14:39 - mmengine - INFO - ==================================
04/16 04:14:39 - mmengine - INFO - Done image [450/2000], fps: 6.6 img/s, times per image: 152.3 ms/img, cuda memory: 1310 MB
04/16 04:14:39 - mmengine - INFO - (GB) mem_used: 24.58 | uss: 2.70 | pss: 2.70 | total_proc: 1
04/16 04:15:04 - mmengine - INFO - ==================================
04/16 04:15:04 - mmengine - INFO - Done image [500/2000], fps: 6.0 img/s, times per image: 165.8 ms/img, cuda memory: 1310 MB
04/16 04:15:05 - mmengine - INFO - (GB) mem_used: 24.73 | uss: 2.70 | pss: 2.70 | total_proc: 1
04/16 04:15:05 - mmengine - INFO - ============== Done ==================
04/16 04:15:05 - mmengine - INFO - Overall fps: 6.0 img/s, times per image: 166.7 ms/img
04/16 04:15:05 - mmengine - INFO - cuda memory: 220 MB
04/16 04:15:05 - mmengine - INFO - (GB) mem_used: 24.74 | uss: 2.70 | pss: 2.70 | total_proc: 1
rose-jinyang commented 1 year ago

Hi @pd162 I found that you used the original number of classes on COCO dataset as num_classes. Why did NOT u change the num_classes as the number of classes on your dataset?

pd162 commented 1 year ago

Hi @pd162 I found that you used the original number of classes on COCO dataset as num_classes. Why did NOT u change the num_classes as the number of classes on your dataset?

In fact, I have modified the num_classes=1 for my task. But it occupied so large graphic memory to train and test that I cannot stand.

rose-jinyang commented 1 year ago

Hi @pd162 Thanks for your reply. I also tried to use num_classes == 1 but the training fails in the loss computation step. I could not find the reason yet. Can we cooperate on this issue?

convergence-lab commented 1 year ago

I have met similar problem with RTMDet detection inference. The inference was quite slow. I noticed init_detector does not return the model on GPU even if device argument pass correctly.

I fixed the issue simply call detector.cuda() right after calling init_detector.

I don't know why the issue occurred. But now I am satisfied.