search

open-mmlab / mmdetection

OpenMMLab Detection Toolbox and Benchmark
https://mmdetection.readthedocs.io
Apache License 2.0
29.28k stars 9.41k forks source link

wrong padding while data preprocess with customized dataset #5536

Closed WangHuancheng closed 3 years ago

WangHuancheng commented 3 years ago

Describe the bug when trying to train my model with customized dataset, there is an Assertion failed, cv2.error: OpenCV(4.1.2) /io/opencv/modules/core/src/copy.cpp:1170: error: (-215:Assertion failed) top >= 0 && bottom >= 0 && left >= 0 && right >= 0 && _src.dims() <= 2 in function 'copyMakeBorder' It looks like there is a none have been send to opencv to generate border,However, It's really hard for me to track how and where the none comes from after all those call.Anyone have any idea about it? Reproduction

  1. What command or script did you run? train.py

  2. Did you make any modifications on the code or config? Did you understand what you have modified?

    model = dict(
type='MaskRCNN',
pretrained=None,
backbone=dict(
    type='SwinTransformer',
    embed_dim=96,
    depths=[2, 2, 6, 2],
    num_heads=[3, 6, 12, 24],
    window_size=7,
    mlp_ratio=4.0,
    qkv_bias=True,
    qk_scale=None,
    drop_rate=0.0,
    attn_drop_rate=0.0,
    drop_path_rate=0.1,
    ape=False,
    patch_norm=True,
    out_indices=(0, 1, 2, 3),
    use_checkpoint=False),
neck=dict(
    type='FPN',
    in_channels=[96, 192, 384, 768],
    out_channels=256,
    num_outs=5),
rpn_head=dict(
    type='RPNHead',
    in_channels=256,
    feat_channels=256,
    anchor_generator=dict(
        type='AnchorGenerator',
        scales=[8],
        ratios=[0.5, 1.0, 2.0],
        strides=[4, 8, 16, 32, 64]),
    bbox_coder=dict(
        type='DeltaXYWHBBoxCoder',
        target_means=[0.0, 0.0, 0.0, 0.0],
        target_stds=[1.0, 1.0, 1.0, 1.0]),
    loss_cls=dict(
        type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
    loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
roi_head=dict(
    type='StandardRoIHead',
    bbox_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    bbox_head=dict(
        type='Shared2FCBBoxHead',
        in_channels=256,
        fc_out_channels=1024,
        roi_feat_size=7,
        num_classes=1,
        bbox_coder=dict(
            type='DeltaXYWHBBoxCoder',
            target_means=[0.0, 0.0, 0.0, 0.0],
            target_stds=[0.1, 0.1, 0.2, 0.2]),
        reg_class_agnostic=False,
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
        loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
    mask_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', output_size=14, sampling_ratio=0),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    mask_head=dict(
        type='FCNMaskHead',
        num_convs=4,
        in_channels=256,
        conv_out_channels=256,
        num_classes=1,
        loss_mask=dict(
            type='CrossEntropyLoss', use_mask=True, loss_weight=1.0))),
train_cfg=dict(
    rpn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.7,
            neg_iou_thr=0.3,
            min_pos_iou=0.3,
            match_low_quality=True,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=256,
            pos_fraction=0.5,
            neg_pos_ub=-1,
            add_gt_as_proposals=False),
        allowed_border=-1,
        pos_weight=-1,
        debug=False),
    rpn_proposal=dict(
        nms_pre=2000,
        max_per_img=1000,
        nms=dict(type='nms', iou_threshold=0.7),
        min_bbox_size=0),
    rcnn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.5,
            neg_iou_thr=0.5,
            min_pos_iou=0.5,
            match_low_quality=True,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=512,
            pos_fraction=0.25,
            neg_pos_ub=-1,
            add_gt_as_proposals=True),
        mask_size=28,
        pos_weight=-1,
        debug=False)),
test_cfg=dict(
    rpn=dict(
        nms_pre=1000,
        max_per_img=1000,
        nms=dict(type='nms', iou_threshold=0.7),
        min_bbox_size=0),
    rcnn=dict(
        score_thr=0.05,
        nms=dict(type='nms', iou_threshold=0.5),
        max_per_img=100,
        mask_thr_binary=0.5)))
dataset_type = 'COCODataset'
data_root = 'data/coco/'
img_norm_cfg = dict(
mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
dict(type='RandomFlip', flip_ratio=0.5),
dict(
    type='AutoAugment',
    policies=[[{
        'type':
        'Resize',
        'img_scale': [(480, 1333), (512, 1333), (544, 1333), (576, 1333),
                      (608, 1333), (640, 1333), (672, 1333), (704, 1333),
                      (736, 1333), (768, 1333), (800, 1333)],
        'multiscale_mode':
        'value',
        'keep_ratio':
        True
    }],
              [{
                  'type': 'Resize',
                  'img_scale': [(400, 1333), (500, 1333), (600, 1333)],
                  'multiscale_mode': 'value',
                  'keep_ratio': True
              }, {
                  'type': 'RandomCrop',
                  'crop_type': 'absolute_range',
                  'crop_size': (384, 600),
                  'allow_negative_crop': True
              }, {
                  'type':
                  'Resize',
                  'img_scale': [(480, 1333), (512, 1333), (544, 1333),
                                (576, 1333), (608, 1333), (640, 1333),
                                (672, 1333), (704, 1333), (736, 1333),
                                (768, 1333), (800, 1333)],
                  'multiscale_mode':
                  'value',
                  'override':
                  True,
                  'keep_ratio':
                  True
              }]]),
dict(
    type='Normalize',
    mean=[123.675, 116.28, 103.53],
    std=[58.395, 57.12, 57.375],
    to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks'])
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
    type='MultiScaleFlipAug',
    img_scale=(1333, 800),
    flip=False,
    transforms=[
        dict(type='Resize', keep_ratio=True),
        dict(type='RandomFlip'),
        dict(
            type='Normalize',
            mean=[123.675, 116.28, 103.53],
            std=[58.395, 57.12, 57.375],
            to_rgb=True),
        dict(type='Pad', size_divisor=32),
        dict(type='ImageToTensor', keys=['img']),
        dict(type='Collect', keys=['img'])
    ])
]
data = dict(
samples_per_gpu=2,
workers_per_gpu=2,
train=dict(
    type='CocoDataset',
    ann_file='train/train.json',
    img_prefix='train/',
    pipeline=[
        dict(type='LoadImageFromFile'),
        dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
        dict(type='RandomFlip', flip_ratio=0.5),
        dict(
            type='AutoAugment',
            policies=[[{
                'type':
                'Resize',
                'img_scale': [(480, 1333), (512, 1333), (544, 1333),
                              (576, 1333), (608, 1333), (640, 1333),
                              (672, 1333), (704, 1333), (736, 1333),
                              (768, 1333), (800, 1333)],
                'multiscale_mode':
                'value',
                'keep_ratio':
                True
            }],
                      [{
                          'type': 'Resize',
                          'img_scale': [(400, 1333), (500, 1333),
                                        (600, 1333)],
                          'multiscale_mode': 'value',
                          'keep_ratio': True
                      }, {
                          'type': 'RandomCrop',
                          'crop_type': 'absolute_range',
                          'crop_size': (384, 600),
                          'allow_negative_crop': True
                      }, {
                          'type':
                          'Resize',
                          'img_scale': [(480, 1333), (512, 1333),
                                        (544, 1333), (576, 1333),
                                        (608, 1333), (640, 1333),
                                        (672, 1333), (704, 1333),
                                        (736, 1333), (768, 1333),
                                        (800, 1333)],
                          'multiscale_mode':
                          'value',
                          'override':
                          True,
                          'keep_ratio':
                          True
                      }]]),
        dict(
            type='Normalize',
            mean=[123.675, 116.28, 103.53],
            std=[58.395, 57.12, 57.375],
            to_rgb=True),
        dict(type='Pad', size_divisor=32),
        dict(type='DefaultFormatBundle'),
        dict(
            type='Collect',
            keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks'])
    ],
    classes=('neuron', )),
val=dict(
    type='CocoDataset',
    ann_file='val/val.json',
    img_prefix='val/',
    pipeline=[
        dict(type='LoadImageFromFile'),
        dict(
            type='MultiScaleFlipAug',
            img_scale=(1333, 800),
            flip=False,
            transforms=[
                dict(type='Resize', keep_ratio=True),
                dict(type='RandomFlip'),
                dict(
                    type='Normalize',
                    mean=[123.675, 116.28, 103.53],
                    std=[58.395, 57.12, 57.375],
                    to_rgb=True),
                dict(type='Pad', size_divisor=32),
                dict(type='ImageToTensor', keys=['img']),
                dict(type='Collect', keys=['img'])
            ])
    ],
    classes=('neuron', )),
test=dict(
    type='CocoDataset',
    ann_file='val/val.json',
    img_prefix='val/',
    pipeline=[
        dict(type='LoadImageFromFile'),
        dict(
            type='MultiScaleFlipAug',
            img_scale=(1333, 800),
            flip=False,
            transforms=[
                dict(type='Resize', keep_ratio=True),
                dict(type='RandomFlip'),
                dict(
                    type='Normalize',
                    mean=[123.675, 116.28, 103.53],
                    std=[58.395, 57.12, 57.375],
                    to_rgb=True),
                dict(type='Pad', size_divisor=32),
                dict(type='ImageToTensor', keys=['img']),
                dict(type='Collect', keys=['img'])
            ])
    ],
    classes=('neuron', )))
evaluation = dict(metric=['bbox', 'segm'])
optimizer = dict(
type='AdamW',
lr=0.0001,
betas=(0.9, 0.999),
weight_decay=0.05,
paramwise_cfg=dict(
    custom_keys=dict(
        absolute_pos_embed=dict(decay_mult=0.0),
        relative_position_bias_table=dict(decay_mult=0.0),
        norm=dict(decay_mult=0.0))))
optimizer_config = dict(grad_clip=None)
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=500,
warmup_ratio=0.001,
step=[8, 11])
runner = dict(type='EpochBasedRunnerAmp', max_epochs=12)
checkpoint_config = dict(interval=1)
log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')])
custom_hooks = [dict(type='NumClassCheckHook')]
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
classes = ('neuron', )
work_dir = './work_dirs/mask_rcnn_swin_tiny_patch4_window7_mstrain_480-800_adamw_1x_allen500'
gpu_ids = range(0, 1)

  3. What dataset did you use?

Environment

  1. Please run python mmdet/utils/collect_env.py to collect necessary environment information and paste it here.
  2. You may add addition that may be helpful for locating the problem, such as
    • How you installed PyTorch [e.g., pip, conda, source]
    • Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

Error traceback

2021-07-05 13:57:07,302 - mmdet - INFO - workflow: [('train', 1)], max: 12 epochs
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Traceback (most recent call last):
  File "tools/train.py", line 187, in <module>
    main()
  File "tools/train.py", line 183, in main
    meta=meta)
  File "/content/Swin-Transformer-Object-Detection/mmdet/apis/train.py", line 185, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/epoch_based_runner.py", line 47, in train
    for i, data_batch in enumerate(self.data_loader):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.7/dist-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
cv2.error: Caught error in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/content/Swin-Transformer-Object-Detection/mmdet/datasets/custom.py", line 193, in __getitem__
    data = self.prepare_train_img(idx)
  File "/content/Swin-Transformer-Object-Detection/mmdet/datasets/custom.py", line 216, in prepare_train_img
    return self.pipeline(results)
  File "/content/Swin-Transformer-Object-Detection/mmdet/datasets/pipelines/compose.py", line 40, in __call__
    data = t(data)
  File "/content/Swin-Transformer-Object-Detection/mmdet/datasets/pipelines/transforms.py", line 534, in __call__
    self._pad_masks(results)
  File "/content/Swin-Transformer-Object-Detection/mmdet/datasets/pipelines/transforms.py", line 515, in _pad_masks
    results[key] = results[key].pad(pad_shape, pad_val=self.pad_val)
  File "/content/Swin-Transformer-Object-Detection/mmdet/core/mask/structures.py", line 305, in pad
    for mask in self.masks
  File "/content/Swin-Transformer-Object-Detection/mmdet/core/mask/structures.py", line 305, in <listcomp>
    for mask in self.masks
  File "/usr/local/lib/python3.7/dist-packages/mmcv/image/geometric.py", line 517, in impad
    value=pad_val)
cv2.error: OpenCV(4.1.2) /io/opencv/modules/core/src/copy.cpp:1170: error: (-215:Assertion failed) top >= 0 && bottom >= 0 && left >= 0 && right >= 0 && _src.dims() <= 2 in function 'copyMakeBorder'
WangHuancheng commented 3 years ago

this is the code where the issue occur first

   img = cv2.copyMakeBorder(
        img,
        padding[1],
        padding[3],
        padding[0],
        padding[2],
        border_type[padding_mode],
        value=pad_val)

    return img

I print the shape of img(by the way its a numpy array) and the padding, below is the result

shape
(480, 544)
[[0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]
padding
(0, 0, -64, 64)
WangHuancheng commented 3 years ago

It turns out that when I generating the annotation, I got the 'width 'height' wrong, which leads to a issue annotation with false size of image.

zpyi commented 3 years ago

It turns out that when I generating the annotation, I got the 'width 'height' wrong, which leads to a issue annotation with false size of image. have you generated the annotations again?

zjwzcx commented 2 years ago

I met the same problem. After I updated the version of mmdet from 2.14.0 to 2.18.0, everything works well.