search

open-mmlab / mmdetection

OpenMMLab Detection Toolbox and Benchmark
https://mmdetection.readthedocs.io
Apache License 2.0
29.61k stars 9.47k forks source link

Why does loss suddenly increase during training when using albu data augmentation？ #2189

Closed whuhangzhang closed 4 years ago

whuhangzhang commented 4 years ago

Checklist

  1. I have searched related issues but cannot get the expected help. https://github.com/open-mmlab/mmdetection/pull/1818
  2. The bug has not been fixed in the latest version. yes

Describe the bug hello @tyomj @ternaus @hellock @yhcao6 ,when i use albu data augmentation like https://github.com/open-mmlab/mmdetection/blob/master/configs/albu_example/mask_rcnn_r50_fpn_1x.py. I modified the original setting of albu and used the random 30 degree rotation and extended anchor ratio, but the loss will suddenly increase after the 15th epon, and this error also exists in htc_x101_32x4d_fpn without semantic, could you please fix this error?

【Note that I have modified the Settings in these places】

  1. https://github.com/open-mmlab/mmdetection/blob/master/configs/albu_example/mask_rcnn_r50_fpn_1x.py#L124
  2. https://github.com/open-mmlab/mmdetection/blob/master/configs/albu_example/mask_rcnn_r50_fpn_1x.py#L22
  3. https://github.com/open-mmlab/mmdetection/blob/master/configs/albu_example/mask_rcnn_r50_fpn_1x.py#L104
# model settings
model = dict(
    type='MaskRCNN',
    pretrained='open-mmlab://resnext101_32x4d',
    backbone=dict(
        type='ResNeXt',
        depth=101,
        groups=32,
        base_width=4,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=1,
        style='pytorch',
        dcn=dict(
            type='DCN',
            # groups=32,
            deformable_groups=1,
            fallback_on_stride=False),
        stage_with_dcn=(False, True, True, True)),
    neck=dict(
        type='FPN',
        in_channels=[256, 512, 1024, 2048],
        out_channels=256,
        num_outs=5),
    rpn_head=dict(
        type='RPNHead',
        in_channels=256,
        feat_channels=256,
        anchor_scales=[8],
        anchor_ratios=[0.2, 0.5, 1.0, 2.0, 5.0],
        anchor_strides=[4, 8, 16, 32, 64],
        target_means=[.0, .0, .0, .0],
        target_stds=[1.0, 1.0, 1.0, 1.0],
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0)),
    bbox_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=7, sample_num=2),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    bbox_head=dict(
        type='SharedFCBBoxHead',
        num_fcs=2,
        in_channels=256,
        fc_out_channels=1024,
        roi_feat_size=7,
        num_classes=2,
        target_means=[0., 0., 0., 0.],
        target_stds=[0.1, 0.1, 0.2, 0.2],
        reg_class_agnostic=False,
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)),
    mask_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=14, sample_num=2),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    mask_head=dict(
        type='FCNMaskHead',
        num_convs=4,
        in_channels=256,
        conv_out_channels=256,
        num_classes=2,
        loss_mask=dict(
            type='CrossEntropyLoss', use_mask=True, loss_weight=1.0)))
# model training and testing settings
train_cfg = dict(
    rpn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.7,
            neg_iou_thr=0.3,
            min_pos_iou=0.3,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=256,
            pos_fraction=0.5,
            neg_pos_ub=-1,
            add_gt_as_proposals=False),
        allowed_border=0,
        pos_weight=-1,
        debug=False),
    rpn_proposal=dict(
        nms_across_levels=False,
        nms_pre=2000,
        nms_post=2000,
        max_num=2000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.5,
            neg_iou_thr=0.5,
            min_pos_iou=0.5,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=512,
            pos_fraction=0.25,
            neg_pos_ub=-1,
            add_gt_as_proposals=True),
        mask_size=28,
        pos_weight=-1,
        debug=False))
test_cfg = dict(
    rpn=dict(
        nms_across_levels=False,
        nms_pre=2000,
        nms_post=1000,
        max_num=1000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        score_thr=0.05,
        nms=dict(type='nms', iou_thr=0.5),
        max_per_img=100,
        mask_thr_binary=0.5))
# dataset settings
dataset_type = 'ScenetextDataset'
data_root = '/data/'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
albu_train_transforms = [
    dict(
        type='ShiftScaleRotate',
        shift_limit=0.0625,
        scale_limit=0.0,
        rotate_limit=30,
        interpolation=1,
        p=0.5),
    dict(
        type='RandomBrightnessContrast',
        brightness_limit=[0.1, 0.3],
        contrast_limit=[0.1, 0.3],
        p=0.2),
    dict(
        type='OneOf',
        transforms=[
            dict(
                type='RGBShift',
                r_shift_limit=10,
                g_shift_limit=10,
                b_shift_limit=10,
                p=1.0),
            dict(
                type='HueSaturationValue',
                hue_shift_limit=20,
                sat_shift_limit=30,
                val_shift_limit=20,
                p=1.0)
        ],
        p=0.1),
    dict(type='JpegCompression', quality_lower=85, quality_upper=95, p=0.2),
    dict(type='ChannelShuffle', p=0.1),
    dict(
        type='OneOf',
        transforms=[
            dict(type='Blur', blur_limit=3, p=1.0),
            dict(type='MedianBlur', blur_limit=3, p=1.0)
        ],
        p=0.1),
]
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
    dict(type='Resize', img_scale=(1024, 960), keep_ratio=True),
    dict(type='Pad', size_divisor=32),
    dict(
        type='Albu',
        transforms=albu_train_transforms,
        bbox_params=dict(
            type='BboxParams',
            format='pascal_voc',
            label_fields=['gt_labels'],
            min_visibility=0.0,
            filter_lost_elements=True),
        keymap={
            'img': 'image',
            'gt_masks': 'masks',
            'gt_bboxes': 'bboxes'
        },
        update_pad_shape=False,
        skip_img_without_anno=True),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='DefaultFormatBundle'),
    dict(
        type='Collect',
        keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks'],
        meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',
                   'pad_shape', 'scale_factor'))
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1024, 960),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
data = dict(
    imgs_per_gpu=2,
    workers_per_gpu=2,
    train=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/gd_xiaode_no_aug_train.json',
        img_prefix=data_root + 'images/',
        pipeline=train_pipeline),
    val=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/gd_xiaode_no_aug_val.json',
        img_prefix=data_root + 'images/',
        pipeline=test_pipeline),
    test=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/gd_xiaode_no_aug_val.json',
        img_prefix=data_root + 'images/',
        pipeline=test_pipeline))
evaluation = dict(interval=36, metric=['bbox', 'segm'])
# optimizer
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
# learning policy
lr_config = dict(
    policy='step',
    warmup='linear',
    warmup_iters=500,
    warmup_ratio=1.0 / 3,
    step=[24, 33])
checkpoint_config = dict(interval=1)
# yapf:disable
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        # dict(type='TensorboardLoggerHook')
    ])
# yapf:enable
# runtime settings
total_epochs = 36
dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = './work_dirs/mask_rcnn_x101_fpn_3x_albu'
load_from = None
resume_from = None
workflow = [('train', 1)]

Reproduction

  1. What command or script did you run? 
    #!/bin/bash
export PYTHONPATH=`pwd`:$PYTHONPATH
CUDA_VISIBLE_DEVICES=2 python tools/train.py mask_rcnn_x101_fpn_3x_expend_anchor_albu.py
  2. Did you make any modifications on the code or config? Did you understand what you have modified? No change!
  3. What dataset did you use? custom datasets (like coco ) Environment
    • OS: centos 7.2
    • GCC :4.9.4
    • PyTorch version : 1.1.0
    • How you installed PyTorch : pip
    • GPU model : V100 *8
    • CUDA and CUDNN version: CUDA9 and cudnn7

Error traceback If applicable, paste the error trackback here.

2020-02-28 22:42:37,631 - mmdet - INFO - Epoch [15][50/1597]    lr: 0.02000, eta: 9:03:08, time: 1.144, data_time: 0.240, memory: 6915, loss_rpn_cls: 0.0123, loss_rpn_bbox: 0.0160, loss_cls: 0.1203, acc: 95.0410, loss_bbox: 0.0759, loss_mask: 0.1827, loss: 0.4072
2020-02-28 22:43:25,793 - mmdet - INFO - Epoch [15][100/1597]   lr: 0.02000, eta: 9:02:25, time: 0.963, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.0137, loss_rpn_bbox: 0.0148, loss_cls: 0.1201, acc: 95.1555, loss_bbox: 0.0725, loss_mask: 0.1803, loss: 0.4013
2020-02-28 22:44:14,066 - mmdet - INFO - Epoch [15][150/1597]   lr: 0.02000, eta: 9:01:41, time: 0.965, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0117, loss_rpn_bbox: 0.0153, loss_cls: 0.1128, acc: 95.3838, loss_bbox: 0.0713, loss_mask: 0.1833, loss: 0.3945
2020-02-28 22:45:02,124 - mmdet - INFO - Epoch [15][200/1597]   lr: 0.02000, eta: 9:00:57, time: 0.961, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0118, loss_rpn_bbox: 0.0165, loss_cls: 0.1187, acc: 95.1553, loss_bbox: 0.0772, loss_mask: 0.1881, loss: 0.4122
2020-02-28 22:45:49,831 - mmdet - INFO - Epoch [15][250/1597]   lr: 0.02000, eta: 9:00:12, time: 0.954, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.0128, loss_rpn_bbox: 0.0164, loss_cls: 0.1198, acc: 95.1387, loss_bbox: 0.0775, loss_mask: 0.1942, loss: 0.4207
2020-02-28 22:46:36,517 - mmdet - INFO - Epoch [15][300/1597]   lr: 0.02000, eta: 8:59:26, time: 0.934, data_time: 0.033, memory: 6915, loss_rpn_cls: 0.0120, loss_rpn_bbox: 0.0160, loss_cls: 0.1191, acc: 95.1389, loss_bbox: 0.0768, loss_mask: 0.1787, loss: 0.4026
2020-02-28 22:47:24,363 - mmdet - INFO - Epoch [15][350/1597]   lr: 0.02000, eta: 8:58:42, time: 0.956, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.0134, loss_rpn_bbox: 0.0164, loss_cls: 0.1173, acc: 95.1985, loss_bbox: 0.0747, loss_mask: 0.1795, loss: 0.4014
2020-02-28 22:48:11,572 - mmdet - INFO - Epoch [15][400/1597]   lr: 0.02000, eta: 8:57:57, time: 0.945, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.0127, loss_rpn_bbox: 0.0162, loss_cls: 0.1184, acc: 95.1401, loss_bbox: 0.0774, loss_mask: 0.1847, loss: 0.4094
2020-02-28 22:48:58,651 - mmdet - INFO - Epoch [15][450/1597]   lr: 0.02000, eta: 8:57:11, time: 0.941, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0131, loss_rpn_bbox: 0.0166, loss_cls: 0.1125, acc: 95.4646, loss_bbox: 0.0741, loss_mask: 0.1919, loss: 0.4081
2020-02-28 22:49:46,141 - mmdet - INFO - Epoch [15][500/1597]   lr: 0.02000, eta: 8:56:26, time: 0.950, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0113, loss_rpn_bbox: 0.0163, loss_cls: 0.1198, acc: 95.1125, loss_bbox: 0.0790, loss_mask: 0.1846, loss: 0.4109
2020-02-28 22:50:33,445 - mmdet - INFO - Epoch [15][550/1597]   lr: 0.02000, eta: 8:55:41, time: 0.946, data_time: 0.046, memory: 6915, loss_rpn_cls: 0.0133, loss_rpn_bbox: 0.0151, loss_cls: 0.1175, acc: 95.1824, loss_bbox: 0.0707, loss_mask: 0.1825, loss: 0.3992
2020-02-28 22:51:20,188 - mmdet - INFO - Epoch [15][600/1597]   lr: 0.02000, eta: 8:54:55, time: 0.935, data_time: 0.034, memory: 6915, loss_rpn_cls: 0.0125, loss_rpn_bbox: 0.0148, loss_cls: 0.1220, acc: 94.9873, loss_bbox: 0.0750, loss_mask: 0.1792, loss: 0.4035
2020-02-28 22:52:08,212 - mmdet - INFO - Epoch [15][650/1597]   lr: 0.02000, eta: 8:54:11, time: 0.961, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.0125, loss_rpn_bbox: 0.0155, loss_cls: 0.1183, acc: 95.1719, loss_bbox: 0.0734, loss_mask: 0.1854, loss: 0.4050
2020-02-28 22:52:55,178 - mmdet - INFO - Epoch [15][700/1597]   lr: 0.02000, eta: 8:53:25, time: 0.939, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0134, loss_rpn_bbox: 0.0146, loss_cls: 0.1218, acc: 95.0764, loss_bbox: 0.0739, loss_mask: 0.1954, loss: 0.4191
2020-02-28 22:53:42,082 - mmdet - INFO - Epoch [15][750/1597]   lr: 0.02000, eta: 8:52:39, time: 0.939, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0129, loss_rpn_bbox: 0.0166, loss_cls: 0.1217, acc: 95.0298, loss_bbox: 0.0768, loss_mask: 0.1796, loss: 0.4075
2020-02-28 22:54:29,907 - mmdet - INFO - Epoch [15][800/1597]   lr: 0.02000, eta: 8:51:55, time: 0.957, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0133, loss_rpn_bbox: 0.0160, loss_cls: 0.1218, acc: 94.9822, loss_bbox: 0.0792, loss_mask: 0.1890, loss: 0.4193
2020-02-28 22:55:17,470 - mmdet - INFO - Epoch [15][850/1597]   lr: 0.02000, eta: 8:51:10, time: 0.951, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0135, loss_rpn_bbox: 0.0161, loss_cls: 0.1229, acc: 94.9690, loss_bbox: 0.0795, loss_mask: 0.1904, loss: 0.4223
2020-02-28 22:56:05,244 - mmdet - INFO - Epoch [15][900/1597]   lr: 0.02000, eta: 8:50:25, time: 0.955, data_time: 0.045, memory: 6915, loss_rpn_cls: 0.0135, loss_rpn_bbox: 0.0179, loss_cls: 0.1201, acc: 95.0413, loss_bbox: 0.0770, loss_mask: 0.1780, loss: 0.4066
2020-02-28 22:56:53,064 - mmdet - INFO - Epoch [15][950/1597]   lr: 0.02000, eta: 8:49:41, time: 0.956, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0136, loss_rpn_bbox: 0.0171, loss_cls: 0.1201, acc: 95.0701, loss_bbox: 0.0767, loss_mask: 0.1869, loss: 0.4143
2020-02-28 22:57:40,798 - mmdet - INFO - Epoch [15][1000/1597]  lr: 0.02000, eta: 8:48:56, time: 0.955, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.0119, loss_rpn_bbox: 0.0154, loss_cls: 0.1244, acc: 94.8640, loss_bbox: 0.0765, loss_mask: 0.1923, loss: 0.4205
2020-02-28 22:58:27,871 - mmdet - INFO - Epoch [15][1050/1597]  lr: 0.02000, eta: 8:48:11, time: 0.941, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0151, loss_rpn_bbox: 0.0168, loss_cls: 0.1245, acc: 95.0212, loss_bbox: 0.0778, loss_mask: 0.1875, loss: 0.4216
2020-02-28 22:59:16,352 - mmdet - INFO - Epoch [15][1100/1597]  lr: 0.02000, eta: 8:47:27, time: 0.969, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.0145, loss_rpn_bbox: 0.0166, loss_cls: 0.1229, acc: 94.9785, loss_bbox: 0.0765, loss_mask: 0.1904, loss: 0.4209
2020-02-28 23:00:04,640 - mmdet - INFO - Epoch [15][1150/1597]  lr: 0.02000, eta: 8:46:43, time: 0.966, data_time: 0.047, memory: 6915, loss_rpn_cls: 0.0160, loss_rpn_bbox: 0.0187, loss_cls: 0.1324, acc: 94.5950, loss_bbox: 0.0819, loss_mask: 0.1932, loss: 0.4422
2020-02-28 23:00:52,217 - mmdet - INFO - Epoch [15][1200/1597]  lr: 0.02000, eta: 8:45:58, time: 0.952, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0138, loss_rpn_bbox: 0.0172, loss_cls: 0.1214, acc: 95.0952, loss_bbox: 0.0787, loss_mask: 0.1928, loss: 0.4240
2020-02-28 23:01:38,856 - mmdet - INFO - Epoch [15][1250/1597]  lr: 0.02000, eta: 8:45:12, time: 0.933, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0142, loss_rpn_bbox: 0.0164, loss_cls: 0.1237, acc: 94.9583, loss_bbox: 0.0805, loss_mask: 0.1981, loss: 0.4330
2020-02-28 23:02:27,509 - mmdet - INFO - Epoch [15][1300/1597]  lr: 0.02000, eta: 8:44:28, time: 0.973, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0145, loss_rpn_bbox: 0.0168, loss_cls: 0.1221, acc: 95.0525, loss_bbox: 0.0794, loss_mask: 0.1952, loss: 0.4280
2020-02-28 23:03:13,700 - mmdet - INFO - Epoch [15][1350/1597]  lr: 0.02000, eta: 8:43:41, time: 0.924, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0204, loss_rpn_bbox: 0.0167, loss_cls: 0.1295, acc: 94.7988, loss_bbox: 0.0797, loss_mask: 0.1961, loss: 0.4425
2020-02-28 23:04:00,766 - mmdet - INFO - Epoch [15][1400/1597]  lr: 0.02000, eta: 8:42:56, time: 0.941, data_time: 0.032, memory: 6915, loss_rpn_cls: 0.0152, loss_rpn_bbox: 0.0186, loss_cls: 0.1317, acc: 94.5842, loss_bbox: 0.0831, loss_mask: 0.1908, loss: 0.4394
2020-02-28 23:04:47,592 - mmdet - INFO - Epoch [15][1450/1597]  lr: 0.02000, eta: 8:42:10, time: 0.937, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0140, loss_rpn_bbox: 0.0158, loss_cls: 0.1272, acc: 94.8086, loss_bbox: 0.0807, loss_mask: 0.1872, loss: 0.4249
2020-02-28 23:05:34,585 - mmdet - INFO - Epoch [15][1500/1597]  lr: 0.02000, eta: 8:41:24, time: 0.940, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0153, loss_rpn_bbox: 0.0150, loss_cls: 0.1278, acc: 94.8647, loss_bbox: 0.0795, loss_mask: 0.1838, loss: 0.4215
2020-02-28 23:06:22,516 - mmdet - INFO - Epoch [15][1550/1597]  lr: 0.02000, eta: 8:40:39, time: 0.959, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0166, loss_rpn_bbox: 0.0177, loss_cls: 0.1294, acc: 94.7173, loss_bbox: 0.0838, loss_mask: 0.1951, loss: 0.4425
2020-02-28 23:08:09,128 - mmdet - INFO - Epoch [16][50/1597]    lr: 0.02000, eta: 8:38:25, time: 1.169, data_time: 0.241, memory: 6915, loss_rpn_cls: 0.0169, loss_rpn_bbox: 0.0173, loss_cls: 0.1327, acc: 94.5635, loss_bbox: 0.0825, loss_mask: 0.2026, loss: 0.4521
2020-02-28 23:08:56,801 - mmdet - INFO - Epoch [16][100/1597]   lr: 0.02000, eta: 8:37:40, time: 0.953, data_time: 0.050, memory: 6915, loss_rpn_cls: 0.0166, loss_rpn_bbox: 0.0161, loss_cls: 0.1234, acc: 94.9868, loss_bbox: 0.0759, loss_mask: 0.1901, loss: 0.4222
2020-02-28 23:09:44,695 - mmdet - INFO - Epoch [16][150/1597]   lr: 0.02000, eta: 8:36:56, time: 0.958, data_time: 0.048, memory: 6915, loss_rpn_cls: 0.0150, loss_rpn_bbox: 0.0165, loss_cls: 0.1207, acc: 95.1270, loss_bbox: 0.0761, loss_mask: 0.1861, loss: 0.4144
2020-02-28 23:10:32,384 - mmdet - INFO - Epoch [16][200/1597]   lr: 0.02000, eta: 8:36:11, time: 0.954, data_time: 0.045, memory: 6915, loss_rpn_cls: 0.0141, loss_rpn_bbox: 0.0152, loss_cls: 0.1205, acc: 95.1179, loss_bbox: 0.0735, loss_mask: 0.1909, loss: 0.4141
2020-02-28 23:11:20,172 - mmdet - INFO - Epoch [16][250/1597]   lr: 0.02000, eta: 8:35:26, time: 0.956, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0157, loss_rpn_bbox: 0.0163, loss_cls: 0.1336, acc: 94.5417, loss_bbox: 0.0827, loss_mask: 0.1949, loss: 0.4431
2020-02-28 23:12:07,609 - mmdet - INFO - Epoch [16][300/1597]   lr: 0.02000, eta: 8:34:41, time: 0.949, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.0135, loss_rpn_bbox: 0.0160, loss_cls: 0.1211, acc: 95.0649, loss_bbox: 0.0762, loss_mask: 0.1870, loss: 0.4138
2020-02-28 23:12:54,386 - mmdet - INFO - Epoch [16][350/1597]   lr: 0.02000, eta: 8:33:55, time: 0.935, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0142, loss_rpn_bbox: 0.0165, loss_cls: 0.1223, acc: 94.9717, loss_bbox: 0.0785, loss_mask: 0.1885, loss: 0.4200
2020-02-28 23:13:40,803 - mmdet - INFO - Epoch [16][400/1597]   lr: 0.02000, eta: 8:33:09, time: 0.928, data_time: 0.044, memory: 6915, loss_rpn_cls: 0.0141, loss_rpn_bbox: 0.0170, loss_cls: 0.1251, acc: 94.8159, loss_bbox: 0.0819, loss_mask: 0.1901, loss: 0.4281
2020-02-28 23:14:27,643 - mmdet - INFO - Epoch [16][450/1597]   lr: 0.02000, eta: 8:32:23, time: 0.937, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0147, loss_rpn_bbox: 0.0189, loss_cls: 0.1269, acc: 94.7927, loss_bbox: 0.0837, loss_mask: 0.2025, loss: 0.4466
2020-02-28 23:15:14,972 - mmdet - INFO - Epoch [16][500/1597]   lr: 0.02000, eta: 8:31:37, time: 0.947, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.0178, loss_rpn_bbox: 0.0221, loss_cls: 0.1535, acc: 93.6848, loss_bbox: 0.1009, loss_mask: 0.2110, loss: 0.5052
2020-02-28 23:16:01,848 - mmdet - INFO - Epoch [16][550/1597]   lr: 0.02000, eta: 8:30:51, time: 0.938, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0177, loss_rpn_bbox: 0.0195, loss_cls: 0.1309, acc: 94.6230, loss_bbox: 0.0845, loss_mask: 0.1893, loss: 0.4419
2020-02-28 23:16:50,853 - mmdet - INFO - Epoch [16][600/1597]   lr: 0.02000, eta: 8:30:08, time: 0.980, data_time: 0.034, memory: 6915, loss_rpn_cls: 0.0168, loss_rpn_bbox: 0.0182, loss_cls: 0.1389, acc: 94.2625, loss_bbox: 0.0876, loss_mask: 0.2042, loss: 0.4657
2020-02-28 23:17:37,525 - mmdet - INFO - Epoch [16][650/1597]   lr: 0.02000, eta: 8:29:22, time: 0.933, data_time: 0.033, memory: 6915, loss_rpn_cls: 0.0169, loss_rpn_bbox: 0.0172, loss_cls: 0.1358, acc: 94.5029, loss_bbox: 0.0831, loss_mask: 0.1952, loss: 0.4481
2020-02-28 23:18:24,291 - mmdet - INFO - Epoch [16][700/1597]   lr: 0.02000, eta: 8:28:36, time: 0.935, data_time: 0.034, memory: 6915, loss_rpn_cls: 0.0145, loss_rpn_bbox: 0.0172, loss_cls: 0.1280, acc: 94.7725, loss_bbox: 0.0805, loss_mask: 0.1892, loss: 0.4294
2020-02-28 23:19:12,031 - mmdet - INFO - Epoch [16][750/1597]   lr: 0.02000, eta: 8:27:51, time: 0.955, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0172, loss_rpn_bbox: 0.0199, loss_cls: 0.1366, acc: 94.3987, loss_bbox: 0.0901, loss_mask: 0.2052, loss: 0.4690
2020-02-28 23:19:58,810 - mmdet - INFO - Epoch [16][800/1597]   lr: 0.02000, eta: 8:27:05, time: 0.936, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0139, loss_rpn_bbox: 0.0163, loss_cls: 0.1243, acc: 94.9497, loss_bbox: 0.0811, loss_mask: 0.1800, loss: 0.4156
2020-02-28 23:20:46,010 - mmdet - INFO - Epoch [16][850/1597]   lr: 0.02000, eta: 8:26:20, time: 0.944, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.0176, loss_rpn_bbox: 0.0191, loss_cls: 0.1345, acc: 94.5435, loss_bbox: 0.0856, loss_mask: 0.2046, loss: 0.4614
2020-02-28 23:21:33,313 - mmdet - INFO - Epoch [16][900/1597]   lr: 0.02000, eta: 8:25:34, time: 0.946, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.0140, loss_rpn_bbox: 0.0168, loss_cls: 0.1254, acc: 94.8811, loss_bbox: 0.0819, loss_mask: 0.1886, loss: 0.4267
2020-02-28 23:22:20,652 - mmdet - INFO - Epoch [16][950/1597]   lr: 0.02000, eta: 8:24:49, time: 0.947, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0165, loss_rpn_bbox: 0.0186, loss_cls: 0.1370, acc: 94.3828, loss_bbox: 0.0901, loss_mask: 0.1965, loss: 0.4588
2020-02-28 23:23:08,513 - mmdet - INFO - Epoch [16][1000/1597]  lr: 0.02000, eta: 8:24:04, time: 0.957, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0159, loss_rpn_bbox: 0.0189, loss_cls: 0.1328, acc: 94.4641, loss_bbox: 0.0856, loss_mask: 0.1901, loss: 0.4433
2020-02-28 23:23:56,384 - mmdet - INFO - Epoch [16][1050/1597]  lr: 0.02000, eta: 8:23:20, time: 0.958, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.0150, loss_rpn_bbox: 0.0186, loss_cls: 0.1323, acc: 94.5413, loss_bbox: 0.0846, loss_mask: 0.1915, loss: 0.4422
2020-02-28 23:24:43,955 - mmdet - INFO - Epoch [16][1100/1597]  lr: 0.02000, eta: 8:22:35, time: 0.951, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0147, loss_rpn_bbox: 0.0174, loss_cls: 0.1341, acc: 94.5359, loss_bbox: 0.0864, loss_mask: 0.1933, loss: 0.4460
2020-02-28 23:25:31,061 - mmdet - INFO - Epoch [16][1150/1597]  lr: 0.02000, eta: 8:21:49, time: 0.942, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.0140, loss_rpn_bbox: 0.0185, loss_cls: 0.1304, acc: 94.6208, loss_bbox: 0.0872, loss_mask: 0.1970, loss: 0.4472
2020-02-28 23:26:18,999 - mmdet - INFO - Epoch [16][1200/1597]  lr: 0.02000, eta: 8:21:04, time: 0.959, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.0131, loss_rpn_bbox: 0.0168, loss_cls: 0.1224, acc: 95.0242, loss_bbox: 0.0774, loss_mask: 0.1834, loss: 0.4131
2020-02-28 23:27:07,040 - mmdet - INFO - Epoch [16][1250/1597]  lr: 0.02000, eta: 8:20:20, time: 0.961, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.0141, loss_rpn_bbox: 0.0164, loss_cls: 0.1237, acc: 94.9224, loss_bbox: 0.0810, loss_mask: 0.1891, loss: 0.4244
2020-02-28 23:27:54,622 - mmdet - INFO - Epoch [16][1300/1597]  lr: 0.02000, eta: 8:19:35, time: 0.951, data_time: 0.031, memory: 6915, loss_rpn_cls: 0.0170, loss_rpn_bbox: 0.0200, loss_cls: 0.1400, acc: 94.2537, loss_bbox: 0.0916, loss_mask: 0.1988, loss: 0.4675
2020-02-28 23:28:42,303 - mmdet - INFO - Epoch [16][1350/1597]  lr: 0.02000, eta: 8:18:50, time: 0.954, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.0248, loss_rpn_bbox: 0.0226, loss_cls: 0.1594, acc: 93.3970, loss_bbox: 0.1073, loss_mask: 0.2028, loss: 0.5169
2020-02-28 23:29:26,618 - mmdet - INFO - Epoch [16][1400/1597]  lr: 0.02000, eta: 8:18:01, time: 0.886, data_time: 0.030, memory: 6915, loss_rpn_cls: 291.5323, loss_rpn_bbox: 67.0836, loss_cls: 135.5149, acc: 91.7811, loss_bbox: 17.3593, loss_mask: 2178.4751, loss: 2689.9652
2020-02-28 23:30:09,803 - mmdet - INFO - Epoch [16][1450/1597]  lr: 0.02000, eta: 8:17:10, time: 0.864, data_time: 0.042, memory: 6915, loss_rpn_cls: 659.2569, loss_rpn_bbox: 428.5792, loss_cls: 4334.8523, acc: 87.3309, loss_bbox: 1169.4782, loss_mask: 1962.3003, loss: 8554.4673
2020-02-28 23:30:52,354 - mmdet - INFO - Epoch [16][1500/1597]  lr: 0.02000, eta: 8:16:18, time: 0.850, data_time: 0.038, memory: 6915, loss_rpn_cls: 2298.3458, loss_rpn_bbox: 941.3212, loss_cls: 6867.5134, acc: 87.2608, loss_bbox: 1659.7873, loss_mask: 8699.6353, loss: 20466.6026
2020-02-28 23:31:34,424 - mmdet - INFO - Epoch [16][1550/1597]  lr: 0.02000, eta: 8:15:26, time: 0.842, data_time: 0.036, memory: 6915, loss_rpn_cls: 34560.5711, loss_rpn_bbox: 20196.0533, loss_cls: 134253.9739, acc: 84.7126, loss_bbox: 173239.8111, loss_mask: 168253.4768, loss: 530503.8719
2020-02-28 23:33:10,507 - mmdet - INFO - Epoch [17][50/1597]    lr: 0.02000, eta: 8:13:09, time: 1.043, data_time: 0.229, memory: 6915, loss_rpn_cls: 888.9794, loss_rpn_bbox: 249.2877, loss_cls: 10332.9805, acc: 86.5781, loss_bbox: 2265.0971, loss_mask: 10465.6031, loss: 24201.9477
2020-02-28 23:33:53,067 - mmdet - INFO - Epoch [17][100/1597]   lr: 0.02000, eta: 8:12:18, time: 0.851, data_time: 0.039, memory: 6915, loss_rpn_cls: 62187.1519, loss_rpn_bbox: 25625.9393, loss_cls: 1685080.2555, acc: 95.3147, loss_bbox: 1130550.9912, loss_mask: 535324.3485, loss: 3438768.6382
2020-02-28 23:34:35,836 - mmdet - INFO - Epoch [17][150/1597]   lr: 0.02000, eta: 8:11:27, time: 0.855, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.5497, loss_rpn_bbox: 0.0657, loss_cls: 656.5872, acc: 97.5288, loss_bbox: 100.8471, loss_mask: 0.6937, loss: 758.7433
2020-02-28 23:35:18,105 - mmdet - INFO - Epoch [17][200/1597]   lr: 0.02000, eta: 8:10:36, time: 0.846, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.4552, loss_rpn_bbox: 0.0637, loss_cls: 0.1110, acc: 97.6616, loss_bbox: 0.0148, loss_mask: 0.7255, loss: 1.3702
2020-02-28 23:36:00,735 - mmdet - INFO - Epoch [17][250/1597]   lr: 0.02000, eta: 8:09:45, time: 0.853, data_time: 0.040, memory: 6915, loss_rpn_cls: 85.2085, loss_rpn_bbox: 16.9726, loss_cls: 190.6585, acc: 97.0966, loss_bbox: 160.9999, loss_mask: 334.0644, loss: 787.9038
2020-02-28 23:36:42,543 - mmdet - INFO - Epoch [17][300/1597]   lr: 0.02000, eta: 8:08:53, time: 0.836, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.3951, loss_rpn_bbox: 0.0642, loss_cls: 0.1067, acc: 97.7776, loss_bbox: 0.0129, loss_mask: 0.6899, loss: 1.2689
2020-02-28 23:37:26,234 - mmdet - INFO - Epoch [17][350/1597]   lr: 0.02000, eta: 8:08:03, time: 0.874, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.3938, loss_rpn_bbox: 0.0681, loss_cls: 0.1155, acc: 97.5400, loss_bbox: 0.0148, loss_mask: 0.6885, loss: 1.2807
2020-02-28 23:38:09,411 - mmdet - INFO - Epoch [17][400/1597]   lr: 0.02000, eta: 8:07:13, time: 0.863, data_time: 0.037, memory: 6915, loss_rpn_cls: 0.3869, loss_rpn_bbox: 0.0660, loss_cls: 0.1145, acc: 97.5669, loss_bbox: 0.0150, loss_mask: 0.6909, loss: 1.2733
2020-02-28 23:38:51,535 - mmdet - INFO - Epoch [17][450/1597]   lr: 0.02000, eta: 8:06:21, time: 0.843, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.3744, loss_rpn_bbox: 0.0644, loss_cls: 0.1095, acc: 97.7017, loss_bbox: 0.0138, loss_mask: 0.6911, loss: 1.2532
2020-02-28 23:39:33,332 - mmdet - INFO - Epoch [17][500/1597]   lr: 0.02000, eta: 8:05:30, time: 0.836, data_time: 0.035, memory: 6915, loss_rpn_cls: 0.3831, loss_rpn_bbox: 0.0686, loss_cls: 0.1144, acc: 97.5696, loss_bbox: 0.0145, loss_mask: 0.6925, loss: 1.2731
2020-02-28 23:40:14,947 - mmdet - INFO - Epoch [17][550/1597]   lr: 0.02000, eta: 8:04:38, time: 0.832, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.3771, loss_rpn_bbox: 0.0675, loss_cls: 0.1096, acc: 97.6973, loss_bbox: 0.0129, loss_mask: 0.6900, loss: 1.2572
2020-02-28 23:40:58,161 - mmdet - INFO - Epoch [17][600/1597]   lr: 0.02000, eta: 8:03:48, time: 0.864, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.3712, loss_rpn_bbox: 0.0653, loss_cls: 0.1109, acc: 97.6641, loss_bbox: 0.0138, loss_mask: 0.6908, loss: 1.2520
2020-02-28 23:41:40,403 - mmdet - INFO - Epoch [17][650/1597]   lr: 0.02000, eta: 8:02:56, time: 0.845, data_time: 0.043, memory: 6915, loss_rpn_cls: 0.3728, loss_rpn_bbox: 0.0647, loss_cls: 0.1109, acc: 97.6626, loss_bbox: 0.0135, loss_mask: 0.6902, loss: 1.2521
2020-02-28 23:42:22,660 - mmdet - INFO - Epoch [17][700/1597]   lr: 0.02000, eta: 8:02:05, time: 0.845, data_time: 0.043, memory: 6915, loss_rpn_cls: 0.3761, loss_rpn_bbox: 0.0680, loss_cls: 0.1137, acc: 97.5886, loss_bbox: 0.0148, loss_mask: 0.6896, loss: 1.2622
2020-02-28 23:43:04,407 - mmdet - INFO - Epoch [17][750/1597]   lr: 0.02000, eta: 8:01:14, time: 0.834, data_time: 0.033, memory: 6915, loss_rpn_cls: 0.3746, loss_rpn_bbox: 0.0670, loss_cls: 0.1115, acc: 97.6489, loss_bbox: 0.0133, loss_mask: 0.6905, loss: 1.2568
2020-02-28 23:43:46,908 - mmdet - INFO - Epoch [17][800/1597]   lr: 0.02000, eta: 8:00:23, time: 0.851, data_time: 0.045, memory: 6915, loss_rpn_cls: 0.3834, loss_rpn_bbox: 0.0692, loss_cls: 0.1144, acc: 97.5701, loss_bbox: 0.0138, loss_mask: 0.6918, loss: 1.2725
2020-02-28 23:44:29,771 - mmdet - INFO - Epoch [17][850/1597]   lr: 0.02000, eta: 7:59:33, time: 0.857, data_time: 0.042, memory: 6915, loss_rpn_cls: 0.3768, loss_rpn_bbox: 0.0666, loss_cls: 0.1259, acc: 97.6091, loss_bbox: 0.0379, loss_mask: 0.6913, loss: 1.2985
2020-02-28 23:45:12,013 - mmdet - INFO - Epoch [17][900/1597]   lr: 0.02000, eta: 7:58:41, time: 0.845, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.3734, loss_rpn_bbox: 0.0655, loss_cls: 0.1141, acc: 97.5771, loss_bbox: 0.0147, loss_mask: 0.6886, loss: 1.2564
2020-02-28 23:45:53,451 - mmdet - INFO - Epoch [17][950/1597]   lr: 0.02000, eta: 7:57:50, time: 0.829, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.3679, loss_rpn_bbox: 0.0650, loss_cls: 0.1133, acc: 97.5999, loss_bbox: 0.0158, loss_mask: 0.6912, loss: 1.2532
2020-02-28 23:46:34,998 - mmdet - INFO - Epoch [17][1000/1597]  lr: 0.02000, eta: 7:56:58, time: 0.831, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.3787, loss_rpn_bbox: 0.0687, loss_cls: 0.1152, acc: 97.5464, loss_bbox: 0.0148, loss_mask: 0.6891, loss: 1.2665
2020-02-28 23:47:15,764 - mmdet - INFO - Epoch [17][1050/1597]  lr: 0.02000, eta: 7:56:05, time: 0.815, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.3618, loss_rpn_bbox: 0.0626, loss_cls: 0.1080, acc: 97.7522, loss_bbox: 0.0141, loss_mask: 0.6911, loss: 1.2376
2020-02-28 23:47:57,786 - mmdet - INFO - Epoch [17][1100/1597]  lr: 0.02000, eta: 7:55:14, time: 0.840, data_time: 0.033, memory: 6915, loss_rpn_cls: 0.3679, loss_rpn_bbox: 0.0649, loss_cls: 0.1119, acc: 97.6370, loss_bbox: 0.0142, loss_mask: 0.6915, loss: 1.2503
2020-02-28 23:48:39,630 - mmdet - INFO - Epoch [17][1150/1597]  lr: 0.02000, eta: 7:54:23, time: 0.837, data_time: 0.033, memory: 6915, loss_rpn_cls: 0.3693, loss_rpn_bbox: 0.0666, loss_cls: 0.1136, acc: 97.5908, loss_bbox: 0.0146, loss_mask: 0.6891, loss: 1.2533
2020-02-28 23:49:21,353 - mmdet - INFO - Epoch [17][1200/1597]  lr: 0.02000, eta: 7:53:31, time: 0.835, data_time: 0.038, memory: 6915, loss_rpn_cls: 0.3704, loss_rpn_bbox: 0.0650, loss_cls: 0.1108, acc: 97.6685, loss_bbox: 0.0132, loss_mask: 0.6900, loss: 1.2494
2020-02-28 23:50:03,601 - mmdet - INFO - Epoch [17][1250/1597]  lr: 0.02000, eta: 7:52:41, time: 0.845, data_time: 0.031, memory: 6915, loss_rpn_cls: 0.3668, loss_rpn_bbox: 0.0659, loss_cls: 0.1081, acc: 97.7375, loss_bbox: 0.0132, loss_mask: 0.6896, loss: 1.2436
2020-02-28 23:50:46,104 - mmdet - INFO - Epoch [17][1300/1597]  lr: 0.02000, eta: 7:51:50, time: 0.850, data_time: 0.045, memory: 6915, loss_rpn_cls: 0.3716, loss_rpn_bbox: 0.0673, loss_cls: 0.1156, acc: 97.5381, loss_bbox: 0.0153, loss_mask: 0.6897, loss: 1.2594
2020-02-28 23:51:27,660 - mmdet - INFO - Epoch [17][1350/1597]  lr: 0.02000, eta: 7:50:59, time: 0.831, data_time: 0.041, memory: 6915, loss_rpn_cls: 0.3822, loss_rpn_bbox: 0.0689, loss_cls: 0.1141, acc: 97.5771, loss_bbox: 0.0133, loss_mask: 0.6891, loss: 1.2676
2020-02-28 23:52:08,417 - mmdet - INFO - Epoch [17][1400/1597]  lr: 0.02000, eta: 7:50:06, time: 0.815, data_time: 0.036, memory: 6915, loss_rpn_cls: 0.3673, loss_rpn_bbox: 0.0656, loss_cls: 0.1120, acc: 97.6343, loss_bbox: 0.0144, loss_mask: 0.6914, loss: 1.2506
2020-02-28 23:52:50,171 - mmdet - INFO - Epoch [17][1450/1597]  lr: 0.02000, eta: 7:49:15, time: 0.835, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.3702, loss_rpn_bbox: 0.0650, loss_cls: 0.1166, acc: 97.5090, loss_bbox: 0.0155, loss_mask: 0.6923, loss: 1.2595
2020-02-28 23:53:32,175 - mmdet - INFO - Epoch [17][1500/1597]  lr: 0.02000, eta: 7:48:24, time: 0.840, data_time: 0.039, memory: 6915, loss_rpn_cls: 0.3677, loss_rpn_bbox: 0.0655, loss_cls: 0.1138, acc: 97.5862, loss_bbox: 0.0158, loss_mask: 0.6918, loss: 1.2547
2020-02-28 23:54:13,574 - mmdet - INFO - Epoch [17][1550/1597]  lr: 0.02000, eta: 7:47:33, time: 0.828, data_time: 0.040, memory: 6915, loss_rpn_cls: 0.3673, loss_rpn_bbox: 0.0646, loss_cls: 0.1089, acc: 97.7170, loss_bbox: 0.0132, loss_mask: 0.6889, loss: 1.2429
whuhangzhang commented 4 years ago

i have sovle this problem, rotate_limit led to the problem

somebody-deep commented 4 years ago

@whuhangzhang How did you set the rotate_limit and fix the problem? Set the rotate_limit=10, or other value?