Divergence while training Mask RCNN with ResNet (50 or 101 backbone) on a custom COCO type format dataset

[ecm200]

Issue Description

I have been successfully training a Mask R-CNN ResNet50FPN (and 101) on a custom dataset of particle images using simple spheroid shapes (green loss functions in graph below). On moving to more complex shapes, such as cuboids, I have found that the model trains successfully for a while, and then diverges and eventually causes NAN in the loss functions. The only difference between the training of the models is the input data particles shapes. The images themselves, the bounding boxes and polygon information, the COCO dataset production has been kept the same as with the successful training of the spheroid particles.

I would welcome any insight that people might have with regards to what could be causing divergent behaviour in the training process. I have found that diverging loss functions occurs for both ResNet50 and 101 variants so far. I have been using the standard learning rate of 0.02 scaled down from a mini-batch of 16 images on 8 gpus, down to 1 gpu using a mini-batch of 2, LR= 0.02/8. Q. Could this be an issue relating to the more complexity in the image, needing a smaller learning rate?

My images have a range of particle content in them, with some having very small amounts of particles in the training image. Q. Would an excess of these types of images, where is relatively little object information potentially cause a gradient issue and divergence later in the process?

The cuboid dataset was training with ~3800 images and validating with 1500 images, so the divergence does not occur straight away, in this example it happened around the 15th epoch, so therefore the network had seen all images in the training set 15 times before the divergent behaviour sets in. Looking at the individual loss functions, it shows that the divergence is present in all loss functions.

As well attaching all the script, customized functions and dataset information, I also at the bottom append the output logs of the training process.

Training loss

Validation loss

Other individual loss functions

Model Configuration

The model configuration is heavily based on the standard COCO instance segmentation problem using variants of the Mask R-CNN architecture with ResNet FPN, the modifications to standard

_base_ = [
    '../configs/_base_/models/mask_rcnn_r50_fpn.py',
    '../configs/_base_/datasets/coco_instance.py',
    '../configs/_base_/schedules/schedule_1x.py', 
    '../configs/_base_/default_runtime.py'
]
dataset_type = 'MorphologiDataset' # 'CocoDataset'
data_root_dir = '/datadrive/drive2/willow_tree_synthetic/mpdsim/train_test_coco_fmt/' # '/datadrive/drive2/willow_tree_synthetic/mpdsim/train_test_coco_fmt/' #'/datadrive/drive0/willow_tree_synthetic/mpdsim/train_test_coco_fmt/'
data_dir = 'cuboidal_square_images_v1p2/' # 'spherical_v1_cuboidal_square_images_v1p1_combined_500_125_4536/' # 'test_dataset_v1p2_100/' #
data_root = data_root_dir+data_dir
classes=['particle']
# Update model due to classes
model = dict(
    pretrained=None, # Don't load the pretrained weights. TODO Find out if this is needed changing number of classes.
    roi_head=dict(
        bbox_head=dict(
            type='Shared2FCBBoxHead',
            in_channels=256,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=len(classes), # Modified to number of classes in this problem. COCO default is 80.
            bbox_coder=dict(
                type='DeltaXYWHBBoxCoder',
                target_means=[0., 0., 0., 0.],
                target_stds=[0.1, 0.1, 0.2, 0.2]),
            reg_class_agnostic=False,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)), # This is L1Loss in standard COCO, specified this way in CityScapes
        mask_head=dict(
            type='FCNMaskHead',
            num_convs=4,
            in_channels=256,
            conv_out_channels=256,
            num_classes=len(classes), # Modified to number of classes in this problem. COCO default is 80.
            loss_mask=dict(
                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0)))
)
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadMorphologiSynImage'),
    dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
    dict(type='Resize', img_scale=(1296, 972), keep_ratio=True),
    dict(type='Normalize', **img_norm_cfg),
    #dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks']),
]
test_pipeline = [
    dict(type='LoadMorphologiSynImage'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1296, 972),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='Normalize', **img_norm_cfg),
            #dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
data = dict(
    samples_per_gpu=2, # TODO Usually set to 2, set to 1 for debugging.
    imgs_per_gpu=2, # TODO Usually set to 2, set to 1 for debugging.
    workers_per_gpu=4,
    train=dict(
        type=dataset_type,
        classes=classes,
        ann_file=data_root + 'annotations/train_coco_changed_removed.json',
        img_prefix=data_root + 'train/',
        pipeline=train_pipeline),
    val=dict(
        type=dataset_type,
        classes=classes,
        ann_file=data_root + 'annotations/valid_coco_changed_removed.json',
        img_prefix=data_root + 'valid/',
        pipeline=test_pipeline),
    test=dict(
        type=dataset_type,
        classes=classes,
        ann_file=data_root + 'annotations/valid_coco_changed_removed.json',
        img_prefix=data_root + 'valid/',
        pipeline=test_pipeline))
test_cfg = dict(
    rpn=dict(
        nms_across_levels=False,
        nms_pre=1000,
        nms_post=1000,
        max_num=1000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        score_thr=0.05,
        nms=dict(type='nms', iou_thr=0.5),
        max_per_img=500, # Default is 100
        mask_thr_binary=0.5)) # Default is 0.5
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
checkpoint_config = dict(type='CheckpointHook', interval=1)
# checkpoint_config = dict(type='ValidationCheckpointHook', metric='acc', metric_ops='max', overwrite_checkpoint=True)
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        dict(type='TensorboardLoggerHook'),
        #dict(type='TensorboardLoggerWithImagesHook')
    ])
total_epochs = 36  # Usually 36
lr_config = dict(step=[24, 33]) # Based on 8 and 11 for 12 Epochs.
gpus = 1
evaluation = dict(interval=1, metric='bbox') # TODO Remove, temporarily here so that evaluation is done on BBOX only at the moment as SEGM not working.

Reproduction

I am running a custom training script that is heavily based on the training script example shipped with MMDetection.

import mmcv
from mmcv import Config, DictAction
from mmcv.runner import init_dist
import torch

from mmdet import __version__
#from mmdet.apis import set_random_seed, train_detector
from mmdet.datasets import build_dataset
from mmdet.models import build_detector
from mmdet.utils import collect_env, get_root_logger

# import __main__ as main
import os
import random
import datetime 
import shutil
import copy
import time
from glob import glob
#from sklearn.model_selection import train_test_split
import albumentations as A
import numpy as np
import argparse

from mmdetection_morphologi_pipelines import LoadMorphologiSynImage
from mmdetection_morphologi_datasets import MorphologiDataset
from mmdetection_morphologi_train import set_random_seed, train_detector
from mmdetection_morphologi_hooks import ValidationCheckpointHook, TensorboardLoggerWithImagesHook
from mmdetection_morphologi_utils import save_pickle

'''
Sub Class of CocoDataset for bespoke functions for Morphologi data.

Ed Morris
Malvern Panalytical
(c) 2020
'''

BASE_DIR = '' # 'advanced_seg/MMDetection_experiments' # Base DIR specification for executing in debug mode
TESTING = False

def parse_args():
    parser = argparse.ArgumentParser(description='Train a detector')
    parser.add_argument('--config', help='train config file path', default=os.path.join(BASE_DIR,'configs_morph/mmdetection_morphologi_mask_rcnn_r50_fpn_1x.py'))
    parser.add_argument('--work_dir', help='the dir to save logs and models', default=os.path.join(BASE_DIR,'output/mixed_NANs'))
    parser.add_argument('--workflow', type=int, help='Workflow type [1] train only, [2] train and validate every epoch', default=2)
    parser.add_argument('--job_name', help='name for output files and dirs', default='cuboidal_square_images_v1p2_') #'spherical_v1_cuboidal_square_images_v1p1_combined_500_125_4536_')
    parser.add_argument(
        '--resume-from', help='the checkpoint file to resume from')
    parser.add_argument(
        '--validate',
        action='store_true',
        help='whether to evaluate the checkpoint during training', default=True)
    group_gpus = parser.add_mutually_exclusive_group()
    group_gpus.add_argument(
        '--gpus',
        type=int,
        help='number of gpus to use '
        '(only applicable to non-distributed training)')
    group_gpus.add_argument(
        '--gpu-ids',
        type=int,
        nargs='+',
        help='ids of gpus to use '
        '(only applicable to non-distributed training)')
    parser.add_argument('--seed', type=int, default=42, help='random seed') # Default is usually 42
    parser.add_argument(
        '--deterministic',
        action='store_true',
        help='whether to set deterministic options for CUDNN backend.')
    parser.add_argument(
        '--options', nargs='+', action=DictAction, help='arguments in dict')
    parser.add_argument(
        '--launcher',
        choices=['none', 'pytorch', 'slurm', 'mpi'],
        default='none',
        help='job launcher')
    parser.add_argument('--local_rank', type=int, default=0)
    parser.add_argument(
        '--autoscale-lr',
        action='store_true',
        help='automatically scale lr with the number of gpus',
        default=True) #Added by ECM as this should [always be used
    args = parser.parse_args()
    if 'LOCAL_RANK' not in os.environ:
        os.environ['LOCAL_RANK'] = str(args.local_rank)

    return args

def main():

    args = parse_args()

    # Output dir and job details
    if args.job_name[-1] == '_':
        job_name_preamble = args.job_name
    else:
        job_name_preamble = args.job_name + '_'

    #### CONFIG
    ## Get the Base Config
    cfg = Config.fromfile(args.config)

    ## Set up Config

    # Get additional keyword arguments for configuration
    if args.options is not None:
        cfg.merge_from_dict(args.options)

    # set cudnn_benchmark
    if cfg.get('cudnn_benchmark', False):
        torch.backends.cudnn.benchmark = True

    # Setup output dir and copy over the exection script, save copy of config and input arguments
    if args.work_dir is not None:
        if TESTING:
            output_base_dir = os.path.join(args.work_dir,'testing')
        else:
            output_base_dir = args.work_dir
    elif cfg.get('work_dir', None) is None:
        if TESTING:
            output_base_dir = os.path.join(args.work_dir,'output/testing')
        else:
            output_base_dir = os.path.join(BASE_DIR,'output')
    cfg.work_dir = os.path.join(output_base_dir,job_name_preamble+cfg.model.type+'_'+cfg.model.backbone.type+str(cfg.model.backbone.depth)+'_'+cfg.model.neck.type+'_'+datetime.datetime.now().strftime('%d%m%Y_%H%M%S'))
    os.makedirs(cfg.work_dir, exist_ok=True)
    # Save this file
    shutil.copyfile(__file__, os.path.join(cfg.work_dir,__file__.split('/')[-1]))
    # Save the input arguments
    args_out_path = os.path.join(cfg.work_dir,'args_in_dict.pkl')
    save_pickle(pkl_object=args.__dict__, fname=args_out_path)
    # Save the config to file
    cfg_out_path = os.path.join(cfg.work_dir,cfg.work_dir.split('/')[-1]+'.cfg')
    cfg_out = open(cfg_out_path, 'w')
    cfg_out.writelines(cfg.text)
    cfg_out.close()

    # init the logger before other steps
    timestamp = time.strftime('%Y%m%d_%H%M%S', time.localtime())
    log_file = os.path.join(cfg.work_dir, '{}.log'.format(timestamp))
    logger = get_root_logger(log_file=log_file, log_level=cfg.log_level)
    logger.info('Created output directory: {}'.format(cfg.work_dir))

    logger.info('Command line arguments passed to training script: ')
    for arg_key,arg_value in args.__dict__.items():
        logger.info('{} :: {}'.format(arg_key, arg_value))

    # Resume from previous iteration
    if args.resume_from is not None:
        cfg.resume_from = args.resume_from

    # Update the default number of GPUs if different from config.
    if args.gpu_ids is not None:
        cfg.gpu_ids = args.gpu_ids
    else:
        cfg.gpu_ids = range(1) if args.gpus is None else range(args.gpus)

    # Update the autoscaler with the number of GPUs if changed from default 8 gpus.
    # ECM Modified so it takes into account the actual mini-batch size, which is dependent on the number of gpus and the images per gpu.
    # It now scales with the default mini-batch of 8 gpus and 2 images per gpu (16), and changes in response to changes in both gpus and/or images per gpu.
    if args.autoscale_lr:
        _old_lr = cfg.optimizer['lr']
        # Custom LR modification
        cfg.optimizer['lr'] =cfg.optimizer['lr'] * (len(cfg.gpu_ids) * cfg.data.imgs_per_gpu) / (16)
        # Example LR modification
        #cfg.optimizer['lr'] = cfg.optimizer['lr'] * len(cfg.gpu_ids) / 8
        logger.info('Applying linear Learning Rate correction. LR changed from {} to {}'.format(_old_lr, cfg.optimizer['lr']))

    # init distributed env first, since logger depends on the dist info.
    if args.launcher == 'none':
        logger.info('Distributed environment has not been initialized.')
        distributed = False
    else:
        logger.info('Distributed environment initialised.')
        distributed = True
        init_dist(args.launcher, **cfg.dist_params)

    ## Set workflow overide
    # Just training or validation will be done through the evaluation hook.
    if (args.workflow == 1):# or (args.validate):
        cfg.workflow = [('train', 1)]
        logger.info('Setting workflow to train only {}'.format(cfg.workflow))
    # Run validation through the val() function.
    elif args.workflow == 2:
        cfg.workflow = [('train', 1), ('val', 1)]
        logger.info('Setting workflow to train and validate {}'.format(cfg.workflow))
    # Report if using COCO validation metrics.
    if args.validate:
        logger.info('COCO Validation will be performed after every {} training epoch(s) using metrics {}'.format(cfg.evaluation.interval, cfg.evaluation.metric))

    # init the meta dict to record some important information such as
    # environment info and seed, which will be logged
    meta = dict()
    # log env info
    env_info_dict = collect_env()
    env_info = '\n'.join([('{}: {}'.format(k, v))
                            for k, v in env_info_dict.items()])
    dash_line = '-' * 60 + '\n'
    logger.info('Environment info:\n' + dash_line + env_info + '\n' +
                dash_line)
    meta['env_info'] = env_info

    # log some basic info
    logger.info('Distributed training: {}'.format(distributed))
    logger.info('Config:\n{}'.format(cfg.text))

    # set random seeds
    if args.seed is not None:
        logger.info(f'Set random seed to {args.seed}, '
                    f'deterministic: {args.deterministic}')
        set_random_seed(args.seed, deterministic=args.deterministic)
    cfg.seed = args.seed
    meta['seed'] = args.seed

    model = build_detector(cfg.model, train_cfg=cfg.train_cfg, test_cfg=cfg.test_cfg)

    datasets = [build_dataset(cfg.data.train)] #, build_dataset(cfg.data.val)]

    if len(cfg.workflow) == 2:
        val_dataset = copy.deepcopy(cfg.data.val)
        val_dataset.pipeline = cfg.data.train.pipeline
        datasets.append(build_dataset(val_dataset))
    if cfg.checkpoint_config is not None:
        # save mmdet version, config file content and class names in
        # checkpoints as meta data
        cfg.checkpoint_config.meta = dict(
            mmdet_version=__version__,
            config=cfg.text,
            CLASSES=datasets[0].CLASSES)

    # add an attribute for visualization convenience
    model.CLASSES = datasets[0].CLASSES
    train_detector(
        model,
        datasets,
        cfg,
        distributed=distributed,
        validate=args.validate,
        timestamp=timestamp,
        meta=meta)

if __name__ == '__main__':

    main()

Custom Code

I have made a custom dataset type, which is heavily based on the COCO dataset type as I have converted my custom dataset into the COCO format, with the same directory structure, and annotations saved into a JSON files for train and test.

I have written a custom image loading hook due to nature of the images being single channel.

#from mmdet.datasets.pipelines import LoadImageFromFile
from mmdet.datasets import PIPELINES
import cv2
import numpy as np
import os.path as osp

@PIPELINES.register_module()
class LoadMorphologiSynImage(object):
    '''
    Sub Class of CocoDataset for bespoke functions for Morphologi data.

    Ed Morris
    Malvern Panalytical
    (c) 2020
    '''

    def __init__(self, image_scale=255.0, image_format=np.float32):

        self.image_scale=image_scale
        self.image_format=image_format

    def __call__(self, results):

        if results['img_prefix'] is not None:
            filename = osp.join(results['img_prefix'],
                                results['img_info']['filename'])
        else:
            filename = results['img_info']['filename']

        img = cv2.imread(filename, cv2.IMREAD_ANYDEPTH | cv2.IMREAD_COLOR)

        # TODO think about whether this should be set to maximum possible value of image (4096) rather than image max.
        #      This might be nullifying the impact of brightness changes and reducing variability of the input dataset.
        img = (img / img.max()) * self.image_scale

        img = self.image_format(img)

        results['filename'] = filename
        results['img'] = img
        results['img_shape'] = img.shape
        results['ori_shape'] = img.shape
        results['flip'] = False

        # Set initial values for default meta_keys
        results['pad_shape'] = img.shape
        results['scale_factor'] = 1.0
        num_channels = 1 if len(img.shape) < 3 else img.shape[2]
        results['img_norm_cfg'] = dict(
            mean=np.zeros(num_channels, dtype=np.float32),
            std=np.ones(num_channels, dtype=np.float32),
            to_rgb=False)
        return results

Dataset

My dataset is a custom dataset which comprises of images of particles of different shapes and sizes. There is only 1 class of object, particle.

import logging
import os.path as osp
import tempfile
import datetime
import time
from collections import defaultdict
import copy

import mmcv
import numpy as np
from mmcv.utils import print_log
from pycocotools.coco import COCO
from pycocotools import mask as maskUtils
#from pycocotools.cocoeval import COCOeval
from terminaltables import AsciiTable

from mmdet.core import eval_recalls
from mmdet.datasets.builder import DATASETS
from mmdet.datasets.coco import CocoDataset

@DATASETS.register_module()
class MorphologiDataset(CocoDataset):
    '''
    Sub Class of CocoDataset for bespoke functions for Morphologi data.

    Ed Morris
    Malvern Panalytical
    (c) 2020
    '''

    CLASSES = ('particle')

    # Amended version of the CocoDataset evaluate method for Morphologi specific applications.
    def evaluate(self,
                 results,
                 metric='bbox',
                 logger=None,
                 jsonfile_prefix=None,
                 classwise=False,
                 proposal_nums=(100, 300, 1000),
                 iou_thrs=np.arange(0.5, 0.96, 0.05)):
        """Evaluation in COCO protocol.

        Args:
            results (list): Testing results of the dataset.
            metric (str | list[str]): Metrics to be evaluated.
            logger (logging.Logger | str | None): Logger used for printing
                related information during evaluation. Default: None.
            jsonfile_prefix (str | None): The prefix of json files. It includes
                the file path and the prefix of filename, e.g., "a/b/prefix".
                If not specified, a temp file will be created. Default: None.
            classwise (bool): Whether to evaluating the AP for each class.
            proposal_nums (Sequence[int]): Proposal number used for evaluating
                recalls, such as recall@100, recall@1000.
                Default: (100, 300, 1000).
            iou_thrs (Sequence[float]): IoU threshold used for evaluating
                recalls. If set to a list, the average recall of all IoUs will
                also be computed. Default: 0.5.

        Returns:
            dict[str: float]
        """

        metrics = metric if isinstance(metric, list) else [metric]
        allowed_metrics = ['bbox', 'segm', 'proposal', 'proposal_fast']
        for metric in metrics:
            if metric not in allowed_metrics:
                raise KeyError(f'metric {metric} is not supported')

        result_files, tmp_dir = self.format_results(results, jsonfile_prefix)

        eval_results = {}
        cocoGt = self.coco
        for metric in metrics:
            msg = f'Evaluating {metric}...'
            if logger is None:
                msg = '\n' + msg
            print_log(msg, logger=logger)

            if metric == 'proposal_fast':
                ar = self.fast_eval_recall(
                    results, proposal_nums, iou_thrs, logger='silent')
                log_msg = []
                for i, num in enumerate(proposal_nums):
                    eval_results[f'AR@{num}'] = ar[i]
                    log_msg.append(f'\nAR@{num}\t{ar[i]:.4f}')
                log_msg = ''.join(log_msg)
                print_log(log_msg, logger=logger)
                continue

            if metric not in result_files:
                raise KeyError(f'{metric} is not in results')
            try:
                cocoDt = cocoGt.loadRes(result_files[metric])
            except IndexError:
                print_log(
                    'The testing results of the whole dataset is empty.',
                    logger=logger,
                    level=logging.ERROR)
                break

            iou_type = 'bbox' if metric == 'proposal' else metric
            cocoEval = COCOeval(cocoGt, cocoDt, iou_type)
            cocoEval.params.catIds = self.cat_ids
            cocoEval.params.imgIds = self.img_ids
            if metric == 'proposal':
                cocoEval.params.useCats = 0
                cocoEval.params.maxDets = list(proposal_nums)
                cocoEval.evaluate()
                cocoEval.accumulate()
                cocoEval.summarize()
                metric_items = [
                    'AR@100', 'AR@300', 'AR@1000', 'AR_s@1000', 'AR_m@1000',
                    'AR_l@1000'
                ]
                for i, item in enumerate(metric_items):
                    val = float(f'{cocoEval.stats[i + 6]:.3f}')
                    eval_results[item] = val
            else:
                cocoEval.evaluate()
                cocoEval.accumulate()
                cocoEval.summarize()
                if classwise:  # Compute per-category AP
                    # Compute per-category AP
                    # from https://github.com/facebookresearch/detectron2/
                    precisions = cocoEval.eval['precision']
                    # precision: (iou, recall, cls, area range, max dets)
                    assert len(self.cat_ids) == precisions.shape[2]

                    results_per_category = []
                    for idx, catId in enumerate(self.cat_ids):
                        # area range index 0: all area ranges
                        # max dets index -1: typically 100 per image
                        nm = self.coco.loadCats(catId)[0]
                        precision = precisions[:, :, idx, 0, -1]
                        precision = precision[precision > -1]
                        if precision.size:
                            ap = np.mean(precision)
                        else:
                            ap = float('nan')
                        results_per_category.append(
                            (f'{nm["name"]}', f'{float(ap):0.3f}'))

                    num_columns = min(6, len(results_per_category) * 2)
                    results_flatten = list(
                        itertools.chain(*results_per_category))
                    headers = ['category', 'AP'] * (num_columns // 2)
                    results_2d = itertools.zip_longest(*[
                        results_flatten[i::num_columns]
                        for i in range(num_columns)
                    ])
                    table_data = [headers]
                    table_data += [result for result in results_2d]
                    table = AsciiTable(table_data)
                    print_log('\n' + table.table, logger=logger)

                metric_items = [
                    'mAP', 'mAP_50', 'mAP_75', 'mAP_s', 'mAP_m', 'mAP_l'
                ]
                for i in range(len(metric_items)):
                    key = f'{metric}_{metric_items[i]}'
                    val = float(f'{cocoEval.stats[i]:.3f}')
                    eval_results[key] = val
                ap = cocoEval.stats[:6]
                eval_results[f'{metric}_mAP_copypaste'] = (
                    f'{ap[0]:.3f} {ap[1]:.3f} {ap[2]:.3f} {ap[3]:.3f} '
                    f'{ap[4]:.3f} {ap[5]:.3f}')
        if tmp_dir is not None:
            tmp_dir.cleanup()
        return eval_results

Environment

sys.platform: linux
Python: 3.7.6 | packaged by conda-forge | (default, Mar  5 2020, 15:27:18) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.243
GPU 0: Tesla P40
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.4.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.5.0
OpenCV: 4.2.0
MMCV: 0.5.1
MMDetection: 2.0.0+6603790
MMDetection Compiler: GCC 7.5
MMDetection CUDA Compiler: 10.1

Output logs of training

2020-08-12 20:14:43,082 - mmdet - INFO - Created output directory: output/mixed_NANs/cuboidal_square_images_v1p2_MaskRCNN_ResNet50_FPN_12082020_201443
2020-08-12 20:14:43,083 - mmdet - INFO - Command line arguments passed to training script: 
2020-08-12 20:14:43,083 - mmdet - INFO - config :: configs_morph/mmdetection_morphologi_mask_rcnn_r50_fpn_1x.py
2020-08-12 20:14:43,083 - mmdet - INFO - work_dir :: output/mixed_NANs
2020-08-12 20:14:43,083 - mmdet - INFO - workflow :: 2
2020-08-12 20:14:43,083 - mmdet - INFO - job_name :: cuboidal_square_images_v1p2_
2020-08-12 20:14:43,083 - mmdet - INFO - resume_from :: None
2020-08-12 20:14:43,083 - mmdet - INFO - validate :: True
2020-08-12 20:14:43,083 - mmdet - INFO - gpus :: None
2020-08-12 20:14:43,083 - mmdet - INFO - gpu_ids :: None
2020-08-12 20:14:43,083 - mmdet - INFO - seed :: 42
2020-08-12 20:14:43,083 - mmdet - INFO - deterministic :: False
2020-08-12 20:14:43,083 - mmdet - INFO - options :: None
2020-08-12 20:14:43,083 - mmdet - INFO - launcher :: none
2020-08-12 20:14:43,083 - mmdet - INFO - local_rank :: 0
2020-08-12 20:14:43,083 - mmdet - INFO - autoscale_lr :: True
2020-08-12 20:14:43,083 - mmdet - INFO - Applying linear Learning Rate correction. LR changed from 0.02 to 0.0025
2020-08-12 20:14:43,084 - mmdet - INFO - Distributed environment has not been initialized.
2020-08-12 20:14:43,084 - mmdet - INFO - Setting workflow to train and validate [('train', 1), ('val', 1)]
2020-08-12 20:14:43,084 - mmdet - INFO - COCO Validation will be performed after every 1 training epoch(s) using metrics bbox
2020-08-12 20:14:43,262 - mmdet - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.7.6 | packaged by conda-forge | (default, Mar  5 2020, 15:27:18) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.243
GPU 0: Tesla P40
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.4.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF, 

TorchVision: 0.5.0
OpenCV: 4.2.0
MMCV: 0.5.1
MMDetection: 2.0.0+6603790
MMDetection Compiler: GCC 7.5
MMDetection CUDA Compiler: 10.1
------------------------------------------------------------

2020-08-12 20:14:43,263 - mmdet - INFO - Distributed training: False
2020-08-12 20:14:43,263 - mmdet - INFO - Config:
/datadrive/drive1/Projects/WillowTree/Repo/advanced_seg/MMDetection_experiments/configs/_base_/models/mask_rcnn_r50_fpn.py
# model settings
model = dict(
    type='MaskRCNN',
    pretrained='torchvision://resnet50',
    backbone=dict(
        type='ResNet',
        depth=50,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=1,
        norm_cfg=dict(type='BN', requires_grad=True),
        norm_eval=True,
        style='pytorch'),
    neck=dict(
        type='FPN',
        in_channels=[256, 512, 1024, 2048],
        out_channels=256,
        num_outs=5),
    rpn_head=dict(
        type='RPNHead',
        in_channels=256,
        feat_channels=256,
        anchor_generator=dict(
            type='AnchorGenerator',
            scales=[8],
            ratios=[0.5, 1.0, 2.0],
            strides=[4, 8, 16, 32, 64]),
        bbox_coder=dict(
            type='DeltaXYWHBBoxCoder',
            target_means=[.0, .0, .0, .0],
            target_stds=[1.0, 1.0, 1.0, 1.0]),
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
        loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
    roi_head=dict(
        type='StandardRoIHead',
        bbox_roi_extractor=dict(
            type='SingleRoIExtractor',
            roi_layer=dict(type='RoIAlign', out_size=7, sample_num=0),
            out_channels=256,
            featmap_strides=[4, 8, 16, 32]),
        bbox_head=dict(
            type='Shared2FCBBoxHead',
            in_channels=256,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=80,
            bbox_coder=dict(
                type='DeltaXYWHBBoxCoder',
                target_means=[0., 0., 0., 0.],
                target_stds=[0.1, 0.1, 0.2, 0.2]),
            reg_class_agnostic=False,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
        mask_roi_extractor=dict(
            type='SingleRoIExtractor',
            roi_layer=dict(type='RoIAlign', out_size=14, sample_num=0),
            out_channels=256,
            featmap_strides=[4, 8, 16, 32]),
        mask_head=dict(
            type='FCNMaskHead',
            num_convs=4,
            in_channels=256,
            conv_out_channels=256,
            num_classes=80,
            loss_mask=dict(
                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0))))
# model training and testing settings
train_cfg = dict(
    rpn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.7,
            neg_iou_thr=0.3,
            min_pos_iou=0.3,
            match_low_quality=True,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=256,
            pos_fraction=0.5,
            neg_pos_ub=-1,
            add_gt_as_proposals=False),
        allowed_border=-1,
        pos_weight=-1,
        debug=False),
    rpn_proposal=dict(
        nms_across_levels=False,
        nms_pre=2000,
        nms_post=1000,
        max_num=1000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.5,
            neg_iou_thr=0.5,
            min_pos_iou=0.5,
            match_low_quality=True,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=512,
            pos_fraction=0.25,
            neg_pos_ub=-1,
            add_gt_as_proposals=True),
        mask_size=28,
        pos_weight=-1,
        debug=False))
test_cfg = dict(
    rpn=dict(
        nms_across_levels=False,
        nms_pre=1000,
        nms_post=1000,
        max_num=1000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        score_thr=0.05,
        nms=dict(type='nms', iou_thr=0.5),
        max_per_img=100,
        mask_thr_binary=0.5))

/datadrive/drive1/Projects/WillowTree/Repo/advanced_seg/MMDetection_experiments/configs/_base_/datasets/coco_detection.py
dataset_type = 'CocoDataset'
data_root = 'advanced_seg/MMDetection_experiments/public_datasets/coco/'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1333, 800),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
data = dict(
    samples_per_gpu=2,
    workers_per_gpu=2,
    train=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_train2017.json',
        img_prefix=data_root + 'train2017/',
        pipeline=train_pipeline),
    val=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_val2017.json',
        img_prefix=data_root + 'val2017/',
        pipeline=test_pipeline),
    test=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_val2017.json',
        img_prefix=data_root + 'val2017/',
        pipeline=test_pipeline))
evaluation = dict(interval=1, metric='bbox')

/datadrive/drive1/Projects/WillowTree/Repo/advanced_seg/MMDetection_experiments/configs/_base_/datasets/coco_instance.py
_base_ = 'coco_detection.py'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
    dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks']),
]
data = dict(train=dict(pipeline=train_pipeline))
evaluation = dict(metric=['bbox', 'segm'])

/datadrive/drive1/Projects/WillowTree/Repo/advanced_seg/MMDetection_experiments/configs/_base_/schedules/schedule_1x.py
# optimizer
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
# learning policy
lr_config = dict(
    policy='step',
    warmup='linear',
    warmup_iters=500,
    warmup_ratio=0.001,
    step=[8, 11])
total_epochs = 12

/datadrive/drive1/Projects/WillowTree/Repo/advanced_seg/MMDetection_experiments/configs/_base_/default_runtime.py
checkpoint_config = dict(interval=1)
# yapf:disable
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        dict(type='TensorboardLoggerHook')
    ])
# yapf:enable
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]

/datadrive/drive1/Projects/WillowTree/Repo/advanced_seg/MMDetection_experiments/configs_morph/mmdetection_morphologi_mask_rcnn_r50_fpn_1x.py
_base_ = [
    '../configs/_base_/models/mask_rcnn_r50_fpn.py',
    '../configs/_base_/datasets/coco_instance.py',
    '../configs/_base_/schedules/schedule_1x.py', 
    '../configs/_base_/default_runtime.py'
]
dataset_type = 'MorphologiDataset' # 'CocoDataset'
data_root_dir = '/datadrive/drive2/willow_tree_synthetic/mpdsim/train_test_coco_fmt/' # '/datadrive/drive2/willow_tree_synthetic/mpdsim/train_test_coco_fmt/' #'/datadrive/drive0/willow_tree_synthetic/mpdsim/train_test_coco_fmt/'
data_dir = 'cuboidal_square_images_v1p2/' # 'spherical_v1_cuboidal_square_images_v1p1_combined_500_125_4536/' # 'test_dataset_v1p2_100/' #
data_root = data_root_dir+data_dir
classes=['particle']
# Update model due to classes
model = dict(
    pretrained=None, # Don't load the pretrained weights. TODO Find out if this is needed changing number of classes.
    roi_head=dict(
        bbox_head=dict(
            type='Shared2FCBBoxHead',
            in_channels=256,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=len(classes), # Modified to number of classes in this problem. COCO default is 80.
            bbox_coder=dict(
                type='DeltaXYWHBBoxCoder',
                target_means=[0., 0., 0., 0.],
                target_stds=[0.1, 0.1, 0.2, 0.2]),
            reg_class_agnostic=False,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)), # This is L1Loss in standard COCO, specified this way in CityScapes
        mask_head=dict(
            type='FCNMaskHead',
            num_convs=4,
            in_channels=256,
            conv_out_channels=256,
            num_classes=len(classes), # Modified to number of classes in this problem. COCO default is 80.
            loss_mask=dict(
                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0)))
)
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadMorphologiSynImage'),
    dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
    dict(type='Resize', img_scale=(1296, 972), keep_ratio=True),
    dict(type='Normalize', **img_norm_cfg),
    #dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks']),
]
test_pipeline = [
    dict(type='LoadMorphologiSynImage'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1296, 972),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='Normalize', **img_norm_cfg),
            #dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
data = dict(
    samples_per_gpu=2, # TODO Usually set to 2, set to 1 for debugging.
    imgs_per_gpu=2, # TODO Usually set to 2, set to 1 for debugging.
    workers_per_gpu=4,
    train=dict(
        type=dataset_type,
        classes=classes,
        ann_file=data_root + 'annotations/train_coco_changed_removed.json',
        img_prefix=data_root + 'train/',
        pipeline=train_pipeline),
    val=dict(
        type=dataset_type,
        classes=classes,
        ann_file=data_root + 'annotations/valid_coco_changed_removed.json',
        img_prefix=data_root + 'valid/',
        pipeline=test_pipeline),
    test=dict(
        type=dataset_type,
        classes=classes,
        ann_file=data_root + 'annotations/valid_coco_changed_removed.json',
        img_prefix=data_root + 'valid/',
        pipeline=test_pipeline))
test_cfg = dict(
    rpn=dict(
        nms_across_levels=False,
        nms_pre=1000,
        nms_post=1000,
        max_num=1000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        score_thr=0.05,
        nms=dict(type='nms', iou_thr=0.5),
        max_per_img=500, # Default is 100
        mask_thr_binary=0.5)) # Default is 0.5
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
checkpoint_config = dict(type='CheckpointHook', interval=1)
# checkpoint_config = dict(type='ValidationCheckpointHook', metric='acc', metric_ops='max', overwrite_checkpoint=True)
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        dict(type='TensorboardLoggerHook'),
        #dict(type='TensorboardLoggerWithImagesHook')
    ])
total_epochs = 36  # Usually 36
lr_config = dict(step=[24, 33]) # Based on 8 and 11 for 12 Epochs.
gpus = 1
evaluation = dict(interval=1, metric='bbox') # TODO Remove, temporarily here so that evaluation is done on BBOX only at the moment as SEGM not working.

2020-08-12 20:14:43,263 - mmdet - INFO - Set random seed to 42, deterministic: False
2020-08-12 20:15:38,239 - mmdet - INFO - Start running, host: edmorris@willow-tree-cnn-gpu-lin64, work_dir: /datadrive/drive1/Projects/WillowTree/Repo/advanced_seg/MMDetection_experiments/output/mixed_NANs/cuboidal_square_images_v1p2_MaskRCNN_ResNet50_FPN_12082020_201443
2020-08-12 20:15:38,239 - mmdet - INFO - workflow: [('train', 1), ('val', 1)], max: 36 epochs
2020-08-12 20:17:20,058 - mmdet - INFO - Epoch [1][50/1350] lr: 0.00025, eta: 1 day, 3:21:06, time: 2.028, data_time: 1.210, memory: 6279, loss_rpn_cls: 0.6925, loss_rpn_bbox: 0.4201, loss_cls: 0.6367, acc: 68.2461, loss_bbox: 0.0095, loss_mask: 0.6023, loss: 2.3611
2020-08-12 20:18:46,475 - mmdet - INFO - Epoch [1][100/1350]    lr: 0.00050, eta: 1 day, 1:18:19, time: 1.729, data_time: 0.964, memory: 6330, loss_rpn_cls: 0.6860, loss_rpn_bbox: 0.4034, loss_cls: 0.5066, acc: 75.0000, loss_bbox: 0.0044, loss_mask: 0.5108, loss: 2.1113
2020-08-12 20:20:13,878 - mmdet - INFO - Epoch [1][150/1350]    lr: 0.00075, eta: 1 day, 0:41:40, time: 1.748, data_time: 1.001, memory: 6330, loss_rpn_cls: 0.6700, loss_rpn_bbox: 0.3237, loss_cls: 0.4522, acc: 75.0000, loss_bbox: 0.0000, loss_mask: 0.4916, loss: 1.9374
2020-08-12 20:21:43,072 - mmdet - INFO - Epoch [1][200/1350]    lr: 0.00100, eta: 1 day, 0:29:51, time: 1.784, data_time: 1.019, memory: 6330, loss_rpn_cls: 0.6397, loss_rpn_bbox: 0.3017, loss_cls: 0.3590, acc: 82.5938, loss_bbox: 0.0000, loss_mask: 0.4885, loss: 1.7890
2020-08-12 20:23:10,916 - mmdet - INFO - Epoch [1][250/1350]    lr: 0.00125, eta: 1 day, 0:17:49, time: 1.757, data_time: 1.002, memory: 6330, loss_rpn_cls: 0.5973, loss_rpn_bbox: 0.2854, loss_cls: 0.2354, acc: 91.4727, loss_bbox: 0.0001, loss_mask: 0.4638, loss: 1.5821
2020-08-12 20:24:39,622 - mmdet - INFO - Epoch [1][300/1350]    lr: 0.00150, eta: 1 day, 0:11:37, time: 1.774, data_time: 0.992, memory: 6330, loss_rpn_cls: 0.5518, loss_rpn_bbox: 0.2757, loss_cls: 0.5435, acc: 77.8789, loss_bbox: 0.1138, loss_mask: 0.4451, loss: 1.9299
2020-08-12 20:26:07,653 - mmdet - INFO - Epoch [1][350/1350]    lr: 0.00175, eta: 1 day, 0:05:13, time: 1.761, data_time: 0.966, memory: 6330, loss_rpn_cls: 0.4981, loss_rpn_bbox: 0.2600, loss_cls: 0.4828, acc: 75.9883, loss_bbox: 0.2074, loss_mask: 0.4449, loss: 1.8931
2020-08-12 20:27:38,955 - mmdet - INFO - Epoch [1][400/1350]    lr: 0.00200, eta: 1 day, 0:06:37, time: 1.826, data_time: 1.017, memory: 6330, loss_rpn_cls: 0.4123, loss_rpn_bbox: 0.2528, loss_cls: 0.4187, acc: 81.4727, loss_bbox: 0.1780, loss_mask: 0.4237, loss: 1.6854
2020-08-12 20:29:05,153 - mmdet - INFO - Epoch [1][450/1350]    lr: 0.00225, eta: 23:58:16, time: 1.724, data_time: 0.912, memory: 6330, loss_rpn_cls: 0.3439, loss_rpn_bbox: 0.2511, loss_cls: 0.4184, acc: 81.8965, loss_bbox: 0.1747, loss_mask: 0.4297, loss: 1.6178
2020-08-12 20:30:31,107 - mmdet - INFO - Epoch [1][500/1350]    lr: 0.00250, eta: 23:50:54, time: 1.719, data_time: 0.926, memory: 6330, loss_rpn_cls: 0.2689, loss_rpn_bbox: 0.2431, loss_cls: 0.4017, acc: 83.4668, loss_bbox: 0.2034, loss_mask: 0.4103, loss: 1.5274
2020-08-12 20:31:59,629 - mmdet - INFO - Epoch [1][550/1350]    lr: 0.00250, eta: 23:48:22, time: 1.770, data_time: 0.971, memory: 6330, loss_rpn_cls: 0.2768, loss_rpn_bbox: 0.2545, loss_cls: 0.3739, acc: 84.3262, loss_bbox: 0.1961, loss_mask: 0.3998, loss: 1.5011
2020-08-12 20:33:33,121 - mmdet - INFO - Epoch [1][600/1350]    lr: 0.00250, eta: 23:52:37, time: 1.870, data_time: 1.050, memory: 6330, loss_rpn_cls: 0.2575, loss_rpn_bbox: 0.2461, loss_cls: 0.3534, acc: 85.2539, loss_bbox: 0.1930, loss_mask: 0.3980, loss: 1.4479
2020-08-12 20:35:01,433 - mmdet - INFO - Epoch [1][650/1350]    lr: 0.00250, eta: 23:49:37, time: 1.766, data_time: 0.942, memory: 6330, loss_rpn_cls: 0.2545, loss_rpn_bbox: 0.2451, loss_cls: 0.3465, acc: 85.6328, loss_bbox: 0.1939, loss_mask: 0.3795, loss: 1.4195
2020-08-12 20:36:33,201 - mmdet - INFO - Epoch [1][700/1350]    lr: 0.00250, eta: 23:50:47, time: 1.835, data_time: 0.996, memory: 6330, loss_rpn_cls: 0.2238, loss_rpn_bbox: 0.2508, loss_cls: 0.3559, acc: 85.4961, loss_bbox: 0.1990, loss_mask: 0.3813, loss: 1.4107
2020-08-12 20:37:58,402 - mmdet - INFO - Epoch [1][750/1350]    lr: 0.00250, eta: 23:44:36, time: 1.704, data_time: 0.869, memory: 6330, loss_rpn_cls: 0.2272, loss_rpn_bbox: 0.2409, loss_cls: 0.3381, acc: 86.2246, loss_bbox: 0.1995, loss_mask: 0.3734, loss: 1.3792
2020-08-12 20:39:26,776 - mmdet - INFO - Epoch [1][800/1350]    lr: 0.00250, eta: 23:42:10, time: 1.767, data_time: 0.952, memory: 6330, loss_rpn_cls: 0.2275, loss_rpn_bbox: 0.2436, loss_cls: 0.3464, acc: 85.6387, loss_bbox: 0.2013, loss_mask: 0.3795, loss: 1.3983
2020-08-12 20:40:55,800 - mmdet - INFO - Epoch [1][850/1350]    lr: 0.00250, eta: 23:40:28, time: 1.780, data_time: 0.955, memory: 6330, loss_rpn_cls: 0.2347, loss_rpn_bbox: 0.2435, loss_cls: 0.3345, acc: 86.4844, loss_bbox: 0.1900, loss_mask: 0.3760, loss: 1.3787
2020-08-12 20:42:23,167 - mmdet - INFO - Epoch [1][900/1350]    lr: 0.00250, eta: 23:37:19, time: 1.747, data_time: 0.927, memory: 6330, loss_rpn_cls: 0.2175, loss_rpn_bbox: 0.2347, loss_cls: 0.3361, acc: 86.1152, loss_bbox: 0.2017, loss_mask: 0.3831, loss: 1.3731
2020-08-12 20:43:52,049 - mmdet - INFO - Epoch [1][950/1350]    lr: 0.00250, eta: 23:35:37, time: 1.778, data_time: 0.866, memory: 6330, loss_rpn_cls: 0.2234, loss_rpn_bbox: 0.2401, loss_cls: 0.3348, acc: 86.2520, loss_bbox: 0.1882, loss_mask: 0.3756, loss: 1.3621
2020-08-12 20:45:19,942 - mmdet - INFO - Epoch [1][1000/1350]   lr: 0.00250, eta: 23:33:09, time: 1.758, data_time: 0.949, memory: 6330, loss_rpn_cls: 0.2028, loss_rpn_bbox: 0.2283, loss_cls: 0.3251, acc: 86.2871, loss_bbox: 0.1866, loss_mask: 0.3720, loss: 1.3148
2020-08-12 20:46:49,421 - mmdet - INFO - Epoch [1][1050/1350]   lr: 0.00250, eta: 23:31:58, time: 1.790, data_time: 0.963, memory: 6384, loss_rpn_cls: 0.2204, loss_rpn_bbox: 0.2339, loss_cls: 0.3450, acc: 85.6895, loss_bbox: 0.1955, loss_mask: 0.3479, loss: 1.3427
2020-08-12 20:48:19,508 - mmdet - INFO - Epoch [1][1100/1350]   lr: 0.00250, eta: 23:31:13, time: 1.802, data_time: 0.965, memory: 6384, loss_rpn_cls: 0.2100, loss_rpn_bbox: 0.2311, loss_cls: 0.3381, acc: 85.8340, loss_bbox: 0.1923, loss_mask: 0.3751, loss: 1.3467
2020-08-12 20:49:47,515 - mmdet - INFO - Epoch [1][1150/1350]   lr: 0.00250, eta: 23:28:57, time: 1.760, data_time: 0.930, memory: 6384, loss_rpn_cls: 0.2192, loss_rpn_bbox: 0.2357, loss_cls: 0.3468, acc: 85.5977, loss_bbox: 0.1915, loss_mask: 0.3621, loss: 1.3553
2020-08-12 20:51:12,137 - mmdet - INFO - Epoch [1][1200/1350]   lr: 0.00250, eta: 23:24:32, time: 1.692, data_time: 0.869, memory: 6384, loss_rpn_cls: 0.1770, loss_rpn_bbox: 0.2251, loss_cls: 0.3365, acc: 85.9238, loss_bbox: 0.1961, loss_mask: 0.3578, loss: 1.2925
2020-08-12 20:52:37,704 - mmdet - INFO - Epoch [1][1250/1350]   lr: 0.00250, eta: 23:20:57, time: 1.711, data_time: 0.899, memory: 6384, loss_rpn_cls: 0.1960, loss_rpn_bbox: 0.2264, loss_cls: 0.3386, acc: 86.0723, loss_bbox: 0.1852, loss_mask: 0.3668, loss: 1.3129
2020-08-12 20:54:07,510 - mmdet - INFO - Epoch [1][1300/1350]   lr: 0.00250, eta: 23:20:06, time: 1.796, data_time: 0.986, memory: 6384, loss_rpn_cls: 0.2119, loss_rpn_bbox: 0.2280, loss_cls: 0.3524, acc: 85.5195, loss_bbox: 0.1937, loss_mask: 0.3676, loss: 1.3536
2020-08-12 20:55:35,133 - mmdet - INFO - Epoch [1][1350/1350]   lr: 0.00250, eta: 23:17:56, time: 1.752, data_time: 0.940, memory: 6384, loss_rpn_cls: 0.2017, loss_rpn_bbox: 0.2204, loss_cls: 0.3385, acc: 86.0371, loss_bbox: 0.1889, loss_mask: 0.3764, loss: 1.3260
2020-08-12 21:56:54,553 - mmdet - INFO - Evaluating bbox...
2020-08-12 22:06:40,325 - mmdet - INFO - Epoch [1][1350/1350]   lr: 0.00250, bbox_mAP: 0.1100, bbox_mAP_50: 0.2520, bbox_mAP_75: 0.0780, bbox_mAP_s: 0.1140, bbox_mAP_m: 0.1700, bbox_mAP_l: 0.0810, bbox_mAP_copypaste: 0.110 0.252 0.078 0.114 0.170 0.081
2020-08-12 22:21:45,191 - mmdet - INFO - Epoch(train) [1][525]  loss_rpn_cls: 0.1948, loss_rpn_bbox: 0.2209, loss_cls: 0.3860, acc: 83.8903, loss_bbox: 0.1927, loss_mask: 0.3909, loss: 1.3852
2020-08-12 22:23:14,602 - mmdet - INFO - Epoch [2][50/1350] lr: 0.00250, eta: 23:16:45, time: 1.786, data_time: 0.953, memory: 6384, loss_rpn_cls: 0.1844, loss_rpn_bbox: 0.2175, loss_cls: 0.3334, acc: 86.3789, loss_bbox: 0.1883, loss_mask: 0.3640, loss: 1.2877
2020-08-12 22:24:36,954 - mmdet - INFO - Epoch [2][100/1350]    lr: 0.00250, eta: 23:11:48, time: 1.647, data_time: 0.829, memory: 6384, loss_rpn_cls: 0.2057, loss_rpn_bbox: 0.2269, loss_cls: 0.3377, acc: 86.2227, loss_bbox: 0.1869, loss_mask: 0.3694, loss: 1.3266
2020-08-12 22:26:00,932 - mmdet - INFO - Epoch [2][150/1350]    lr: 0.00250, eta: 23:07:55, time: 1.680, data_time: 0.846, memory: 6384, loss_rpn_cls: 0.1909, loss_rpn_bbox: 0.2245, loss_cls: 0.3444, acc: 85.7441, loss_bbox: 0.1903, loss_mask: 0.3516, loss: 1.3016
2020-08-12 22:27:23,509 - mmdet - INFO - Epoch [2][200/1350]    lr: 0.00250, eta: 23:03:30, time: 1.652, data_time: 0.850, memory: 6384, loss_rpn_cls: 0.1983, loss_rpn_bbox: 0.2213, loss_cls: 0.3420, acc: 85.9570, loss_bbox: 0.1897, loss_mask: 0.3503, loss: 1.3016
2020-08-12 22:28:49,537 - mmdet - INFO - Epoch [2][250/1350]    lr: 0.00250, eta: 23:00:57, time: 1.721, data_time: 0.894, memory: 6384, loss_rpn_cls: 0.2017, loss_rpn_bbox: 0.2249, loss_cls: 0.3383, acc: 86.2051, loss_bbox: 0.1889, loss_mask: 0.3544, loss: 1.3082
2020-08-12 22:30:10,143 - mmdet - INFO - Epoch [2][300/1350]    lr: 0.00250, eta: 22:55:55, time: 1.612, data_time: 0.778, memory: 6384, loss_rpn_cls: 0.1949, loss_rpn_bbox: 0.2144, loss_cls: 0.3235, acc: 87.2539, loss_bbox: 0.1815, loss_mask: 0.3544, loss: 1.2687
2020-08-12 22:31:34,695 - mmdet - INFO - Epoch [2][350/1350]    lr: 0.00250, eta: 22:52:54, time: 1.691, data_time: 0.853, memory: 6384, loss_rpn_cls: 0.2046, loss_rpn_bbox: 0.2267, loss_cls: 0.3413, acc: 86.0195, loss_bbox: 0.1828, loss_mask: 0.3640, loss: 1.3195
2020-08-12 22:32:56,271 - mmdet - INFO - Epoch [2][400/1350]    lr: 0.00250, eta: 22:48:39, time: 1.631, data_time: 0.810, memory: 6384, loss_rpn_cls: 0.1961, loss_rpn_bbox: 0.2245, loss_cls: 0.3352, acc: 86.4434, loss_bbox: 0.1843, loss_mask: 0.3425, loss: 1.2826
2020-08-12 22:34:20,813 - mmdet - INFO - Epoch [2][450/1350]    lr: 0.00250, eta: 22:45:51, time: 1.691, data_time: 0.867, memory: 6384, loss_rpn_cls: 0.2041, loss_rpn_bbox: 0.2237, loss_cls: 0.3347, acc: 86.7715, loss_bbox: 0.1860, loss_mask: 0.3446, loss: 1.2932
2020-08-12 22:35:46,663 - mmdet - INFO - Epoch [2][500/1350]    lr: 0.00250, eta: 22:43:40, time: 1.717, data_time: 0.898, memory: 6384, loss_rpn_cls: 0.1821, loss_rpn_bbox: 0.2313, loss_cls: 0.3316, acc: 86.6836, loss_bbox: 0.1814, loss_mask: 0.3604, loss: 1.2867
2020-08-12 22:37:11,117 - mmdet - INFO - Epoch [2][550/1350]    lr: 0.00250, eta: 22:40:57, time: 1.689, data_time: 0.845, memory: 6384, loss_rpn_cls: 0.2102, loss_rpn_bbox: 0.2293, loss_cls: 0.3373, acc: 86.4785, loss_bbox: 0.1838, loss_mask: 0.3494, loss: 1.3100
2020-08-12 22:38:34,918 - mmdet - INFO - Epoch [2][600/1350]    lr: 0.00250, eta: 22:38:03, time: 1.676, data_time: 0.834, memory: 6384, loss_rpn_cls: 0.1993, loss_rpn_bbox: 0.2221, loss_cls: 0.3226, acc: 87.0078, loss_bbox: 0.1810, loss_mask: 0.3473, loss: 1.2724
2020-08-12 22:40:01,944 - mmdet - INFO - Epoch [2][650/1350]    lr: 0.00250, eta: 22:36:29, time: 1.741, data_time: 0.888, memory: 6384, loss_rpn_cls: 0.2001, loss_rpn_bbox: 0.2314, loss_cls: 0.3414, acc: 86.1992, loss_bbox: 0.1878, loss_mask: 0.3593, loss: 1.3200
2020-08-12 22:41:22,119 - mmdet - INFO - Epoch [2][700/1350]    lr: 0.00250, eta: 22:32:19, time: 1.603, data_time: 0.761, memory: 6384, loss_rpn_cls: 0.1860, loss_rpn_bbox: 0.2246, loss_cls: 0.3270, acc: 86.8066, loss_bbox: 0.1815, loss_mask: 0.3632, loss: 1.2823
2020-08-12 22:42:45,453 - mmdet - INFO - Epoch [2][750/1350]    lr: 0.00250, eta: 22:29:27, time: 1.667, data_time: 0.849, memory: 6384, loss_rpn_cls: 0.1914, loss_rpn_bbox: 0.2233, loss_cls: 0.3270, acc: 86.8633, loss_bbox: 0.1810, loss_mask: 0.3626, loss: 1.2854
2020-08-12 22:44:05,710 - mmdet - INFO - Epoch [2][800/1350]    lr: 0.00250, eta: 22:25:33, time: 1.605, data_time: 0.783, memory: 6384, loss_rpn_cls: 0.1798, loss_rpn_bbox: 0.2195, loss_cls: 0.3158, acc: 87.1582, loss_bbox: 0.1768, loss_mask: 0.3588, loss: 1.2507
2020-08-12 22:45:33,091 - mmdet - INFO - Epoch [2][850/1350]    lr: 0.00250, eta: 22:24:16, time: 1.748, data_time: 0.907, memory: 6384, loss_rpn_cls: 0.1891, loss_rpn_bbox: 0.2152, loss_cls: 0.3076, acc: 87.7363, loss_bbox: 0.1741, loss_mask: 0.3517, loss: 1.2378
2020-08-12 22:46:54,896 - mmdet - INFO - Epoch [2][900/1350]    lr: 0.00250, eta: 22:21:04, time: 1.636, data_time: 0.801, memory: 6384, loss_rpn_cls: 0.1716, loss_rpn_bbox: 0.2152, loss_cls: 0.3058, acc: 87.6660, loss_bbox: 0.1712, loss_mask: 0.3558, loss: 1.2196
2020-08-12 22:48:18,305 - mmdet - INFO - Epoch [2][950/1350]    lr: 0.00250, eta: 22:18:29, time: 1.668, data_time: 0.843, memory: 6384, loss_rpn_cls: 0.1762, loss_rpn_bbox: 0.2080, loss_cls: 0.3046, acc: 87.4609, loss_bbox: 0.1737, loss_mask: 0.3555, loss: 1.2180
2020-08-12 22:49:40,653 - mmdet - INFO - Epoch [2][1000/1350]   lr: 0.00250, eta: 22:15:36, time: 1.647, data_time: 0.819, memory: 6384, loss_rpn_cls: 0.1849, loss_rpn_bbox: 0.2212, loss_cls: 0.3148, acc: 87.3848, loss_bbox: 0.1789, loss_mask: 0.3434, loss: 1.2432
2020-08-12 22:51:06,370 - mmdet - INFO - Epoch [2][1050/1350]   lr: 0.00250, eta: 22:13:52, time: 1.714, data_time: 0.865, memory: 6384, loss_rpn_cls: 0.2005, loss_rpn_bbox: 0.2310, loss_cls: 0.3353, acc: 86.4863, loss_bbox: 0.1814, loss_mask: 0.3421, loss: 1.2903
2020-08-12 22:52:28,828 - mmdet - INFO - Epoch [2][1100/1350]   lr: 0.00250, eta: 22:11:07, time: 1.649, data_time: 0.788, memory: 6384, loss_rpn_cls: 0.1931, loss_rpn_bbox: 0.2173, loss_cls: 0.3280, acc: 86.6660, loss_bbox: 0.1796, loss_mask: 0.3492, loss: 1.2673
2020-08-12 22:53:56,728 - mmdet - INFO - Epoch [2][1150/1350]   lr: 0.00250, eta: 22:10:05, time: 1.758, data_time: 0.894, memory: 6384, loss_rpn_cls: 0.2092, loss_rpn_bbox: 0.2300, loss_cls: 0.3381, acc: 86.3125, loss_bbox: 0.1826, loss_mask: 0.3472, loss: 1.3071
2020-08-12 22:55:20,605 - mmdet - INFO - Epoch [2][1200/1350]   lr: 0.00250, eta: 22:07:51, time: 1.678, data_time: 0.737, memory: 6384, loss_rpn_cls: 0.1946, loss_rpn_bbox: 0.2186, loss_cls: 0.3247, acc: 86.8516, loss_bbox: 0.1762, loss_mask: 0.3415, loss: 1.2556
2020-08-12 22:56:42,409 - mmdet - INFO - Epoch [2][1250/1350]   lr: 0.00250, eta: 22:05:01, time: 1.636, data_time: 0.822, memory: 6384, loss_rpn_cls: 0.1722, loss_rpn_bbox: 0.2094, loss_cls: 0.3117, acc: 87.4316, loss_bbox: 0.1696, loss_mask: 0.3443, loss: 1.2073
2020-08-12 22:58:04,694 - mmdet - INFO - Epoch [2][1300/1350]   lr: 0.00250, eta: 22:02:23, time: 1.646, data_time: 0.819, memory: 6384, loss_rpn_cls: 0.1766, loss_rpn_bbox: 0.2171, loss_cls: 0.3280, acc: 86.5215, loss_bbox: 0.1769, loss_mask: 0.3503, loss: 1.2489
2020-08-12 22:59:26,941 - mmdet - INFO - Epoch [2][1350/1350]   lr: 0.00250, eta: 21:59:47, time: 1.645, data_time: 0.820, memory: 6384, loss_rpn_cls: 0.1744, loss_rpn_bbox: 0.2189, loss_cls: 0.3257, acc: 86.6289, loss_bbox: 0.1830, loss_mask: 0.3374, loss: 1.2393
2020-08-13 00:14:38,620 - mmdet - INFO - Evaluating bbox...
2020-08-13 00:24:23,646 - mmdet - INFO - Epoch [2][1350/1350]   lr: 0.00250, bbox_mAP: 0.1370, bbox_mAP_50: 0.2960, bbox_mAP_75: 0.1100, bbox_mAP_s: 0.1230, bbox_mAP_m: 0.2080, bbox_mAP_l: 0.1300, bbox_mAP_copypaste: 0.137 0.296 0.110 0.123 0.208 0.130
2020-08-13 00:39:24,093 - mmdet - INFO - Epoch(train) [2][525]  loss_rpn_cls: 0.1777, loss_rpn_bbox: 0.2074, loss_cls: 0.3851, acc: 84.0342, loss_bbox: 0.1734, loss_mask: 0.3460, loss: 1.2896
2020-08-13 00:40:54,183 - mmdet - INFO - Epoch [3][50/1350] lr: 0.00250, eta: 21:59:22, time: 1.799, data_time: 0.973, memory: 6384, loss_rpn_cls: 0.1910, loss_rpn_bbox: 0.2209, loss_cls: 0.3174, acc: 87.4629, loss_bbox: 0.1767, loss_mask: 0.3533, loss: 1.2594
2020-08-13 00:42:14,769 - mmdet - INFO - Epoch [3][100/1350]    lr: 0.00250, eta: 21:56:22, time: 1.612, data_time: 0.784, memory: 6384, loss_rpn_cls: 0.1573, loss_rpn_bbox: 0.2074, loss_cls: 0.3012, acc: 87.6992, loss_bbox: 0.1639, loss_mask: 0.3448, loss: 1.1746
2020-08-13 00:43:39,683 - mmdet - INFO - Epoch [3][150/1350]    lr: 0.00250, eta: 21:54:35, time: 1.698, data_time: 0.857, memory: 6384, loss_rpn_cls: 0.1929, loss_rpn_bbox: 0.2283, loss_cls: 0.3154, acc: 87.3809, loss_bbox: 0.1756, loss_mask: 0.3464, loss: 1.2587
2020-08-13 00:45:01,050 - mmdet - INFO - Epoch [3][200/1350]    lr: 0.00250, eta: 21:51:52, time: 1.627, data_time: 0.802, memory: 6384, loss_rpn_cls: 0.1668, loss_rpn_bbox: 0.2159, loss_cls: 0.3238, acc: 86.6152, loss_bbox: 0.1729, loss_mask: 0.3568, loss: 1.2363
2020-08-13 00:46:25,781 - mmdet - INFO - Epoch [3][250/1350]    lr: 0.00250, eta: 21:50:05, time: 1.695, data_time: 0.858, memory: 6384, loss_rpn_cls: 0.1623, loss_rpn_bbox: 0.2120, loss_cls: 0.3069, acc: 87.7324, loss_bbox: 0.1717, loss_mask: 0.3322, loss: 1.1850
2020-08-13 00:47:47,819 - mmdet - INFO - Epoch [3][300/1350]    lr: 0.00250, eta: 21:47:37, time: 1.641, data_time: 0.799, memory: 6384, loss_rpn_cls: 0.1691, loss_rpn_bbox: 0.2084, loss_cls: 0.3061, acc: 87.6699, loss_bbox: 0.1697, loss_mask: 0.3404, loss: 1.1937
2020-08-13 00:49:12,577 - mmdet - INFO - Epoch [3][350/1350]    lr: 0.00250, eta: 21:45:52, time: 1.695, data_time: 0.865, memory: 6384, loss_rpn_cls: 0.1726, loss_rpn_bbox: 0.2185, loss_cls: 0.3085, acc: 87.3750, loss_bbox: 0.1715, loss_mask: 0.3420, loss: 1.2131
2020-08-13 00:50:33,633 - mmdet - INFO - Epoch [3][400/1350]    lr: 0.00250, eta: 21:43:13, time: 1.621, data_time: 0.787, memory: 6384, loss_rpn_cls: 0.1621, loss_rpn_bbox: 0.2067, loss_cls: 0.3011, acc: 87.9551, loss_bbox: 0.1680, loss_mask: 0.3439, loss: 1.1818
2020-08-13 00:51:58,011 - mmdet - INFO - Epoch [3][450/1350]    lr: 0.00250, eta: 21:41:25, time: 1.688, data_time: 0.845, memory: 6384, loss_rpn_cls: 0.1643, loss_rpn_bbox: 0.2099, loss_cls: 0.3013, acc: 87.8535, loss_bbox: 0.1675, loss_mask: 0.3416, loss: 1.1846
2020-08-13 00:53:21,498 - mmdet - INFO - Epoch [3][500/1350]    lr: 0.00250, eta: 21:39:25, time: 1.670, data_time: 0.823, memory: 6384, loss_rpn_cls: 0.1702, loss_rpn_bbox: 0.2192, loss_cls: 0.3115, acc: 87.4629, loss_bbox: 0.1719, loss_mask: 0.3462, loss: 1.2190
2020-08-13 00:54:43,888 - mmdet - INFO - Epoch [3][550/1350]    lr: 0.00250, eta: 21:37:11, time: 1.648, data_time: 0.807, memory: 6384, loss_rpn_cls: 0.1679, loss_rpn_bbox: 0.2108, loss_cls: 0.2920, acc: 88.1758, loss_bbox: 0.1638, loss_mask: 0.3457, loss: 1.1801
2020-08-13 00:56:07,201 - mmdet - INFO - Epoch [3][600/1350]    lr: 0.00250, eta: 21:35:10, time: 1.666, data_time: 0.848, memory: 6384, loss_rpn_cls: 0.1634, loss_rpn_bbox: 0.2129, loss_cls: 0.3027, acc: 87.8809, loss_bbox: 0.1667, loss_mask: 0.3462, loss: 1.1918
2020-08-13 00:57:33,241 - mmdet - INFO - Epoch [3][650/1350]    lr: 0.00250, eta: 21:33:48, time: 1.721, data_time: 0.872, memory: 6384, loss_rpn_cls: 0.1723, loss_rpn_bbox: 0.2082, loss_cls: 0.3043, acc: 87.7930, loss_bbox: 0.1671, loss_mask: 0.3289, loss: 1.1808
2020-08-13 00:58:55,713 - mmdet - INFO - Epoch [3][700/1350]    lr: 0.00250, eta: 21:31:39, time: 1.649, data_time: 0.800, memory: 6384, loss_rpn_cls: 0.1730, loss_rpn_bbox: 0.2185, loss_cls: 0.3108, acc: 87.3594, loss_bbox: 0.1711, loss_mask: 0.3306, loss: 1.2039
2020-08-13 01:00:20,709 - mmdet - INFO - Epoch [3][750/1350]    lr: 0.00250, eta: 21:30:03, time: 1.700, data_time: 0.848, memory: 6384, loss_rpn_cls: 0.1702, loss_rpn_bbox: 0.2178, loss_cls: 0.3079, acc: 87.4805, loss_bbox: 0.1720, loss_mask: 0.3400, loss: 1.2079
2020-08-13 01:01:45,446 - mmdet - INFO - Epoch [3][800/1350]    lr: 0.00250, eta: 21:28:25, time: 1.695, data_time: 0.837, memory: 6384, loss_rpn_cls: 0.1749, loss_rpn_bbox: 0.2124, loss_cls: 0.3024, acc: 87.5918, loss_bbox: 0.1689, loss_mask: 0.3324, loss: 1.1910
2020-08-13 01:03:12,563 - mmdet - INFO - Epoch [3][850/1350]    lr: 0.00250, eta: 21:27:17, time: 1.742, data_time: 0.898, memory: 6384, loss_rpn_cls: 0.1974, loss_rpn_bbox: 0.2248, loss_cls: 0.3213, acc: 87.0820, loss_bbox: 0.1721, loss_mask: 0.3310, loss: 1.2465
2020-08-13 01:04:37,167 - mmdet - INFO - Epoch [3][900/1350]    lr: 0.00250, eta: 21:25:37, time: 1.692, data_time: 0.856, memory: 6384, loss_rpn_cls: 0.1729, loss_rpn_bbox: 0.2161, loss_cls: 0.3111, acc: 87.2520, loss_bbox: 0.1724, loss_mask: 0.3501, loss: 1.2226
2020-08-13 01:06:03,681 - mmdet - INFO - Epoch [3][950/1350]    lr: 0.00250, eta: 21:24:21, time: 1.730, data_time: 0.882, memory: 6384, loss_rpn_cls: 0.1798, loss_rpn_bbox: 0.2167, loss_cls: 0.3054, acc: 87.6172, loss_bbox: 0.1686, loss_mask: 0.3407, loss: 1.2113
2020-08-13 01:07:24,563 - mmdet - INFO - Epoch [3][1000/1350]   lr: 0.00250, eta: 21:21:57, time: 1.618, data_time: 0.713, memory: 6384, loss_rpn_cls: 0.1621, loss_rpn_bbox: 0.2162, loss_cls: 0.3070, acc: 87.6445, loss_bbox: 0.1667, loss_mask: 0.3522, loss: 1.2042
2020-08-13 01:08:49,644 - mmdet - INFO - Epoch [3][1050/1350]   lr: 0.00250, eta: 21:20:25, time: 1.702, data_time: 0.896, memory: 6384, loss_rpn_cls: 0.1600, loss_rpn_bbox: 0.2106, loss_cls: 0.2904, acc: 87.9785, loss_bbox: 0.1654, loss_mask: 0.3470, loss: 1.1733
2020-08-13 01:10:09,817 - mmdet - INFO - Epoch [3][1100/1350]   lr: 0.00250, eta: 21:17:54, time: 1.603, data_time: 0.771, memory: 6384, loss_rpn_cls: 0.1877, loss_rpn_bbox: 0.2214, loss_cls: 0.3134, acc: 87.1914, loss_bbox: 0.1667, loss_mask: 0.3428, loss: 1.2320
2020-08-13 01:11:34,488 - mmdet - INFO - Epoch [3][1150/1350]   lr: 0.00250, eta: 21:16:18, time: 1.693, data_time: 0.861, memory: 6384, loss_rpn_cls: 0.1833, loss_rpn_bbox: 0.2232, loss_cls: 0.3071, acc: 87.4629, loss_bbox: 0.1689, loss_mask: 0.3533, loss: 1.2358
2020-08-13 01:12:55,780 - mmdet - INFO - Epoch [3][1200/1350]   lr: 0.00250, eta: 21:14:04, time: 1.626, data_time: 0.812, memory: 6384, loss_rpn_cls: 0.1700, loss_rpn_bbox: 0.2074, loss_cls: 0.2937, acc: 88.0117, loss_bbox: 0.1639, loss_mask: 0.3466, loss: 1.1815
2020-08-13 01:14:21,942 - mmdet - INFO - Epoch [3][1250/1350]   lr: 0.00250, eta: 21:12:46, time: 1.723, data_time: 0.880, memory: 6384, loss_rpn_cls: 0.1859, loss_rpn_bbox: 0.2129, loss_cls: 0.3065, acc: 87.7012, loss_bbox: 0.1704, loss_mask: 0.3378, loss: 1.2136
2020-08-13 01:15:44,294 - mmdet - INFO - Epoch [3][1300/1350]   lr: 0.00250, eta: 21:10:45, time: 1.647, data_time: 0.789, memory: 6384, loss_rpn_cls: 0.1672, loss_rpn_bbox: 0.2192, loss_cls: 0.3081, acc: 87.4121, loss_bbox: 0.1654, loss_mask: 0.3353, loss: 1.1952
2020-08-13 01:17:09,430 - mmdet - INFO - Epoch [3][1350/1350]   lr: 0.00250, eta: 21:09:16, time: 1.703, data_time: 0.865, memory: 6384, loss_rpn_cls: 0.1768, loss_rpn_bbox: 0.2115, loss_cls: 0.3097, acc: 87.3984, loss_bbox: 0.1720, loss_mask: 0.3360, loss: 1.2060
2020-08-13 02:17:47,557 - mmdet - INFO - Evaluating bbox...
2020-08-13 02:27:34,041 - mmdet - INFO - Epoch [3][1350/1350]   lr: 0.00250, bbox_mAP: 0.1480, bbox_mAP_50: 0.3140, bbox_mAP_75: 0.1220, bbox_mAP_s: 0.1290, bbox_mAP_m: 0.2370, bbox_mAP_l: 0.1430, bbox_mAP_copypaste: 0.148 0.314 0.122 0.129 0.237 0.143
2020-08-13 02:42:40,429 - mmdet - INFO - Epoch(train) [3][525]  loss_rpn_cls: 0.1599, loss_rpn_bbox: 0.1996, loss_cls: 0.3589, acc: 84.9756, loss_bbox: 0.1672, loss_mask: 0.3401, loss: 1.2257
2020-08-13 02:44:12,575 - mmdet - INFO - Epoch [4][50/1350] lr: 0.00250, eta: 21:09:01, time: 1.840, data_time: 1.028, memory: 6384, loss_rpn_cls: 0.1843, loss_rpn_bbox: 0.2180, loss_cls: 0.3099, acc: 87.4961, loss_bbox: 0.1684, loss_mask: 0.3390, loss: 1.2195
2020-08-13 02:45:33,774 - mmdet - INFO - Epoch [4][100/1350]    lr: 0.00250, eta: 21:06:49, time: 1.624, data_time: 0.820, memory: 6384, loss_rpn_cls: 0.1480, loss_rpn_bbox: 0.2032, loss_cls: 0.2846, acc: 88.6094, loss_bbox: 0.1563, loss_mask: 0.3361, loss: 1.1281
2020-08-13 02:46:55,306 - mmdet - INFO - Epoch [4][150/1350]    lr: 0.00250, eta: 21:04:42, time: 1.631, data_time: 0.860, memory: 6384, loss_rpn_cls: 0.1621, loss_rpn_bbox: 0.2090, loss_cls: 0.3044, acc: 87.6211, loss_bbox: 0.1698, loss_mask: 0.3413, loss: 1.1866
2020-08-13 02:48:17,529 - mmdet - INFO - Epoch [4][200/1350]    lr: 0.00250, eta: 21:02:42, time: 1.644, data_time: 0.858, memory: 6384, loss_rpn_cls: 0.1577, loss_rpn_bbox: 0.2054, loss_cls: 0.2921, acc: 88.0723, loss_bbox: 0.1635, loss_mask: 0.3463, loss: 1.1649
2020-08-13 02:49:41,162 - mmdet - INFO - Epoch [4][250/1350]    lr: 0.00250, eta: 21:00:59, time: 1.673, data_time: 0.868, memory: 6384, loss_rpn_cls: 0.1778, loss_rpn_bbox: 0.2155, loss_cls: 0.2981, acc: 87.9355, loss_bbox: 0.1674, loss_mask: 0.3375, loss: 1.1963
2020-08-13 02:51:07,062 - mmdet - INFO - Epoch [4][300/1350]    lr: 0.00250, eta: 20:59:38, time: 1.718, data_time: 0.903, memory: 6384, loss_rpn_cls: 0.1689, loss_rpn_bbox: 0.2109, loss_cls: 0.3017, acc: 87.7109, loss_bbox: 0.1632, loss_mask: 0.3340, loss: 1.1786
2020-08-13 02:52:27,571 - mmdet - INFO - Epoch [4][350/1350]    lr: 0.00250, eta: 20:57:24, time: 1.610, data_time: 0.792, memory: 6384, loss_rpn_cls: 0.1565, loss_rpn_bbox: 0.2073, loss_cls: 0.2933, acc: 88.2129, loss_bbox: 0.1603, loss_mask: 0.3371, loss: 1.1545
2020-08-13 02:53:52,832 - mmdet - INFO - Epoch [4][400/1350]    lr: 0.00250, eta: 20:55:58, time: 1.705, data_time: 0.886, memory: 6384, loss_rpn_cls: 0.1482, loss_rpn_bbox: 0.1978, loss_cls: 0.2930, acc: 88.2344, loss_bbox: 0.1612, loss_mask: 0.3373, loss: 1.1376
2020-08-13 02:55:14,857 - mmdet - INFO - Epoch [4][450/1350]    lr: 0.00250, eta: 20:54:00, time: 1.640, data_time: 0.823, memory: 6384, loss_rpn_cls: 0.1697, loss_rpn_bbox: 0.2181, loss_cls: 0.2986, acc: 87.8438, loss_bbox: 0.1667, loss_mask: 0.3418, loss: 1.1949
2020-08-13 02:56:39,869 - mmdet - INFO - Epoch [4][500/1350]    lr: 0.00250, eta: 20:52:32, time: 1.700, data_time: 0.884, memory: 6384, loss_rpn_cls: 0.1667, loss_rpn_bbox: 0.2194, loss_cls: 0.3123, acc: 87.1230, loss_bbox: 0.1691, loss_mask: 0.3414, loss: 1.2089
2020-08-13 02:58:05,884 - mmdet - INFO - Epoch [4][550/1350]    lr: 0.00250, eta: 20:51:13, time: 1.720, data_time: 0.896, memory: 6384, loss_rpn_cls: 0.1492, loss_rpn_bbox: 0.2059, loss_cls: 0.2903, acc: 88.1211, loss_bbox: 0.1532, loss_mask: 0.3402, loss: 1.1387
2020-08-13 02:59:33,955 - mmdet - INFO - Epoch [4][600/1350]    lr: 0.00250, eta: 20:50:14, time: 1.761, data_time: 0.922, memory: 6384, loss_rpn_cls: 0.1545, loss_rpn_bbox: 0.2024, loss_cls: 0.2868, acc: 88.3809, loss_bbox: 0.1589, loss_mask: 0.3255, loss: 1.1282
2020-08-13 03:00:57,859 - mmdet - INFO - Epoch [4][650/1350]    lr: 0.00250, eta: 20:48:35, time: 1.678, data_time: 0.829, memory: 6384, loss_rpn_cls: 0.1555, loss_rpn_bbox: 0.2071, loss_cls: 0.2882, acc: 88.4453, loss_bbox: 0.1565, loss_mask: 0.3266, loss: 1.1338
2020-08-13 03:02:21,920 - mmdet - INFO - Epoch [4][700/1350]    lr: 0.00250, eta: 20:46:58, time: 1.681, data_time: 0.856, memory: 6384, loss_rpn_cls: 0.1594, loss_rpn_bbox: 0.2066, loss_cls: 0.2984, acc: 87.8047, loss_bbox: 0.1679, loss_mask: 0.3381, loss: 1.1704
2020-08-13 03:03:47,110 - mmdet - INFO - Epoch [4][750/1350]    lr: 0.00250, eta: 20:45:32, time: 1.704, data_time: 0.879, memory: 6384, loss_rpn_cls: 0.1665, loss_rpn_bbox: 0.2155, loss_cls: 0.2989, acc: 87.7812, loss_bbox: 0.1655, loss_mask: 0.3356, loss: 1.1819
2020-08-13 03:05:12,107 - mmdet - INFO - Epoch [4][800/1350]    lr: 0.00250, eta: 20:44:04, time: 1.700, data_time: 0.858, memory: 6384, loss_rpn_cls: 0.1605, loss_rpn_bbox: 0.2114, loss_cls: 0.3082, acc: 87.4922, loss_bbox: 0.1674, loss_mask: 0.3233, loss: 1.1708
2020-08-13 03:06:35,380 - mmdet - INFO - Epoch [4][850/1350]    lr: 0.00250, eta: 20:42:20, time: 1.665, data_time: 0.815, memory: 6384, loss_rpn_cls: 0.1749, loss_rpn_bbox: 0.2203, loss_cls: 0.3095, acc: 87.4219, loss_bbox: 0.1668, loss_mask: 0.3371, loss: 1.2086
2020-08-13 03:07:56,289 - mmdet - INFO - Epoch [4][900/1350]    lr: 0.00250, eta: 20:40:16, time: 1.618, data_time: 0.815, memory: 6384, loss_rpn_cls: 0.1503, loss_rpn_bbox: 0.2007, loss_cls: 0.2813, acc: 88.5098, loss_bbox: 0.1571, loss_mask: 0.3535, loss: 1.1429
2020-08-13 03:09:20,632 - mmdet - INFO - Epoch [4][950/1350]    lr: 0.00250, eta: 20:38:43, time: 1.687, data_time: 0.860, memory: 6384, loss_rpn_cls: 0.1586, loss_rpn_bbox: 0.2020, loss_cls: 0.2897, acc: 88.0645, loss_bbox: 0.1577, loss_mask: 0.3355, loss: 1.1435
2020-08-13 03:10:50,813 - mmdet - INFO - Epoch [4][1000/1350]   lr: 0.00250, eta: 20:38:01, time: 1.804, data_time: 0.950, memory: 6384, loss_rpn_cls: 0.1640, loss_rpn_bbox: 0.2084, loss_cls: 0.2981, acc: 87.9277, loss_bbox: 0.1660, loss_mask: 0.3398, loss: 1.1762
2020-08-13 03:12:13,778 - mmdet - INFO - Epoch [4][1050/1350]   lr: 0.00250, eta: 20:36:15, time: 1.659, data_time: 0.805, memory: 6384, loss_rpn_cls: 0.1602, loss_rpn_bbox: 0.2054, loss_cls: 0.2912, acc: 88.0332, loss_bbox: 0.1578, loss_mask: 0.3423, loss: 1.1570
2020-08-13 03:13:35,293 - mmdet - INFO - Epoch [4][1100/1350]   lr: 0.00250, eta: 20:34:19, time: 1.630, data_time: 0.783, memory: 6384, loss_rpn_cls: 0.1649, loss_rpn_bbox: 0.2138, loss_cls: 0.3087, acc: 87.5137, loss_bbox: 0.1714, loss_mask: 0.3393, loss: 1.1980
2020-08-13 03:14:56,381 - mmdet - INFO - Epoch [4][1150/1350]   lr: 0.00250, eta: 20:32:19, time: 1.622, data_time: 0.768, memory: 6384, loss_rpn_cls: 0.1607, loss_rpn_bbox: 0.2045, loss_cls: 0.2923, acc: 88.0586, loss_bbox: 0.1615, loss_mask: 0.3334, loss: 1.1523
2020-08-13 03:16:22,689 - mmdet - INFO - Epoch [4][1200/1350]   lr: 0.00250, eta: 20:31:03, time: 1.726, data_time: 0.879, memory: 6384, loss_rpn_cls: 0.1547, loss_rpn_bbox: 0.2053, loss_cls: 0.2901, acc: 88.1914, loss_bbox: 0.1620, loss_mask: 0.3334, loss: 1.1455
2020-08-13 03:17:44,290 - mmdet - INFO - Epoch [4][1250/1350]   lr: 0.00250, eta: 20:29:08, time: 1.632, data_time: 0.780, memory: 6384, loss_rpn_cls: 0.1613, loss_rpn_bbox: 0.2144, loss_cls: 0.2919, acc: 88.0000, loss_bbox: 0.1586, loss_mask: 0.3432, loss: 1.1694
2020-08-13 03:19:08,199 - mmdet - INFO - Epoch [4][1300/1350]   lr: 0.00250, eta: 20:27:33, time: 1.678, data_time: 0.839, memory: 6384, loss_rpn_cls: 0.1600, loss_rpn_bbox: 0.2133, loss_cls: 0.3008, acc: 87.7090, loss_bbox: 0.1590, loss_mask: 0.3304, loss: 1.1635
2020-08-13 03:20:31,714 - mmdet - INFO - Epoch [4][1350/1350]   lr: 0.00250, eta: 20:25:55, time: 1.670, data_time: 0.825, memory: 6384, loss_rpn_cls: 0.1676, loss_rpn_bbox: 0.2046, loss_cls: 0.2894, acc: 88.2871, loss_bbox: 0.1578, loss_mask: 0.3356, loss: 1.1550
2020-08-13 04:20:48,743 - mmdet - INFO - Evaluating bbox...
2020-08-13 04:30:42,148 - mmdet - INFO - Epoch [4][1350/1350]   lr: 0.00250, bbox_mAP: 0.1620, bbox_mAP_50: 0.3390, bbox_mAP_75: 0.1360, bbox_mAP_s: 0.1410, bbox_mAP_m: 0.2500, bbox_mAP_l: 0.1880, bbox_mAP_copypaste: 0.162 0.339 0.136 0.141 0.250 0.188
2020-08-13 04:45:46,685 - mmdet - INFO - Epoch(train) [4][525]  loss_rpn_cls: 0.1593, loss_rpn_bbox: 0.2003, loss_cls: 0.3406, acc: 85.6038, loss_bbox: 0.1597, loss_mask: 0.3334, loss: 1.1932
2020-08-13 04:47:17,568 - mmdet - INFO - Epoch [5][50/1350] lr: 0.00250, eta: 20:25:14, time: 1.815, data_time: 0.999, memory: 6384, loss_rpn_cls: 0.1670, loss_rpn_bbox: 0.2130, loss_cls: 0.2958, acc: 87.9160, loss_bbox: 0.1601, loss_mask: 0.3328, loss: 1.1688
2020-08-13 04:48:40,525 - mmdet - INFO - Epoch [5][100/1350]    lr: 0.00250, eta: 20:23:32, time: 1.659, data_time: 0.834, memory: 6384, loss_rpn_cls: 0.1501, loss_rpn_bbox: 0.2033, loss_cls: 0.2907, acc: 88.0801, loss_bbox: 0.1612, loss_mask: 0.3319, loss: 1.1372
2020-08-13 04:50:06,121 - mmdet - INFO - Epoch [5][150/1350]    lr: 0.00250, eta: 20:22:10, time: 1.712, data_time: 0.880, memory: 6384, loss_rpn_cls: 0.1665, loss_rpn_bbox: 0.2166, loss_cls: 0.2976, acc: 87.8281, loss_bbox: 0.1632, loss_mask: 0.3285, loss: 1.1724
2020-08-13 04:51:30,548 - mmdet - INFO - Epoch [5][200/1350]    lr: 0.00250, eta: 20:20:39, time: 1.689, data_time: 0.849, memory: 6384, loss_rpn_cls: 0.1638, loss_rpn_bbox: 0.2049, loss_cls: 0.2968, acc: 87.8730, loss_bbox: 0.1588, loss_mask: 0.3348, loss: 1.1591
2020-08-13 04:52:58,010 - mmdet - INFO - Epoch [5][250/1350]    lr: 0.00250, eta: 20:19:31, time: 1.749, data_time: 0.913, memory: 6384, loss_rpn_cls: 0.1661, loss_rpn_bbox: 0.2112, loss_cls: 0.2924, acc: 87.9492, loss_bbox: 0.1597, loss_mask: 0.3352, loss: 1.1645
2020-08-13 04:54:19,344 - mmdet - INFO - Epoch [5][300/1350]    lr: 0.00250, eta: 20:17:37, time: 1.627, data_time: 0.783, memory: 6384, loss_rpn_cls: 0.1621, loss_rpn_bbox: 0.1987, loss_cls: 0.2928, acc: 88.1191, loss_bbox: 0.1605, loss_mask: 0.3290, loss: 1.1431
2020-08-13 04:55:44,434 - mmdet - INFO - Epoch [5][350/1350]    lr: 0.00250, eta: 20:16:12, time: 1.702, data_time: 0.863, memory: 6384, loss_rpn_cls: 0.1586, loss_rpn_bbox: 0.2036, loss_cls: 0.2953, acc: 88.0371, loss_bbox: 0.1596, loss_mask: 0.3337, loss: 1.1507
2020-08-13 04:57:06,111 - mmdet - INFO - Epoch [5][400/1350]    lr: 0.00250, eta: 20:14:21, time: 1.634, data_time: 0.819, memory: 6384, loss_rpn_cls: 0.1718, loss_rpn_bbox: 0.2128, loss_cls: 0.2998, acc: 87.5742, loss_bbox: 0.1634, loss_mask: 0.3335, loss: 1.1813
2020-08-13 04:58:28,885 - mmdet - INFO - Epoch [5][450/1350]    lr: 0.00250, eta: 20:12:38, time: 1.655, data_time: 0.830, memory: 6384, loss_rpn_cls: 0.1510, loss_rpn_bbox: 0.2045, loss_cls: 0.3005, acc: 87.6055, loss_bbox: 0.1646, loss_mask: 0.3437, loss: 1.1644
2020-08-13 04:59:50,438 - mmdet - INFO - Epoch [5][500/1350]    lr: 0.00250, eta: 20:10:48, time: 1.631, data_time: 0.827, memory: 6384, loss_rpn_cls: 0.1545, loss_rpn_bbox: 0.2063, loss_cls: 0.2997, acc: 87.7441, loss_bbox: 0.1646, loss_mask: 0.3439, loss: 1.1690
2020-08-13 05:01:15,126 - mmdet - INFO - Epoch [5][550/1350]    lr: 0.00250, eta: 20:09:20, time: 1.694, data_time: 0.891, memory: 6384, loss_rpn_cls: 0.1594, loss_rpn_bbox: 0.2105, loss_cls: 0.2888, acc: 88.2422, loss_bbox: 0.1584, loss_mask: 0.3279, loss: 1.1450
2020-08-13 05:02:40,962 - mmdet - INFO - Epoch [5][600/1350]    lr: 0.00250, eta: 20:08:00, time: 1.717, data_time: 0.906, memory: 6384, loss_rpn_cls: 0.1553, loss_rpn_bbox: 0.2058, loss_cls: 0.2827, acc: 88.4766, loss_bbox: 0.1605, loss_mask: 0.3379, loss: 1.1422
2020-08-13 05:04:05,415 - mmdet - INFO - Epoch [5][650/1350]    lr: 0.00250, eta: 20:06:31, time: 1.689, data_time: 0.864, memory: 6384, loss_rpn_cls: 0.1664, loss_rpn_bbox: 0.2039, loss_cls: 0.2919, acc: 88.3223, loss_bbox: 0.1611, loss_mask: 0.3340, loss: 1.1572
2020-08-13 05:05:28,004 - mmdet - INFO - Epoch [5][700/1350]    lr: 0.00250, eta: 20:04:49, time: 1.652, data_time: 0.831, memory: 6384, loss_rpn_cls: 0.1554, loss_rpn_bbox: 0.2012, loss_cls: 0.2871, acc: 88.3047, loss_bbox: 0.1593, loss_mask: 0.3326, loss: 1.1357
2020-08-13 05:06:52,837 - mmdet - INFO - Epoch [5][750/1350]    lr: 0.00250, eta: 20:03:22, time: 1.697, data_time: 0.871, memory: 6384, loss_rpn_cls: 0.1310, loss_rpn_bbox: 0.1967, loss_cls: 0.2901, acc: 88.0840, loss_bbox: 0.1587, loss_mask: 0.3357, loss: 1.1123
2020-08-13 05:08:14,117 - mmdet - INFO - Epoch [5][800/1350]    lr: 0.00250, eta: 20:01:31, time: 1.626, data_time: 0.803, memory: 6384, loss_rpn_cls: 0.1456, loss_rpn_bbox: 0.2034, loss_cls: 0.2818, acc: 88.3145, loss_bbox: 0.1546, loss_mask: 0.3537, loss: 1.1391
2020-08-13 05:09:37,169 - mmdet - INFO - Epoch [5][850/1350]    lr: 0.00250, eta: 19:59:53, time: 1.661, data_time: 0.829, memory: 6384, loss_rpn_cls: 0.1520, loss_rpn_bbox: 0.2020, loss_cls: 0.2863, acc: 88.3438, loss_bbox: 0.1582, loss_mask: 0.3341, loss: 1.1326
2020-08-13 05:10:59,095 - mmdet - INFO - Epoch [5][900/1350]    lr: 0.00250, eta: 19:58:07, time: 1.639, data_time: 0.854, memory: 6384, loss_rpn_cls: 0.1493, loss_rpn_bbox: 0.1992, loss_cls: 0.2794, acc: 88.8047, loss_bbox: 0.1565, loss_mask: 0.3339, loss: 1.1184
2020-08-13 05:12:25,243 - mmdet - INFO - Epoch [5][950/1350]    lr: 0.00250, eta: 19:56:50, time: 1.723, data_time: 0.899, memory: 6384, loss_rpn_cls: 0.1531, loss_rpn_bbox: 0.2061, loss_cls: 0.2823, acc: 88.5703, loss_bbox: 0.1527, loss_mask: 0.3263, loss: 1.1205
2020-08-13 05:13:50,473 - mmdet - INFO - Epoch [5][1000/1350]   lr: 0.00250, eta: 19:55:27, time: 1.705, data_time: 0.856, memory: 6384, loss_rpn_cls: 0.1609, loss_rpn_bbox: 0.2139, loss_cls: 0.2893, acc: 88.0645, loss_bbox: 0.1600, loss_mask: 0.3412, loss: 1.1654
2020-08-13 05:15:11,964 - mmdet - INFO - Epoch [5][1050/1350]   lr: 0.00250, eta: 19:53:39, time: 1.630, data_time: 0.783, memory: 6384, loss_rpn_cls: 0.1486, loss_rpn_bbox: 0.2037, loss_cls: 0.2927, acc: 88.0938, loss_bbox: 0.1616, loss_mask: 0.3373, loss: 1.1438
2020-08-13 05:16:39,785 - mmdet - INFO - Epoch [5][1100/1350]   lr: 0.00250, eta: 19:52:33, time: 1.756, data_time: 0.902, memory: 6384, loss_rpn_cls: 0.1699, loss_rpn_bbox: 0.2110, loss_cls: 0.3044, acc: 87.4316, loss_bbox: 0.1645, loss_mask: 0.3385, loss: 1.1883
2020-08-13 05:18:00,141 - mmdet - INFO - Epoch [5][1150/1350]   lr: 0.00250, eta: 19:50:38, time: 1.607, data_time: 0.753, memory: 6384, loss_rpn_cls: 0.1403, loss_rpn_bbox: 0.1946, loss_cls: 0.2801, acc: 88.5996, loss_bbox: 0.1536, loss_mask: 0.3330, loss: 1.1015
2020-08-13 05:19:25,925 - mmdet - INFO - Epoch [5][1200/1350]   lr: 0.00250, eta: 19:49:18, time: 1.716, data_time: 0.865, memory: 6384, loss_rpn_cls: 0.1591, loss_rpn_bbox: 0.2040, loss_cls: 0.2857, acc: 88.4375, loss_bbox: 0.1552, loss_mask: 0.3362, loss: 1.1402
2020-08-13 05:20:45,613 - mmdet - INFO - Epoch [5][1250/1350]   lr: 0.00250, eta: 19:47:20, time: 1.594, data_time: 0.756, memory: 6384, loss_rpn_cls: 0.1512, loss_rpn_bbox: 0.2009, loss_cls: 0.2886, acc: 88.2090, loss_bbox: 0.1540, loss_mask: 0.3292, loss: 1.1239
2020-08-13 05:22:12,670 - mmdet - INFO - Epoch [5][1300/1350]   lr: 0.00250, eta: 19:46:09, time: 1.741, data_time: 0.913, memory: 6384, loss_rpn_cls: 0.1542, loss_rpn_bbox: 0.2084, loss_cls: 0.2911, acc: 88.1387, loss_bbox: 0.1592, loss_mask: 0.3371, loss: 1.1501
2020-08-13 05:23:33,647 - mmdet - INFO - Epoch [5][1350/1350]   lr: 0.00250, eta: 19:44:19, time: 1.620, data_time: 0.780, memory: 6384, loss_rpn_cls: 0.1746, loss_rpn_bbox: 0.2148, loss_cls: 0.3047, acc: 87.5020, loss_bbox: 0.1678, loss_mask: 0.3286, loss: 1.1905
2020-08-13 06:27:07,556 - mmdet - INFO - Evaluating bbox...
2020-08-13 06:36:52,958 - mmdet - INFO - Epoch [5][1350/1350]   lr: 0.00250, bbox_mAP: 0.1410, bbox_mAP_50: 0.2960, bbox_mAP_75: 0.1180, bbox_mAP_s: 0.1180, bbox_mAP_m: 0.2760, bbox_mAP_l: 0.1260, bbox_mAP_copypaste: 0.141 0.296 0.118 0.118 0.276 0.126
2020-08-13 06:51:55,333 - mmdet - INFO - Epoch(train) [5][525]  loss_rpn_cls: 0.1511, loss_rpn_bbox: 0.1954, loss_cls: 0.3668, acc: 84.6548, loss_bbox: 0.1572, loss_mask: 0.3236, loss: 1.1941
2020-08-13 06:53:25,398 - mmdet - INFO - Epoch [6][50/1350] lr: 0.00250, eta: 19:43:25, time: 1.799, data_time: 0.981, memory: 6384, loss_rpn_cls: 0.1487, loss_rpn_bbox: 0.2088, loss_cls: 0.3013, acc: 87.6523, loss_bbox: 0.1647, loss_mask: 0.3248, loss: 1.1482
2020-08-13 06:54:47,135 - mmdet - INFO - Epoch [6][100/1350]    lr: 0.00250, eta: 19:41:41, time: 1.635, data_time: 0.815, memory: 6384, loss_rpn_cls: 0.1506, loss_rpn_bbox: 0.2055, loss_cls: 0.2886, acc: 88.0625, loss_bbox: 0.1566, loss_mask: 0.3340, loss: 1.1353
2020-08-13 06:56:12,601 - mmdet - INFO - Epoch [6][150/1350]    lr: 0.00250, eta: 19:40:19, time: 1.709, data_time: 0.895, memory: 6384, loss_rpn_cls: 0.1413, loss_rpn_bbox: 0.1959, loss_cls: 0.2790, acc: 88.5293, loss_bbox: 0.1535, loss_mask: 0.3278, loss: 1.0975
2020-08-13 06:57:36,555 - mmdet - INFO - Epoch [6][200/1350]    lr: 0.00250, eta: 19:38:49, time: 1.679, data_time: 0.838, memory: 6384, loss_rpn_cls: 0.1533, loss_rpn_bbox: 0.2008, loss_cls: 0.2897, acc: 88.1152, loss_bbox: 0.1561, loss_mask: 0.3326, loss: 1.1326
2020-08-13 06:58:58,259 - mmdet - INFO - Epoch [6][250/1350]    lr: 0.00250, eta: 19:37:05, time: 1.634, data_time: 0.805, memory: 6384, loss_rpn_cls: 0.1512, loss_rpn_bbox: 0.2019, loss_cls: 0.2893, acc: 88.1211, loss_bbox: 0.1612, loss_mask: 0.3323, loss: 1.1360
2020-08-13 07:00:22,533 - mmdet - INFO - Epoch [6][300/1350]    lr: 0.00250, eta: 19:35:36, time: 1.685, data_time: 0.889, memory: 6384, loss_rpn_cls: 0.1496, loss_rpn_bbox: 0.2059, loss_cls: 0.2871, acc: 88.3945, loss_bbox: 0.1536, loss_mask: 0.3234, loss: 1.1196
2020-08-13 07:01:46,391 - mmdet - INFO - Epoch [6][350/1350]    lr: 0.00250, eta: 19:34:05, time: 1.677, data_time: 0.871, memory: 6384, loss_rpn_cls: 0.1533, loss_rpn_bbox: 0.2072, loss_cls: 0.2902, acc: 88.0215, loss_bbox: 0.1608, loss_mask: 0.3337, loss: 1.1452
2020-08-13 07:03:09,206 - mmdet - INFO - Epoch [6][400/1350]    lr: 0.00250, eta: 19:32:28, time: 1.656, data_time: 0.843, memory: 6384, loss_rpn_cls: 0.1608, loss_rpn_bbox: 0.2055, loss_cls: 0.2959, acc: 87.9102, loss_bbox: 0.1590, loss_mask: 0.3317, loss: 1.1529
2020-08-13 07:04:32,660 - mmdet - INFO - Epoch [6][450/1350]    lr: 0.00250, eta: 19:30:56, time: 1.669, data_time: 0.855, memory: 6384, loss_rpn_cls: 0.1495, loss_rpn_bbox: 0.1975, loss_cls: 0.2804, acc: 88.4023, loss_bbox: 0.1532, loss_mask: 0.3394, loss: 1.1201
2020-08-13 07:05:53,768 - mmdet - INFO - Epoch [6][500/1350]    lr: 0.00250, eta: 19:29:09, time: 1.622, data_time: 0.784, memory: 6384, loss_rpn_cls: 0.1621, loss_rpn_bbox: 0.2116, loss_cls: 0.2962, acc: 87.8105, loss_bbox: 0.1607, loss_mask: 0.3313, loss: 1.1618
2020-08-13 07:07:19,261 - mmdet - INFO - Epoch [6][550/1350]    lr: 0.00250, eta: 19:27:48, time: 1.710, data_time: 0.879, memory: 6384, loss_rpn_cls: 0.1483, loss_rpn_bbox: 0.1988, loss_cls: 0.2860, acc: 88.3223, loss_bbox: 0.1583, loss_mask: 0.3332, loss: 1.1246
2020-08-13 07:08:40,168 - mmdet - INFO - Epoch [6][600/1350]    lr: 0.00250, eta: 19:26:01, time: 1.618, data_time: 0.782, memory: 6384, loss_rpn_cls: 0.1430, loss_rpn_bbox: 0.1919, loss_cls: 0.2710, acc: 88.8086, loss_bbox: 0.1538, loss_mask: 0.3340, loss: 1.0937
2020-08-13 07:10:07,565 - mmdet - INFO - Epoch [6][650/1350]    lr: 0.00250, eta: 19:24:51, time: 1.748, data_time: 0.896, memory: 6384, loss_rpn_cls: 0.1507, loss_rpn_bbox: 0.2032, loss_cls: 0.2773, acc: 88.7559, loss_bbox: 0.1585, loss_mask: 0.3310, loss: 1.1207
2020-08-13 07:11:29,683 - mmdet - INFO - Epoch [6][700/1350]    lr: 0.00250, eta: 19:23:11, time: 1.642, data_time: 0.798, memory: 6384, loss_rpn_cls: 0.1640, loss_rpn_bbox: 0.2138, loss_cls: 0.3034, acc: 87.6406, loss_bbox: 0.1663, loss_mask: 0.3340, loss: 1.1815
2020-08-13 07:12:51,956 - mmdet - INFO - Epoch [6][750/1350]    lr: 0.00250, eta: 19:21:33, time: 1.645, data_time: 0.822, memory: 6384, loss_rpn_cls: 0.1577, loss_rpn_bbox: 0.2103, loss_cls: 0.2974, acc: 87.6348, loss_bbox: 0.1651, loss_mask: 0.3360, loss: 1.1665
2020-08-13 07:14:14,121 - mmdet - INFO - Epoch [6][800/1350]    lr: 0.00250, eta: 19:19:54, time: 1.643, data_time: 0.860, memory: 6384, loss_rpn_cls: 0.1519, loss_rpn_bbox: 0.2019, loss_cls: 0.2763, acc: 88.6465, loss_bbox: 0.1507, loss_mask: 0.3356, loss: 1.1164
2020-08-13 07:15:40,124 - mmdet - INFO - Epoch [6][850/1350]    lr: 0.00250, eta: 19:18:36, time: 1.720, data_time: 0.891, memory: 6384, loss_rpn_cls: 0.1621, loss_rpn_bbox: 0.2117, loss_cls: 0.2912, acc: 88.0898, loss_bbox: 0.1565, loss_mask: 0.3298, loss: 1.1514
2020-08-13 07:17:01,337 - mmdet - INFO - Epoch [6][900/1350]    lr: 0.00250, eta: 19:16:52, time: 1.624, data_time: 0.794, memory: 6384, loss_rpn_cls: 0.1448, loss_rpn_bbox: 0.1994, loss_cls: 0.2812, acc: 88.3555, loss_bbox: 0.1567, loss_mask: 0.3335, loss: 1.1155
2020-08-13 07:18:25,283 - mmdet - INFO - Epoch [6][950/1350]    lr: 0.00250, eta: 19:15:23, time: 1.679, data_time: 0.849, memory: 6384, loss_rpn_cls: 0.1593, loss_rpn_bbox: 0.2077, loss_cls: 0.2866, acc: 88.0957, loss_bbox: 0.1578, loss_mask: 0.3285, loss: 1.1400
2020-08-13 07:19:50,134 - mmdet - INFO - Epoch [6][1000/1350]   lr: 0.00250, eta: 19:13:58, time: 1.697, data_time: 0.860, memory: 6384, loss_rpn_cls: 0.1813, loss_rpn_bbox: 0.2122, loss_cls: 0.2939, acc: 87.8926, loss_bbox: 0.1586, loss_mask: 0.3270, loss: 1.1731
2020-08-13 07:21:14,902 - mmdet - INFO - Epoch [6][1050/1350]   lr: 0.00250, eta: 19:12:34, time: 1.695, data_time: 0.869, memory: 6384, loss_rpn_cls: 0.1535, loss_rpn_bbox: 0.2004, loss_cls: 0.2844, acc: 88.1875, loss_bbox: 0.1529, loss_mask: 0.3312, loss: 1.1225
2020-08-13 07:22:35,612 - mmdet - INFO - Epoch [6][1100/1350]   lr: 0.00250, eta: 19:10:48, time: 1.614, data_time: 0.780, memory: 6384, loss_rpn_cls: 0.1489, loss_rpn_bbox: 0.2086, loss_cls: 0.2823, acc: 88.4883, loss_bbox: 0.1576, loss_mask: 0.3267, loss: 1.1241
2020-08-13 07:23:59,954 - mmdet - INFO - Epoch [6][1150/1350]   lr: 0.00250, eta: 19:09:21, time: 1.687, data_time: 0.860, memory: 6384, loss_rpn_cls: 0.1468, loss_rpn_bbox: 0.2008, loss_cls: 0.2760, acc: 88.7441, loss_bbox: 0.1515, loss_mask: 0.3403, loss: 1.1155
2020-08-13 07:25:21,153 - mmdet - INFO - Epoch [6][1200/1350]   lr: 0.00250, eta: 19:07:39, time: 1.624, data_time: 0.789, memory: 6384, loss_rpn_cls: 0.1649, loss_rpn_bbox: 0.2048, loss_cls: 0.2844, acc: 88.3652, loss_bbox: 0.1587, loss_mask: 0.3252, loss: 1.1380
2020-08-13 07:26:40,817 - mmdet - INFO - Epoch [6][1250/1350]   lr: 0.00250, eta: 19:05:48, time: 1.593, data_time: 0.786, memory: 6384, loss_rpn_cls: 0.1279, loss_rpn_bbox: 0.1926, loss_cls: 0.2810, acc: 88.1777, loss_bbox: 0.1553, loss_mask: 0.3414, loss: 1.0981
2020-08-13 07:28:02,055 - mmdet - INFO - Epoch [6][1300/1350]   lr: 0.00250, eta: 19:04:06, time: 1.625, data_time: 0.822, memory: 6384, loss_rpn_cls: 0.1398, loss_rpn_bbox: 0.2003, loss_cls: 0.2818, acc: 88.4219, loss_bbox: 0.1573, loss_mask: 0.3399, loss: 1.1192
2020-08-13 07:29:25,766 - mmdet - INFO - Epoch [6][1350/1350]   lr: 0.00250, eta: 19:02:37, time: 1.674, data_time: 0.847, memory: 6384, loss_rpn_cls: 0.1498, loss_rpn_bbox: 0.1977, loss_cls: 0.2891, acc: 88.0938, loss_bbox: 0.1623, loss_mask: 0.3407, loss: 1.1396
2020-08-13 08:30:47,711 - mmdet - INFO - Evaluating bbox...
2020-08-13 08:40:35,720 - mmdet - INFO - Epoch [6][1350/1350]   lr: 0.00250, bbox_mAP: 0.1560, bbox_mAP_50: 0.3330, bbox_mAP_75: 0.1250, bbox_mAP_s: 0.1280, bbox_mAP_m: 0.2630, bbox_mAP_l: 0.2140, bbox_mAP_copypaste: 0.156 0.333 0.125 0.128 0.263 0.214
2020-08-13 08:55:36,812 - mmdet - INFO - Epoch(train) [6][525]  loss_rpn_cls: 0.1539, loss_rpn_bbox: 0.2039, loss_cls: 0.3460, acc: 85.7507, loss_bbox: 0.1632, loss_mask: 0.3392, loss: 1.2063
2020-08-13 08:57:05,862 - mmdet - INFO - Epoch [7][50/1350] lr: 0.00250, eta: 19:01:34, time: 1.779, data_time: 0.957, memory: 6384, loss_rpn_cls: 0.1374, loss_rpn_bbox: 0.1910, loss_cls: 0.2728, acc: 88.7188, loss_bbox: 0.1549, loss_mask: 0.3312, loss: 1.0873
2020-08-13 08:58:27,391 - mmdet - INFO - Epoch [7][100/1350]    lr: 0.00250, eta: 18:59:54, time: 1.631, data_time: 0.822, memory: 6384, loss_rpn_cls: 0.1619, loss_rpn_bbox: 0.2090, loss_cls: 0.2923, acc: 88.0352, loss_bbox: 0.1549, loss_mask: 0.3206, loss: 1.1386
2020-08-13 08:59:51,437 - mmdet - INFO - Epoch [7][150/1350]    lr: 0.00250, eta: 18:58:26, time: 1.681, data_time: 0.842, memory: 6384, loss_rpn_cls: 0.1611, loss_rpn_bbox: 0.2140, loss_cls: 0.3035, acc: 87.5488, loss_bbox: 0.1663, loss_mask: 0.3454, loss: 1.1903
2020-08-13 09:01:15,172 - mmdet - INFO - Epoch [7][200/1350]    lr: 0.00250, eta: 18:56:57, time: 1.675, data_time: 0.838, memory: 6384, loss_rpn_cls: 0.1604, loss_rpn_bbox: 0.2064, loss_cls: 0.2849, acc: 88.3184, loss_bbox: 0.1548, loss_mask: 0.3425, loss: 1.1490
2020-08-13 09:02:37,627 - mmdet - INFO - Epoch [7][250/1350]    lr: 0.00250, eta: 18:55:22, time: 1.649, data_time: 0.841, memory: 6384, loss_rpn_cls: 0.1619, loss_rpn_bbox: 0.2081, loss_cls: 0.2862, acc: 88.1562, loss_bbox: 0.1601, loss_mask: 0.3311, loss: 1.1474
2020-08-13 09:04:00,501 - mmdet - INFO - Epoch [7][300/1350]    lr: 0.00250, eta: 18:53:49, time: 1.657, data_time: 0.862, memory: 6384, loss_rpn_cls: 0.1462, loss_rpn_bbox: 0.1926, loss_cls: 0.2705, acc: 88.8555, loss_bbox: 0.1498, loss_mask: 0.3320, loss: 1.0910
2020-08-13 09:05:24,834 - mmdet - INFO - Epoch [7][350/1350]    lr: 0.00250, eta: 18:52:23, time: 1.687, data_time: 0.850, memory: 6384, loss_rpn_cls: 0.1457, loss_rpn_bbox: 0.1975, loss_cls: 0.2771, acc: 88.5820, loss_bbox: 0.1496, loss_mask: 0.3293, loss: 1.0993
2020-08-13 09:06:46,711 - mmdet - INFO - Epoch [7][400/1350]    lr: 0.00250, eta: 18:50:45, time: 1.638, data_time: 0.795, memory: 6384, loss_rpn_cls: 0.1538, loss_rpn_bbox: 0.2017, loss_cls: 0.2790, acc: 88.5449, loss_bbox: 0.1489, loss_mask: 0.3267, loss: 1.1101
2020-08-13 09:08:10,415 - mmdet - INFO - Epoch [7][450/1350]    lr: 0.00250, eta: 18:49:17, time: 1.674, data_time: 0.850, memory: 6384, loss_rpn_cls: 0.1393, loss_rpn_bbox: 0.1966, loss_cls: 0.2823, acc: 88.1680, loss_bbox: 0.1526, loss_mask: 0.3334, loss: 1.1042
2020-08-13 09:09:31,502 - mmdet - INFO - Epoch [7][500/1350]    lr: 0.00250, eta: 18:47:36, time: 1.622, data_time: 0.797, memory: 6384, loss_rpn_cls: 0.1560, loss_rpn_bbox: 0.2058, loss_cls: 0.2887, acc: 88.1055, loss_bbox: 0.1608, loss_mask: 0.3387, loss: 1.1500
2020-08-13 09:10:55,325 - mmdet - INFO - Epoch [7][550/1350]    lr: 0.00250, eta: 18:46:08, time: 1.676, data_time: 0.844, memory: 6384, loss_rpn_cls: 0.1552, loss_rpn_bbox: 0.2020, loss_cls: 0.2769, acc: 88.7363, loss_bbox: 0.1547, loss_mask: 0.3496, loss: 1.1384
2020-08-13 09:12:18,480 - mmdet - INFO - Epoch [7][600/1350]    lr: 0.00250, eta: 18:44:37, time: 1.663, data_time: 0.823, memory: 6384, loss_rpn_cls: 0.1548, loss_rpn_bbox: 0.2058, loss_cls: 0.2985, acc: 87.7070, loss_bbox: 0.1660, loss_mask: 0.3407, loss: 1.1658
2020-08-13 09:13:46,323 - mmdet - INFO - Epoch [7][650/1350]    lr: 0.00250, eta: 18:43:27, time: 1.757, data_time: 0.908, memory: 6384, loss_rpn_cls: 0.1559, loss_rpn_bbox: 0.2006, loss_cls: 0.2730, acc: 88.7188, loss_bbox: 0.1547, loss_mask: 0.3355, loss: 1.1197
2020-08-13 09:15:07,934 - mmdet - INFO - Epoch [7][700/1350]    lr: 0.00250, eta: 18:41:49, time: 1.632, data_time: 0.786, memory: 6384, loss_rpn_cls: 0.1491, loss_rpn_bbox: 0.1978, loss_cls: 0.2769, acc: 88.5781, loss_bbox: 0.1542, loss_mask: 0.3313, loss: 1.1093
2020-08-13 09:16:29,918 - mmdet - INFO - Epoch [7][750/1350]    lr: 0.00250, eta: 18:40:13, time: 1.640, data_time: 0.838, memory: 6384, loss_rpn_cls: 0.1498, loss_rpn_bbox: 0.1984, loss_cls: 0.2810, acc: 88.3066, loss_bbox: 0.1570, loss_mask: 0.3419, loss: 1.1281
2020-08-13 09:17:53,003 - mmdet - INFO - Epoch [7][800/1350]    lr: 0.00250, eta: 18:38:42, time: 1.662, data_time: 0.845, memory: 6384, loss_rpn_cls: 0.1527, loss_rpn_bbox: 0.1965, loss_cls: 0.2761, acc: 88.7832, loss_bbox: 0.1554, loss_mask: 0.3309, loss: 1.1115
2020-08-13 09:19:19,549 - mmdet - INFO - Epoch [7][850/1350]    lr: 0.00250, eta: 18:37:26, time: 1.731, data_time: 0.887, memory: 6384, loss_rpn_cls: 0.1331, loss_rpn_bbox: 0.1918, loss_cls: 0.2637, acc: 89.1230, loss_bbox: 0.1472, loss_mask: 0.3350, loss: 1.0708
2020-08-13 09:20:40,980 - mmdet - INFO - Epoch [7][900/1350]    lr: 0.00250, eta: 18:35:48, time: 1.629, data_time: 0.779, memory: 6384, loss_rpn_cls: 0.1553, loss_rpn_bbox: 0.2029, loss_cls: 0.2934, acc: 87.7090, loss_bbox: 0.1599, loss_mask: 0.3270, loss: 1.1386
2020-08-13 09:22:04,298 - mmdet - INFO - Epoch [7][950/1350]    lr: 0.00250, eta: 18:34:18, time: 1.666, data_time: 0.815, memory: 6384, loss_rpn_cls: 0.1403, loss_rpn_bbox: 0.1985, loss_cls: 0.2786, acc: 88.4785, loss_bbox: 0.1544, loss_mask: 0.3297, loss: 1.1015
2020-08-13 09:23:25,124 - mmdet - INFO - Epoch [7][1000/1350]   lr: 0.00250, eta: 18:32:38, time: 1.616, data_time: 0.782, memory: 6384, loss_rpn_cls: 0.1469, loss_rpn_bbox: 0.1993, loss_cls: 0.2786, acc: 88.4277, loss_bbox: 0.1559, loss_mask: 0.3261, loss: 1.1067
2020-08-13 09:24:48,542 - mmdet - INFO - Epoch [7][1050/1350]   lr: 0.00250, eta: 18:31:08, time: 1.668, data_time: 0.820, memory: 6384, loss_rpn_cls: 0.1435, loss_rpn_bbox: 0.1976, loss_cls: 0.2836, acc: 88.2930, loss_bbox: 0.1540, loss_mask: 0.3372, loss: 1.1159
2020-08-13 09:26:11,575 - mmdet - INFO - Epoch [7][1100/1350]   lr: 0.00250, eta: 18:29:38, time: 1.661, data_time: 0.812, memory: 6384, loss_rpn_cls: 0.1331, loss_rpn_bbox: 0.1965, loss_cls: 0.2739, acc: 88.7969, loss_bbox: 0.1498, loss_mask: 0.3359, loss: 1.0892
2020-08-13 09:27:36,213 - mmdet - INFO - Epoch [7][1150/1350]   lr: 0.00250, eta: 18:28:14, time: 1.693, data_time: 0.841, memory: 6384, loss_rpn_cls: 0.1438, loss_rpn_bbox: 0.1982, loss_cls: 0.2710, acc: 88.6328, loss_bbox: 0.1508, loss_mask: 0.3373, loss: 1.1011
2020-08-13 09:28:59,877 - mmdet - INFO - Epoch [7][1200/1350]   lr: 0.00250, eta: 18:26:46, time: 1.673, data_time: 0.815, memory: 6384, loss_rpn_cls: 0.1614, loss_rpn_bbox: 0.1982, loss_cls: 0.2880, acc: 88.2109, loss_bbox: 0.1569, loss_mask: 0.3212, loss: 1.1257
2020-08-13 09:30:24,603 - mmdet - INFO - Epoch [7][1250/1350]   lr: 0.00250, eta: 18:25:22, time: 1.695, data_time: 0.833, memory: 6384, loss_rpn_cls: 0.1487, loss_rpn_bbox: 0.2016, loss_cls: 0.2846, acc: 88.4922, loss_bbox: 0.1533, loss_mask: 0.3259, loss: 1.1141
2020-08-13 09:31:47,540 - mmdet - INFO - Epoch [7][1300/1350]   lr: 0.00250, eta: 18:23:51, time: 1.659, data_time: 0.796, memory: 6384, loss_rpn_cls: 0.1479, loss_rpn_bbox: 0.1977, loss_cls: 0.2818, acc: 88.3086, loss_bbox: 0.1555, loss_mask: 0.3216, loss: 1.1045
2020-08-13 09:33:12,249 - mmdet - INFO - Epoch [7][1350/1350]   lr: 0.00250, eta: 18:22:28, time: 1.694, data_time: 0.834, memory: 6384, loss_rpn_cls: 0.1532, loss_rpn_bbox: 0.2071, loss_cls: 0.2990, acc: 87.6270, loss_bbox: 0.1639, loss_mask: 0.3352, loss: 1.1584
2020-08-13 10:23:12,435 - mmdet - INFO - Evaluating bbox...
2020-08-13 10:32:59,591 - mmdet - INFO - Epoch [7][1350/1350]   lr: 0.00250, bbox_mAP: 0.1640, bbox_mAP_50: 0.3450, bbox_mAP_75: 0.1350, bbox_mAP_s: 0.1350, bbox_mAP_m: 0.2770, bbox_mAP_l: 0.1780, bbox_mAP_copypaste: 0.164 0.345 0.135 0.135 0.277 0.178
2020-08-13 10:48:05,632 - mmdet - INFO - Epoch(train) [7][525]  loss_rpn_cls: 0.1465, loss_rpn_bbox: 0.1937, loss_cls: 0.3457, acc: 85.9356, loss_bbox: 0.1607, loss_mask: 0.3395, loss: 1.1861
2020-08-13 10:49:36,272 - mmdet - INFO - Epoch [8][50/1350] lr: 0.00250, eta: 18:21:28, time: 1.810, data_time: 0.972, memory: 6384, loss_rpn_cls: 0.1396, loss_rpn_bbox: 0.2019, loss_cls: 0.2761, acc: 88.5352, loss_bbox: 0.1548, loss_mask: 0.3377, loss: 1.1101
2020-08-13 10:50:55,900 - mmdet - INFO - Epoch [8][100/1350]    lr: 0.00250, eta: 18:19:44, time: 1.593, data_time: 0.794, memory: 6384, loss_rpn_cls: 0.1569, loss_rpn_bbox: 0.2044, loss_cls: 0.2905, acc: 87.9277, loss_bbox: 0.1565, loss_mask: 0.3314, loss: 1.1396
2020-08-13 10:52:18,260 - mmdet - INFO - Epoch [8][150/1350]    lr: 0.00250, eta: 18:18:11, time: 1.647, data_time: 0.883, memory: 6384, loss_rpn_cls: 0.1418, loss_rpn_bbox: 0.1975, loss_cls: 0.2750, acc: 88.6230, loss_bbox: 0.1514, loss_mask: 0.3359, loss: 1.1015
2020-08-13 10:53:42,494 - mmdet - INFO - Epoch [8][200/1350]    lr: 0.00250, eta: 18:16:45, time: 1.685, data_time: 0.882, memory: 6384, loss_rpn_cls: 0.1561, loss_rpn_bbox: 0.2118, loss_cls: 0.2976, acc: 87.7207, loss_bbox: 0.1656, loss_mask: 0.3246, loss: 1.1556
2020-08-13 10:55:05,758 - mmdet - INFO - Epoch [8][250/1350]    lr: 0.00250, eta: 18:15:16, time: 1.665, data_time: 0.870, memory: 6384, loss_rpn_cls: 0.1452, loss_rpn_bbox: 0.1982, loss_cls: 0.2837, acc: 88.2344, loss_bbox: 0.1558, loss_mask: 0.3201, loss: 1.1030
2020-08-13 10:56:27,218 - mmdet - INFO - Epoch [8][300/1350]    lr: 0.00250, eta: 18:13:39, time: 1.629, data_time: 0.841, memory: 6384, loss_rpn_cls: 0.1573, loss_rpn_bbox: 0.2078, loss_cls: 0.2895, acc: 87.9258, loss_bbox: 0.1533, loss_mask: 0.3394, loss: 1.1474
2020-08-13 10:57:51,438 - mmdet - INFO - Epoch [8][350/1350]    lr: 0.00250, eta: 18:12:14, time: 1.684, data_time: 0.890, memory: 6384, loss_rpn_cls: 0.1342, loss_rpn_bbox: 0.1957, loss_cls: 0.2798, acc: 88.3418, loss_bbox: 0.1517, loss_mask: 0.3412, loss: 1.1026
2020-08-13 10:59:12,723 - mmdet - INFO - Epoch [8][400/1350]    lr: 0.00250, eta: 18:10:37, time: 1.626, data_time: 0.825, memory: 6384, loss_rpn_cls: 0.1540, loss_rpn_bbox: 0.2106, loss_cls: 0.2877, acc: 87.9668, loss_bbox: 0.1585, loss_mask: 0.3315, loss: 1.1422
2020-08-13 11:00:39,491 - mmdet - INFO - Epoch [8][450/1350]    lr: 0.00250, eta: 18:09:22, time: 1.735, data_time: 0.904, memory: 6384, loss_rpn_cls: 0.1443, loss_rpn_bbox: 0.1982, loss_cls: 0.2797, acc: 88.4160, loss_bbox: 0.1594, loss_mask: 0.3385, loss: 1.1202
2020-08-13 11:02:01,676 - mmdet - INFO - Epoch [8][500/1350]    lr: 0.00250, eta: 18:07:48, time: 1.644, data_time: 0.832, memory: 6384, loss_rpn_cls: 0.1456, loss_rpn_bbox: 0.1991, loss_cls: 0.2821, acc: 88.5371, loss_bbox: 0.1542, loss_mask: 0.3334, loss: 1.1143
2020-08-13 11:03:29,760 - mmdet - INFO - Epoch [8][550/1350]    lr: 0.00250, eta: 18:06:38, time: 1.762, data_time: 0.941, memory: 6384, loss_rpn_cls: 0.1907, loss_rpn_bbox: 0.2126, loss_cls: 0.3012, acc: 87.4414, loss_bbox: 0.1658, loss_mask: 0.3204, loss: 1.1905
2020-08-13 11:04:50,920 - mmdet - INFO - Epoch [8][600/1350]    lr: 0.00250, eta: 18:05:01, time: 1.623, data_time: 0.781, memory: 6384, loss_rpn_cls: 0.1440, loss_rpn_bbox: 0.2030, loss_cls: 0.2809, acc: 88.2324, loss_bbox: 0.1545, loss_mask: 0.3293, loss: 1.1117
2020-08-13 11:06:14,411 - mmdet - INFO - Epoch [8][650/1350]    lr: 0.00250, eta: 18:03:33, time: 1.670, data_time: 0.840, memory: 6384, loss_rpn_cls: 0.1349, loss_rpn_bbox: 0.1973, loss_cls: 0.2755, acc: 88.6309, loss_bbox: 0.1529, loss_mask: 0.3295, loss: 1.0902
2020-08-13 11:07:37,593 - mmdet - INFO - Epoch [8][700/1350]    lr: 0.00250, eta: 18:02:04, time: 1.664, data_time: 0.874, memory: 6384, loss_rpn_cls: 0.1563, loss_rpn_bbox: 0.2046, loss_cls: 0.2792, acc: 88.5879, loss_bbox: 0.1545, loss_mask: 0.3165, loss: 1.1111
2020-08-13 11:09:01,912 - mmdet - INFO - Epoch [8][750/1350]    lr: 0.00250, eta: 18:00:39, time: 1.686, data_time: 0.883, memory: 6384, loss_rpn_cls: 0.1520, loss_rpn_bbox: 0.1977, loss_cls: 0.2697, acc: 88.9102, loss_bbox: 0.1498, loss_mask: 0.3302, loss: 1.0994
2020-08-13 11:10:23,389 - mmdet - INFO - Epoch [8][800/1350]    lr: 0.00250, eta: 17:59:03, time: 1.630, data_time: 0.828, memory: 6384, loss_rpn_cls: 0.1411, loss_rpn_bbox: 0.1982, loss_cls: 0.2787, acc: 88.3789, loss_bbox: 0.1561, loss_mask: 0.3359, loss: 1.1100
2020-08-13 11:11:46,609 - mmdet - INFO - Epoch [8][850/1350]    lr: 0.00250, eta: 17:57:35, time: 1.664, data_time: 0.854, memory: 6384, loss_rpn_cls: 0.1430, loss_rpn_bbox: 0.1928, loss_cls: 0.2716, acc: 88.7715, loss_bbox: 0.1507, loss_mask: 0.3371, loss: 1.0951
2020-08-13 11:13:10,628 - mmdet - INFO - Epoch [8][900/1350]    lr: 0.00250, eta: 17:56:09, time: 1.680, data_time: 0.837, memory: 6384, loss_rpn_cls: 0.1379, loss_rpn_bbox: 0.1954, loss_cls: 0.2685, acc: 88.6914, loss_bbox: 0.1507, loss_mask: 0.3383, loss: 1.0909
2020-08-13 11:14:33,588 - mmdet - INFO - Epoch [8][950/1350]    lr: 0.00250, eta: 17:54:39, time: 1.659, data_time: 0.813, memory: 6384, loss_rpn_cls: 0.1422, loss_rpn_bbox: 0.1917, loss_cls: 0.2762, acc: 88.4297, loss_bbox: 0.1509, loss_mask: 0.3262, loss: 1.0872
2020-08-13 11:15:55,782 - mmdet - INFO - Epoch [8][1000/1350]   lr: 0.00250, eta: 17:53:07, time: 1.644, data_time: 0.815, memory: 6384, loss_rpn_cls: 0.1420, loss_rpn_bbox: 0.1987, loss_cls: 0.2738, acc: 88.8691, loss_bbox: 0.1497, loss_mask: 0.3303, loss: 1.0944
2020-08-13 11:17:20,612 - mmdet - INFO - Epoch [8][1050/1350]   lr: 0.00250, eta: 17:51:44, time: 1.697, data_time: 0.884, memory: 6384, loss_rpn_cls: 0.1432, loss_rpn_bbox: 0.1941, loss_cls: 0.2783, acc: 88.4160, loss_bbox: 0.1536, loss_mask: 0.3402, loss: 1.1094
2020-08-13 11:18:42,928 - mmdet - INFO - Epoch [8][1100/1350]   lr: 0.00250, eta: 17:50:12, time: 1.646, data_time: 0.790, memory: 6384, loss_rpn_cls: 0.1392, loss_rpn_bbox: 0.1924, loss_cls: 0.2715, acc: 88.6973, loss_bbox: 0.1512, loss_mask: 0.3315, loss: 1.0858
2020-08-13 11:20:08,731 - mmdet - INFO - Epoch [8][1150/1350]   lr: 0.00250, eta: 17:48:53, time: 1.716, data_time: 0.861, memory: 6384, loss_rpn_cls: 0.1447, loss_rpn_bbox: 0.1989, loss_cls: 0.2789, acc: 88.6406, loss_bbox: 0.1549, loss_mask: 0.3345, loss: 1.1118
2020-08-13 11:21:29,408 - mmdet - INFO - Epoch [8][1200/1350]   lr: 0.00250, eta: 17:47:15, time: 1.614, data_time: 0.769, memory: 6384, loss_rpn_cls: 0.1236, loss_rpn_bbox: 0.1924, loss_cls: 0.2701, acc: 88.8164, loss_bbox: 0.1493, loss_mask: 0.3341, loss: 1.0694
2020-08-13 11:22:52,261 - mmdet - INFO - Epoch [8][1250/1350]   lr: 0.00250, eta: 17:45:45, time: 1.657, data_time: 0.825, memory: 6384, loss_rpn_cls: 0.1488, loss_rpn_bbox: 0.2007, loss_cls: 0.2777, acc: 88.5293, loss_bbox: 0.1546, loss_mask: 0.3351, loss: 1.1169
2020-08-13 11:24:15,152 - mmdet - INFO - Epoch [8][1300/1350]   lr: 0.00250, eta: 17:44:16, time: 1.658, data_time: 0.821, memory: 6384, loss_rpn_cls: 0.1656, loss_rpn_bbox: 0.2093, loss_cls: 0.2943, acc: 87.6875, loss_bbox: 0.1595, loss_mask: 0.3427, loss: 1.1714
2020-08-13 11:25:37,401 - mmdet - INFO - Epoch [8][1350/1350]   lr: 0.00250, eta: 17:42:44, time: 1.645, data_time: 0.813, memory: 6384, loss_rpn_cls: 0.1401, loss_rpn_bbox: 0.1951, loss_cls: 0.2826, acc: 88.3359, loss_bbox: 0.1620, loss_mask: 0.3349, loss: 1.1147
2020-08-13 12:19:41,629 - mmdet - INFO - Evaluating bbox...
2020-08-13 12:29:17,658 - mmdet - INFO - Epoch [8][1350/1350]   lr: 0.00250, bbox_mAP: 0.1650, bbox_mAP_50: 0.3380, bbox_mAP_75: 0.1420, bbox_mAP_s: 0.1260, bbox_mAP_m: 0.3040, bbox_mAP_l: 0.2240, bbox_mAP_copypaste: 0.165 0.338 0.142 0.126 0.304 0.224
2020-08-13 12:44:37,786 - mmdet - INFO - Epoch(train) [8][525]  loss_rpn_cls: 0.1363, loss_rpn_bbox: 0.1925, loss_cls: 0.3202, acc: 86.6101, loss_bbox: 0.1550, loss_mask: 0.3312, loss: 1.1353
2020-08-13 12:46:08,872 - mmdet - INFO - Epoch [9][50/1350] lr: 0.00250, eta: 17:41:43, time: 1.819, data_time: 1.007, memory: 6384, loss_rpn_cls: 0.1535, loss_rpn_bbox: 0.2045, loss_cls: 0.2832, acc: 88.4551, loss_bbox: 0.1580, loss_mask: 0.3368, loss: 1.1361
2020-08-13 12:47:31,995 - mmdet - INFO - Epoch [9][100/1350]    lr: 0.00250, eta: 17:40:14, time: 1.662, data_time: 0.873, memory: 6384, loss_rpn_cls: 0.1416, loss_rpn_bbox: 0.1969, loss_cls: 0.2840, acc: 88.0684, loss_bbox: 0.1577, loss_mask: 0.3260, loss: 1.1063
2020-08-13 12:48:55,460 - mmdet - INFO - Epoch [9][150/1350]    lr: 0.00250, eta: 17:38:47, time: 1.669, data_time: 0.887, memory: 6384, loss_rpn_cls: 0.1505, loss_rpn_bbox: 0.2047, loss_cls: 0.2922, acc: 87.9004, loss_bbox: 0.1585, loss_mask: 0.3364, loss: 1.1422
2020-08-13 12:50:19,488 - mmdet - INFO - Epoch [9][200/1350]    lr: 0.00250, eta: 17:37:21, time: 1.681, data_time: 0.912, memory: 6384, loss_rpn_cls: 0.1334, loss_rpn_bbox: 0.1941, loss_cls: 0.2629, acc: 89.1074, loss_bbox: 0.1444, loss_mask: 0.3305, loss: 1.0652
2020-08-13 12:51:44,244 - mmdet - INFO - Epoch [9][250/1350]    lr: 0.00250, eta: 17:35:58, time: 1.695, data_time: 0.890, memory: 6384, loss_rpn_cls: 0.1309, loss_rpn_bbox: 0.1934, loss_cls: 0.2683, acc: 88.8027, loss_bbox: 0.1494, loss_mask: 0.3325, loss: 1.0744
2020-08-13 12:53:08,086 - mmdet - INFO - Epoch [9][300/1350]    lr: 0.00250, eta: 17:34:32, time: 1.677, data_time: 0.894, memory: 6384, loss_rpn_cls: 0.1564, loss_rpn_bbox: 0.2037, loss_cls: 0.2903, acc: 87.8438, loss_bbox: 0.1588, loss_mask: 0.3346, loss: 1.1439
2020-08-13 12:54:33,628 - mmdet - INFO - Epoch [9][350/1350]    lr: 0.00250, eta: 17:33:11, time: 1.711, data_time: 0.919, memory: 6384, loss_rpn_cls: 0.1296, loss_rpn_bbox: 0.1933, loss_cls: 0.2736, acc: 88.6230, loss_bbox: 0.1513, loss_mask: 0.3337, loss: 1.0814
2020-08-13 12:55:59,618 - mmdet - INFO - Epoch [9][400/1350]    lr: 0.00250, eta: 17:31:53, time: 1.720, data_time: 0.926, memory: 6384, loss_rpn_cls: 0.1242, loss_rpn_bbox: 0.1864, loss_cls: 0.2677, acc: 88.9805, loss_bbox: 0.1482, loss_mask: 0.3342, loss: 1.0607
2020-08-13 12:57:21,706 - mmdet - INFO - Epoch [9][450/1350]    lr: 0.00250, eta: 17:30:21, time: 1.642, data_time: 0.825, memory: 6384, loss_rpn_cls: 0.1266, loss_rpn_bbox: 0.1891, loss_cls: 0.2677, acc: 89.1191, loss_bbox: 0.1493, loss_mask: 0.3353, loss: 1.0680
2020-08-13 12:58:47,261 - mmdet - INFO - Epoch [9][500/1350]    lr: 0.00250, eta: 17:29:00, time: 1.711, data_time: 0.895, memory: 6384, loss_rpn_cls: 0.1438, loss_rpn_bbox: 0.2029, loss_cls: 0.2845, acc: 88.0098, loss_bbox: 0.1612, loss_mask: 0.3353, loss: 1.1277
2020-08-13 13:00:12,292 - mmdet - INFO - Epoch [9][550/1350]    lr: 0.00250, eta: 17:27:38, time: 1.701, data_time: 0.881, memory: 6384, loss_rpn_cls: 0.1477, loss_rpn_bbox: 0.2043, loss_cls: 0.2776, acc: 88.4316, loss_bbox: 0.1555, loss_mask: 0.3365, loss: 1.1216
2020-08-13 13:01:41,310 - mmdet - INFO - Epoch [9][600/1350]    lr: 0.00250, eta: 17:26:29, time: 1.780, data_time: 0.943, memory: 6384, loss_rpn_cls: 0.1358, loss_rpn_bbox: 0.1933, loss_cls: 0.2760, acc: 88.4688, loss_bbox: 0.1532, loss_mask: 0.3339, loss: 1.0922
2020-08-13 13:03:02,542 - mmdet - INFO - Epoch [9][650/1350]    lr: 0.00250, eta: 17:24:54, time: 1.625, data_time: 0.800, memory: 6384, loss_rpn_cls: 0.1439, loss_rpn_bbox: 0.2001, loss_cls: 0.2983, acc: 87.4922, loss_bbox: 0.1657, loss_mask: 0.3346, loss: 1.1426
2020-08-13 13:04:28,162 - mmdet - INFO - Epoch [9][700/1350]    lr: 0.00250, eta: 17:23:34, time: 1.712, data_time: 0.894, memory: 6384, loss_rpn_cls: 0.1367, loss_rpn_bbox: 0.2029, loss_cls: 0.2811, acc: 88.4160, loss_bbox: 0.1547, loss_mask: 0.3391, loss: 1.1145
2020-08-13 13:05:49,596 - mmdet - INFO - Epoch [9][750/1350]    lr: 0.00250, eta: 17:22:00, time: 1.629, data_time: 0.815, memory: 6384, loss_rpn_cls: 0.1387, loss_rpn_bbox: 0.1971, loss_cls: 0.2773, acc: 88.4629, loss_bbox: 0.1513, loss_mask: 0.3155, loss: 1.0799
2020-08-13 13:07:16,391 - mmdet - INFO - Epoch [9][800/1350]    lr: 0.00250, eta: 17:20:43, time: 1.736, data_time: 0.939, memory: 6384, loss_rpn_cls: 0.1287, loss_rpn_bbox: 0.1977, loss_cls: 0.2746, acc: 88.3809, loss_bbox: 0.1484, loss_mask: 0.3409, loss: 1.0903
2020-08-13 13:08:38,689 - mmdet - INFO - Epoch [9][850/1350]    lr: 0.00250, eta: 17:19:12, time: 1.646, data_time: 0.838, memory: 6384, loss_rpn_cls: 0.1331, loss_rpn_bbox: 0.1973, loss_cls: 0.2717, acc: 88.8125, loss_bbox: 0.1539, loss_mask: 0.3294, loss: 1.0854
2020-08-13 13:10:04,363 - mmdet - INFO - Epoch [9][900/1350]    lr: 0.00250, eta: 17:17:52, time: 1.713, data_time: 0.902, memory: 6384, loss_rpn_cls: 0.1510, loss_rpn_bbox: 0.2079, loss_cls: 0.2946, acc: 87.5742, loss_bbox: 0.1654, loss_mask: 0.3320, loss: 1.1509
2020-08-13 13:11:25,953 - mmdet - INFO - Epoch [9][950/1350]    lr: 0.00250, eta: 17:16:19, time: 1.632, data_time: 0.820, memory: 6384, loss_rpn_cls: 0.1395, loss_rpn_bbox: 0.1980, loss_cls: 0.2872, acc: 87.9453, loss_bbox: 0.1590, loss_mask: 0.3313, loss: 1.1150
2020-08-13 13:12:52,489 - mmdet - INFO - Epoch [9][1000/1350]   lr: 0.00250, eta: 17:15:01, time: 1.731, data_time: 0.920, memory: 6384, loss_rpn_cls: 0.1311, loss_rpn_bbox: 0.1905, loss_cls: 0.2674, acc: 88.8750, loss_bbox: 0.1480, loss_mask: 0.3231, loss: 1.0601
2020-08-13 13:14:16,407 - mmdet - INFO - Epoch [9][1050/1350]   lr: 0.00250, eta: 17:13:35, time: 1.678, data_time: 0.852, memory: 6384, loss_rpn_cls: 0.1425, loss_rpn_bbox: 0.1940, loss_cls: 0.2656, acc: 88.9902, loss_bbox: 0.1529, loss_mask: 0.3370, loss: 1.0919
2020-08-13 13:15:44,377 - mmdet - INFO - Epoch [9][1100/1350]   lr: 0.00250, eta: 17:12:22, time: 1.759, data_time: 0.913, memory: 6384, loss_rpn_cls: 0.1586, loss_rpn_bbox: 0.2105, loss_cls: 0.2902, acc: 87.8848, loss_bbox: 0.1600, loss_mask: 0.3364, loss: 1.1557
2020-08-13 13:17:08,658 - mmdet - INFO - Epoch [9][1150/1350]   lr: 0.00250, eta: 17:10:57, time: 1.686, data_time: 0.850, memory: 6384, loss_rpn_cls: 0.1426, loss_rpn_bbox: 0.1989, loss_cls: 0.2903, acc: 87.7500, loss_bbox: 0.1601, loss_mask: 0.3316, loss: 1.1234
2020-08-13 13:18:36,239 - mmdet - INFO - Epoch [9][1200/1350]   lr: 0.00250, eta: 17:09:43, time: 1.752, data_time: 0.893, memory: 6384, loss_rpn_cls: 0.1505, loss_rpn_bbox: 0.2001, loss_cls: 0.2871, acc: 88.0488, loss_bbox: 0.1577, loss_mask: 0.3335, loss: 1.1289
2020-08-13 13:20:02,314 - mmdet - INFO - Epoch [9][1250/1350]   lr: 0.00250, eta: 17:08:23, time: 1.722, data_time: 0.866, memory: 6384, loss_rpn_cls: 0.1430, loss_rpn_bbox: 0.2045, loss_cls: 0.2847, acc: 88.0527, loss_bbox: 0.1554, loss_mask: 0.3325, loss: 1.1202
2020-08-13 13:21:26,844 - mmdet - INFO - Epoch [9][1300/1350]   lr: 0.00250, eta: 17:06:59, time: 1.691, data_time: 0.872, memory: 6384, loss_rpn_cls: 0.1334, loss_rpn_bbox: 0.1952, loss_cls: 0.2724, acc: 88.6562, loss_bbox: 0.1476, loss_mask: 0.3314, loss: 1.0800
2020-08-13 13:22:50,426 - mmdet - INFO - Epoch [9][1350/1350]   lr: 0.00250, eta: 17:05:32, time: 1.672, data_time: 0.860, memory: 6384, loss_rpn_cls: 0.1468, loss_rpn_bbox: 0.2011, loss_cls: 0.2863, acc: 88.0957, loss_bbox: 0.1593, loss_mask: 0.3316, loss: 1.1250
2020-08-13 14:16:46,990 - mmdet - INFO - Evaluating bbox...
2020-08-13 14:26:30,211 - mmdet - INFO - Epoch [9][1350/1350]   lr: 0.00250, bbox_mAP: 0.1650, bbox_mAP_50: 0.3360, bbox_mAP_75: 0.1360, bbox_mAP_s: 0.1310, bbox_mAP_m: 0.2910, bbox_mAP_l: 0.2450, bbox_mAP_copypaste: 0.165 0.336 0.136 0.131 0.291 0.245
2020-08-13 14:41:47,550 - mmdet - INFO - Epoch(train) [9][525]  loss_rpn_cls: 0.1332, loss_rpn_bbox: 0.1891, loss_cls: 0.3330, acc: 86.2999, loss_bbox: 0.1552, loss_mask: 0.3386, loss: 1.1490
2020-08-13 14:43:20,537 - mmdet - INFO - Epoch [10][50/1350]    lr: 0.00250, eta: 17:04:33, time: 1.857, data_time: 1.028, memory: 6384, loss_rpn_cls: 0.1321, loss_rpn_bbox: 0.1884, loss_cls: 0.2605, acc: 89.0898, loss_bbox: 0.1490, loss_mask: 0.3383, loss: 1.0683
2020-08-13 14:44:43,830 - mmdet - INFO - Epoch [10][100/1350]   lr: 0.00250, eta: 17:03:05, time: 1.666, data_time: 0.878, memory: 6384, loss_rpn_cls: 0.1348, loss_rpn_bbox: 0.1989, loss_cls: 0.2681, acc: 88.8770, loss_bbox: 0.1459, loss_mask: 0.3198, loss: 1.0675
2020-08-13 14:46:10,635 - mmdet - INFO - Epoch [10][150/1350]   lr: 0.00250, eta: 17:01:48, time: 1.736, data_time: 0.926, memory: 6384, loss_rpn_cls: 0.1338, loss_rpn_bbox: 0.1998, loss_cls: 0.2823, acc: 88.3184, loss_bbox: 0.1547, loss_mask: 0.3420, loss: 1.1126
2020-08-13 14:47:34,989 - mmdet - INFO - Epoch [10][200/1350]   lr: 0.00250, eta: 17:00:23, time: 1.687, data_time: 0.862, memory: 6384, loss_rpn_cls: 0.1280, loss_rpn_bbox: 0.1940, loss_cls: 0.2759, acc: 88.4961, loss_bbox: 0.1537, loss_mask: 0.3294, loss: 1.0811
2020-08-13 14:49:03,262 - mmdet - INFO - Epoch [10][250/1350]   lr: 0.00250, eta: 16:59:10, time: 1.765, data_time: 0.926, memory: 6384, loss_rpn_cls: 0.1352, loss_rpn_bbox: 0.1975, loss_cls: 0.2739, acc: 88.7207, loss_bbox: 0.1535, loss_mask: 0.3262, loss: 1.0864
2020-08-13 14:50:27,742 - mmdet - INFO - Epoch [10][300/1350]   lr: 0.00250, eta: 16:57:45, time: 1.690, data_time: 0.849, memory: 6384, loss_rpn_cls: 0.1481, loss_rpn_bbox: 0.2097, loss_cls: 0.2904, acc: 87.9727, loss_bbox: 0.1575, loss_mask: 0.3293, loss: 1.1350
2020-08-13 14:51:54,720 - mmdet - INFO - Epoch [10][350/1350]   lr: 0.00250, eta: 16:56:28, time: 1.740, data_time: 0.900, memory: 6384, loss_rpn_cls: 0.1361, loss_rpn_bbox: 0.1966, loss_cls: 0.2754, acc: 88.4980, loss_bbox: 0.1517, loss_mask: 0.3343, loss: 1.0941
2020-08-13 14:53:15,047 - mmdet - INFO - Epoch [10][400/1350]   lr: 0.00250, eta: 16:54:52, time: 1.607, data_time: 0.770, memory: 6384, loss_rpn_cls: 0.1314, loss_rpn_bbox: 0.1993, loss_cls: 0.2729, acc: 88.7578, loss_bbox: 0.1540, loss_mask: 0.3419, loss: 1.0995
2020-08-13 14:54:40,073 - mmdet - INFO - Epoch [10][450/1350]   lr: 0.00250, eta: 16:53:29, time: 1.700, data_time: 0.888, memory: 6384, loss_rpn_cls: 0.1271, loss_rpn_bbox: 0.1917, loss_cls: 0.2665, acc: 88.9922, loss_bbox: 0.1522, loss_mask: 0.3240, loss: 1.0616
2020-08-13 14:56:04,307 - mmdet - INFO - Epoch [10][500/1350]   lr: 0.00250, eta: 16:52:04, time: 1.685, data_time: 0.880, memory: 6384, loss_rpn_cls: 0.1389, loss_rpn_bbox: 0.1972, loss_cls: 0.2764, acc: 88.6621, loss_bbox: 0.1568, loss_mask: 0.3242, loss: 1.0936
2020-08-13 14:57:28,601 - mmdet - INFO - Epoch [10][550/1350]   lr: 0.00250, eta: 16:50:39, time: 1.686, data_time: 0.873, memory: 6384, loss_rpn_cls: 0.1310, loss_rpn_bbox: 0.1936, loss_cls: 0.2748, acc: 88.4727, loss_bbox: 0.1540, loss_mask: 0.3360, loss: 1.0894
2020-08-13 14:58:48,585 - mmdet - INFO - Epoch [10][600/1350]   lr: 0.00250, eta: 16:49:02, time: 1.600, data_time: 0.798, memory: 6384, loss_rpn_cls: 0.1216, loss_rpn_bbox: 0.1952, loss_cls: 0.2696, acc: 88.7012, loss_bbox: 0.1525, loss_mask: 0.3471, loss: 1.0859
2020-08-13 15:00:14,488 - mmdet - INFO - Epoch [10][650/1350]   lr: 0.00250, eta: 16:47:42, time: 1.718, data_time: 0.903, memory: 6384, loss_rpn_cls: 0.1485, loss_rpn_bbox: 0.2016, loss_cls: 0.2902, acc: 88.0371, loss_bbox: 0.1593, loss_mask: 0.3294, loss: 1.1289
2020-08-13 15:01:42,331 - mmdet - INFO - Epoch [10][700/1350]   lr: 0.00250, eta: 16:46:27, time: 1.757, data_time: 0.930, memory: 6384, loss_rpn_cls: 0.1412, loss_rpn_bbox: 0.1942, loss_cls: 0.2767, acc: 88.4609, loss_bbox: 0.1529, loss_mask: 0.3430, loss: 1.1080
2020-08-13 15:03:06,587 - mmdet - INFO - Epoch [10][750/1350]   lr: 0.00250, eta: 16:45:02, time: 1.685, data_time: 0.867, memory: 6384, loss_rpn_cls: 0.1526, loss_rpn_bbox: 0.2110, loss_cls: 0.2901, acc: 87.8340, loss_bbox: 0.1557, loss_mask: 0.3327, loss: 1.1422
2020-08-13 15:04:30,102 - mmdet - INFO - Epoch [10][800/1350]   lr: 0.00250, eta: 16:43:35, time: 1.670, data_time: 0.881, memory: 6384, loss_rpn_cls: 0.1391, loss_rpn_bbox: 0.2001, loss_cls: 0.2883, acc: 87.9590, loss_bbox: 0.1575, loss_mask: 0.3407, loss: 1.1257
2020-08-13 15:05:53,328 - mmdet - INFO - Epoch [10][850/1350]   lr: 0.00250, eta: 16:42:07, time: 1.665, data_time: 0.874, memory: 6384, loss_rpn_cls: 0.1504, loss_rpn_bbox: 0.2029, loss_cls: 0.2831, acc: 88.0859, loss_bbox: 0.1552, loss_mask: 0.3318, loss: 1.1234
2020-08-13 15:07:17,942 - mmdet - INFO - Epoch [10][900/1350]   lr: 0.00250, eta: 16:40:43, time: 1.692, data_time: 0.892, memory: 6384, loss_rpn_cls: 0.1448, loss_rpn_bbox: 0.2032, loss_cls: 0.2885, acc: 88.0352, loss_bbox: 0.1604, loss_mask: 0.3351, loss: 1.1321
2020-08-13 15:08:46,840 - mmdet - INFO - Epoch [10][950/1350]   lr: 0.00250, eta: 16:39:30, time: 1.778, data_time: 0.953, memory: 6384, loss_rpn_cls: 0.1380, loss_rpn_bbox: 0.2003, loss_cls: 0.2873, acc: 87.9336, loss_bbox: 0.1591, loss_mask: 0.3425, loss: 1.1272
2020-08-13 15:10:15,581 - mmdet - INFO - Epoch [10][1000/1350]  lr: 0.00250, eta: 16:38:17, time: 1.775, data_time: 0.940, memory: 6384, loss_rpn_cls: 0.1394, loss_rpn_bbox: 0.2012, loss_cls: 0.2940, acc: 87.6484, loss_bbox: 0.1590, loss_mask: 0.3360, loss: 1.1296
2020-08-13 15:11:39,235 - mmdet - INFO - Epoch [10][1050/1350]  lr: 0.00250, eta: 16:36:51, time: 1.673, data_time: 0.838, memory: 6384, loss_rpn_cls: 0.1272, loss_rpn_bbox: 0.1960, loss_cls: 0.2789, acc: 88.2207, loss_bbox: 0.1535, loss_mask: 0.3287, loss: 1.0844
2020-08-13 15:13:08,401 - mmdet - INFO - Epoch [10][1100/1350]  lr: 0.00250, eta: 16:35:39, time: 1.783, data_time: 0.932, memory: 6384, loss_rpn_cls: 0.1367, loss_rpn_bbox: 0.2096, loss_cls: 0.2784, acc: 88.4219, loss_bbox: 0.1538, loss_mask: 0.3381, loss: 1.1167
2020-08-13 15:14:32,300 - mmdet - INFO - Epoch [10][1150/1350]  lr: 0.00250, eta: 16:34:13, time: 1.678, data_time: 0.822, memory: 6384, loss_rpn_cls: 0.1350, loss_rpn_bbox: 0.1980, loss_cls: 0.2789, acc: 88.3789, loss_bbox: 0.1521, loss_mask: 0.3319, loss: 1.0959
2020-08-13 15:15:57,311 - mmdet - INFO - Epoch [10][1200/1350]  lr: 0.00250, eta: 16:32:50, time: 1.700, data_time: 0.850, memory: 6384, loss_rpn_cls: 0.1379, loss_rpn_bbox: 0.1880, loss_cls: 0.2716, acc: 88.5508, loss_bbox: 0.1521, loss_mask: 0.3346, loss: 1.0842
2020-08-13 15:17:21,014 - mmdet - INFO - Epoch [10][1250/1350]  lr: 0.00250, eta: 16:31:23, time: 1.674, data_time: 0.817, memory: 6384, loss_rpn_cls: 0.1407, loss_rpn_bbox: 0.2022, loss_cls: 0.2808, acc: 88.3496, loss_bbox: 0.1580, loss_mask: 0.3390, loss: 1.1207
2020-08-13 15:18:46,848 - mmdet - INFO - Epoch [10][1300/1350]  lr: 0.00250, eta: 16:30:02, time: 1.717, data_time: 0.791, memory: 6384, loss_rpn_cls: 0.1230, loss_rpn_bbox: 0.1972, loss_cls: 0.2777, acc: 88.4316, loss_bbox: 0.1561, loss_mask: 0.3390, loss: 1.0930
2020-08-13 15:20:11,611 - mmdet - INFO - Epoch [10][1350/1350]  lr: 0.00250, eta: 16:28:38, time: 1.695, data_time: 0.872, memory: 6384, loss_rpn_cls: 0.1275, loss_rpn_bbox: 0.1902, loss_cls: 0.2780, acc: 88.3867, loss_bbox: 0.1576, loss_mask: 0.3362, loss: 1.0896
2020-08-13 16:07:04,366 - mmdet - INFO - Evaluating bbox...
2020-08-13 16:16:25,722 - mmdet - INFO - Epoch [10][1350/1350]  lr: 0.00250, bbox_mAP: 0.1710, bbox_mAP_50: 0.3500, bbox_mAP_75: 0.1490, bbox_mAP_s: 0.1360, bbox_mAP_m: 0.2950, bbox_mAP_l: 0.2380, bbox_mAP_copypaste: 0.171 0.350 0.149 0.136 0.295 0.238
2020-08-13 16:31:51,210 - mmdet - INFO - Epoch(train) [10][525] loss_rpn_cls: 0.1280, loss_rpn_bbox: 0.1861, loss_cls: 0.3093, acc: 87.2822, loss_bbox: 0.1538, loss_mask: 0.3305, loss: 1.1076
2020-08-13 16:33:25,201 - mmdet - INFO - Epoch [11][50/1350]    lr: 0.00250, eta: 16:27:38, time: 1.877, data_time: 1.053, memory: 6384, loss_rpn_cls: 0.1309, loss_rpn_bbox: 0.1929, loss_cls: 0.2741, acc: 88.4590, loss_bbox: 0.1516, loss_mask: 0.3467, loss: 1.0961
2020-08-13 16:34:52,410 - mmdet - INFO - Epoch [11][100/1350]   lr: 0.00250, eta: 16:26:20, time: 1.744, data_time: 0.902, memory: 6384, loss_rpn_cls: 0.1445, loss_rpn_bbox: 0.2005, loss_cls: 0.2720, acc: 88.8789, loss_bbox: 0.1519, loss_mask: 0.3279, loss: 1.0968
2020-08-13 16:36:21,131 - mmdet - INFO - Epoch [11][150/1350]   lr: 0.00250, eta: 16:25:06, time: 1.774, data_time: 0.938, memory: 6384, loss_rpn_cls: 0.1330, loss_rpn_bbox: 0.1936, loss_cls: 0.2702, acc: 88.6641, loss_bbox: 0.1531, loss_mask: 0.3346, loss: 1.0844
2020-08-13 16:37:45,780 - mmdet - INFO - Epoch [11][200/1350]   lr: 0.00250, eta: 16:23:42, time: 1.693, data_time: 0.857, memory: 6384, loss_rpn_cls: 0.1246, loss_rpn_bbox: 0.1929, loss_cls: 0.2831, acc: 88.2109, loss_bbox: 0.1545, loss_mask: 0.3239, loss: 1.0789
2020-08-13 16:39:11,073 - mmdet - INFO - Epoch [11][250/1350]   lr: 0.00250, eta: 16:22:19, time: 1.706, data_time: 0.884, memory: 6384, loss_rpn_cls: 0.1376, loss_rpn_bbox: 0.1994, loss_cls: 0.2855, acc: 88.0273, loss_bbox: 0.1573, loss_mask: 0.3322, loss: 1.1121
2020-08-13 16:40:38,718 - mmdet - INFO - Epoch [11][300/1350]   lr: 0.00250, eta: 16:21:03, time: 1.753, data_time: 0.936, memory: 6384, loss_rpn_cls: 0.1384, loss_rpn_bbox: 0.1984, loss_cls: 0.2797, acc: 88.4082, loss_bbox: 0.1567, loss_mask: 0.3273, loss: 1.1005
2020-08-13 16:42:06,575 - mmdet - INFO - Epoch [11][350/1350]   lr: 0.00250, eta: 16:19:46, time: 1.757, data_time: 0.940, memory: 6384, loss_rpn_cls: 0.1345, loss_rpn_bbox: 0.1921, loss_cls: 0.2704, acc: 88.7598, loss_bbox: 0.1485, loss_mask: 0.3244, loss: 1.0700
2020-08-13 16:43:30,019 - mmdet - INFO - Epoch [11][400/1350]   lr: 0.00250, eta: 16:18:19, time: 1.669, data_time: 0.835, memory: 6384, loss_rpn_cls: 0.1424, loss_rpn_bbox: 0.2056, loss_cls: 0.2906, acc: 87.8945, loss_bbox: 0.1605, loss_mask: 0.3251, loss: 1.1241
2020-08-13 16:44:55,487 - mmdet - INFO - Epoch [11][450/1350]   lr: 0.00250, eta: 16:16:56, time: 1.709, data_time: 0.892, memory: 6384, loss_rpn_cls: 0.1539, loss_rpn_bbox: 0.2084, loss_cls: 0.2942, acc: 87.7070, loss_bbox: 0.1613, loss_mask: 0.3316, loss: 1.1494
2020-08-13 16:46:20,833 - mmdet - INFO - Epoch [11][500/1350]   lr: 0.00250, eta: 16:15:34, time: 1.707, data_time: 0.908, memory: 6384, loss_rpn_cls: 0.1410, loss_rpn_bbox: 0.2019, loss_cls: 0.2859, acc: 87.9023, loss_bbox: 0.1589, loss_mask: 0.3364, loss: 1.1241
2020-08-13 16:47:45,753 - mmdet - INFO - Epoch [11][550/1350]   lr: 0.00250, eta: 16:14:10, time: 1.698, data_time: 0.885, memory: 6384, loss_rpn_cls: 0.1203, loss_rpn_bbox: 0.1957, loss_cls: 0.2758, acc: 88.3477, loss_bbox: 0.1538, loss_mask: 0.3260, loss: 1.0716
2020-08-13 16:49:11,268 - mmdet - INFO - Epoch [11][600/1350]   lr: 0.00250, eta: 16:12:48, time: 1.710, data_time: 0.888, memory: 6384, loss_rpn_cls: 0.1300, loss_rpn_bbox: 0.1980, loss_cls: 0.2706, acc: 88.8340, loss_bbox: 0.1532, loss_mask: 0.3412, loss: 1.0929
2020-08-13 16:50:39,166 - mmdet - INFO - Epoch [11][650/1350]   lr: 0.00250, eta: 16:11:31, time: 1.758, data_time: 0.934, memory: 6384, loss_rpn_cls: 0.1286, loss_rpn_bbox: 0.1990, loss_cls: 0.2815, acc: 88.0645, loss_bbox: 0.1558, loss_mask: 0.3444, loss: 1.1093
2020-08-13 16:52:03,531 - mmdet - INFO - Epoch [11][700/1350]   lr: 0.00250, eta: 16:10:06, time: 1.687, data_time: 0.842, memory: 6384, loss_rpn_cls: 0.1318, loss_rpn_bbox: 0.1953, loss_cls: 0.2881, acc: 87.9570, loss_bbox: 0.1571, loss_mask: 0.3339, loss: 1.1062
2020-08-13 16:53:28,750 - mmdet - INFO - Epoch [11][750/1350]   lr: 0.00250, eta: 16:08:43, time: 1.704, data_time: 0.873, memory: 6384, loss_rpn_cls: 0.1293, loss_rpn_bbox: 0.1964, loss_cls: 0.2835, acc: 88.1855, loss_bbox: 0.1557, loss_mask: 0.3350, loss: 1.0999
2020-08-13 16:54:49,767 - mmdet - INFO - Epoch [11][800/1350]   lr: 0.00250, eta: 16:07:10, time: 1.620, data_time: 0.804, memory: 6384, loss_rpn_cls: 0.1133, loss_rpn_bbox: 0.1869, loss_cls: 0.2653, acc: 88.9316, loss_bbox: 0.1497, loss_mask: 0.3354, loss: 1.0507
2020-08-13 16:56:13,948 - mmdet - INFO - Epoch [11][850/1350]   lr: 0.00250, eta: 16:05:44, time: 1.684, data_time: 0.889, memory: 6384, loss_rpn_cls: 0.1216, loss_rpn_bbox: 0.1951, loss_cls: 0.2793, acc: 88.2793, loss_bbox: 0.1555, loss_mask: 0.3266, loss: 1.0781
2020-08-13 16:57:38,950 - mmdet - INFO - Epoch [11][900/1350]   lr: 0.00250, eta: 16:04:20, time: 1.700, data_time: 0.899, memory: 6384, loss_rpn_cls: 0.1254, loss_rpn_bbox: 0.1974, loss_cls: 0.2726, acc: 88.4453, loss_bbox: 0.1543, loss_mask: 0.3365, loss: 1.0862
2020-08-13 16:59:02,981 - mmdet - INFO - Epoch [11][950/1350]   lr: 0.00250, eta: 16:02:54, time: 1.681, data_time: 0.877, memory: 6384, loss_rpn_cls: 0.1255, loss_rpn_bbox: 0.1986, loss_cls: 0.2659, acc: 88.8574, loss_bbox: 0.1496, loss_mask: 0.3410, loss: 1.0806
2020-08-13 17:00:31,782 - mmdet - INFO - Epoch [11][1000/1350]  lr: 0.00250, eta: 16:01:40, time: 1.776, data_time: 0.962, memory: 6384, loss_rpn_cls: 0.1387, loss_rpn_bbox: 0.2020, loss_cls: 0.2806, acc: 88.2598, loss_bbox: 0.1534, loss_mask: 0.3423, loss: 1.1170
2020-08-13 17:02:02,952 - mmdet - INFO - Epoch [11][1050/1350]  lr: 0.00250, eta: 16:00:31, time: 1.823, data_time: 0.893, memory: 6384, loss_rpn_cls: 0.1320, loss_rpn_bbox: 0.2024, loss_cls: 0.2784, acc: 88.2930, loss_bbox: 0.1589, loss_mask: 0.3365, loss: 1.1083
2020-08-13 17:03:23,459 - mmdet - INFO - Epoch [11][1100/1350]  lr: 0.00250, eta: 15:58:56, time: 1.610, data_time: 0.811, memory: 6384, loss_rpn_cls: 0.1178, loss_rpn_bbox: 0.1909, loss_cls: 0.2678, acc: 88.8535, loss_bbox: 0.1510, loss_mask: 0.3335, loss: 1.0611
2020-08-13 17:04:49,252 - mmdet - INFO - Epoch [11][1150/1350]  lr: 0.00250, eta: 15:57:34, time: 1.716, data_time: 0.901, memory: 6384, loss_rpn_cls: 0.1370, loss_rpn_bbox: 0.2063, loss_cls: 0.2925, acc: 87.8164, loss_bbox: 0.1609, loss_mask: 0.3341, loss: 1.1309
2020-08-13 17:06:15,973 - mmdet - INFO - Epoch [11][1200/1350]  lr: 0.00250, eta: 15:56:15, time: 1.734, data_time: 0.923, memory: 6384, loss_rpn_cls: 0.1386, loss_rpn_bbox: 0.2033, loss_cls: 0.2737, acc: 88.5957, loss_bbox: 0.1517, loss_mask: 0.3343, loss: 1.1015
2020-08-13 17:07:43,253 - mmdet - INFO - Epoch [11][1250/1350]  lr: 0.00250, eta: 15:54:56, time: 1.746, data_time: 0.919, memory: 6384, loss_rpn_cls: 0.1277, loss_rpn_bbox: 0.2086, loss_cls: 0.2789, acc: 88.3086, loss_bbox: 0.1571, loss_mask: 0.3563, loss: 1.1286
2020-08-13 17:09:12,388 - mmdet - INFO - Epoch [11][1300/1350]  lr: 0.00250, eta: 15:53:42, time: 1.783, data_time: 0.944, memory: 6384, loss_rpn_cls: 0.1350, loss_rpn_bbox: 0.2024, loss_cls: 0.2787, acc: 88.1816, loss_bbox: 0.1602, loss_mask: 0.3395, loss: 1.1158
2020-08-13 17:10:37,372 - mmdet - INFO - Epoch [11][1350/1350]  lr: 0.00250, eta: 15:52:18, time: 1.700, data_time: 0.859, memory: 6384, loss_rpn_cls: 0.1242, loss_rpn_bbox: 0.1962, loss_cls: 0.2871, acc: 88.2031, loss_bbox: 0.1606, loss_mask: 0.3358, loss: 1.1039
2020-08-13 18:05:54,335 - mmdet - INFO - Evaluating bbox...
2020-08-13 18:15:38,420 - mmdet - INFO - Epoch [11][1350/1350]  lr: 0.00250, bbox_mAP: 0.1600, bbox_mAP_50: 0.3390, bbox_mAP_75: 0.1300, bbox_mAP_s: 0.1240, bbox_mAP_m: 0.3050, bbox_mAP_l: 0.2230, bbox_mAP_copypaste: 0.160 0.339 0.130 0.124 0.305 0.223
2020-08-13 18:31:04,028 - mmdet - INFO - Epoch(train) [11][525] loss_rpn_cls: 0.1593, loss_rpn_bbox: 0.1915, loss_cls: 0.3079, acc: 87.1393, loss_bbox: 0.1613, loss_mask: 0.3371, loss: 1.1571
2020-08-13 18:32:37,467 - mmdet - INFO - Epoch [12][50/1350]    lr: 0.00250, eta: 15:51:13, time: 1.866, data_time: 1.044, memory: 6384, loss_rpn_cls: 0.1356, loss_rpn_bbox: 0.2046, loss_cls: 0.2935, acc: 87.6680, loss_bbox: 0.1602, loss_mask: 0.3278, loss: 1.1216
2020-08-13 18:34:03,762 - mmdet - INFO - Epoch [12][100/1350]   lr: 0.00250, eta: 15:49:52, time: 1.726, data_time: 0.924, memory: 6384, loss_rpn_cls: 0.1325, loss_rpn_bbox: 0.1990, loss_cls: 0.2817, acc: 88.2305, loss_bbox: 0.1607, loss_mask: 0.3417, loss: 1.1156
2020-08-13 18:35:32,063 - mmdet - INFO - Epoch [12][150/1350]   lr: 0.00250, eta: 15:48:35, time: 1.766, data_time: 0.938, memory: 6384, loss_rpn_cls: 0.1672, loss_rpn_bbox: 0.2111, loss_cls: 0.3057, acc: 87.1348, loss_bbox: 0.1661, loss_mask: 0.3331, loss: 1.1831
2020-08-13 18:36:55,701 - mmdet - INFO - Epoch [12][200/1350]   lr: 0.00250, eta: 15:47:08, time: 1.673, data_time: 0.839, memory: 6384, loss_rpn_cls: 0.1073, loss_rpn_bbox: 0.1851, loss_cls: 0.2709, acc: 88.5742, loss_bbox: 0.1538, loss_mask: 0.3424, loss: 1.0594
2020-08-13 18:38:24,944 - mmdet - INFO - Epoch [12][250/1350]   lr: 0.00250, eta: 15:45:54, time: 1.785, data_time: 0.946, memory: 6384, loss_rpn_cls: 0.1282, loss_rpn_bbox: 0.2082, loss_cls: 0.2883, acc: 87.7969, loss_bbox: 0.1607, loss_mask: 0.3367, loss: 1.1222
2020-08-13 18:39:47,686 - mmdet - INFO - Epoch [12][300/1350]   lr: 0.00250, eta: 15:44:25, time: 1.655, data_time: 0.816, memory: 6384, loss_rpn_cls: 0.1389, loss_rpn_bbox: 0.2049, loss_cls: 0.2815, acc: 88.4434, loss_bbox: 0.1536, loss_mask: 0.3340, loss: 1.1129
2020-08-13 18:41:12,482 - mmdet - INFO - Epoch [12][350/1350]   lr: 0.00250, eta: 15:43:00, time: 1.696, data_time: 0.901, memory: 6384, loss_rpn_cls: 0.1416, loss_rpn_bbox: 0.2069, loss_cls: 0.2854, acc: 88.0840, loss_bbox: 0.1584, loss_mask: 0.3373, loss: 1.1296
2020-08-13 18:42:37,415 - mmdet - INFO - Epoch [12][400/1350]   lr: 0.00250, eta: 15:41:36, time: 1.699, data_time: 0.897, memory: 6384, loss_rpn_cls: 0.1424, loss_rpn_bbox: 0.2091, loss_cls: 0.2968, acc: 87.3984, loss_bbox: 0.1632, loss_mask: 0.3325, loss: 1.1440
2020-08-13 18:44:05,658 - mmdet - INFO - Epoch [12][450/1350]   lr: 0.00250, eta: 15:40:19, time: 1.765, data_time: 0.950, memory: 6384, loss_rpn_cls: 0.1229, loss_rpn_bbox: 0.2016, loss_cls: 0.2883, acc: 88.0625, loss_bbox: 0.1596, loss_mask: 0.3340, loss: 1.1062
2020-08-13 18:45:29,533 - mmdet - INFO - Epoch [12][500/1350]   lr: 0.00250, eta: 15:38:52, time: 1.677, data_time: 0.849, memory: 6384, loss_rpn_cls: 0.1333, loss_rpn_bbox: 0.2003, loss_cls: 0.2859, acc: 88.0059, loss_bbox: 0.1606, loss_mask: 0.3367, loss: 1.1169
2020-08-13 18:46:55,073 - mmdet - INFO - Epoch [12][550/1350]   lr: 0.00250, eta: 15:37:29, time: 1.711, data_time: 0.898, memory: 6384, loss_rpn_cls: 0.1334, loss_rpn_bbox: 0.2045, loss_cls: 0.2886, acc: 88.0645, loss_bbox: 0.1603, loss_mask: 0.3432, loss: 1.1300
2020-08-13 18:48:19,727 - mmdet - INFO - Epoch [12][600/1350]   lr: 0.00250, eta: 15:36:05, time: 1.693, data_time: 0.875, memory: 6384, loss_rpn_cls: 0.1430, loss_rpn_bbox: 0.2060, loss_cls: 0.2947, acc: 87.5527, loss_bbox: 0.1605, loss_mask: 0.3345, loss: 1.1386
2020-08-13 18:49:48,382 - mmdet - INFO - Epoch [12][650/1350]   lr: 0.00250, eta: 15:34:48, time: 1.773, data_time: 0.948, memory: 6384, loss_rpn_cls: 0.1430, loss_rpn_bbox: 0.1998, loss_cls: 0.2782, acc: 88.2461, loss_bbox: 0.1574, loss_mask: 0.3450, loss: 1.1234
2020-08-13 18:51:13,472 - mmdet - INFO - Epoch [12][700/1350]   lr: 0.00250, eta: 15:33:24, time: 1.702, data_time: 0.858, memory: 6384, loss_rpn_cls: 0.1450, loss_rpn_bbox: 0.2066, loss_cls: 0.2859, acc: 88.0840, loss_bbox: 0.1611, loss_mask: 0.3487, loss: 1.1473
2020-08-13 18:52:39,979 - mmdet - INFO - Epoch [12][750/1350]   lr: 0.00250, eta: 15:32:03, time: 1.730, data_time: 0.913, memory: 6384, loss_rpn_cls: 0.1202, loss_rpn_bbox: 0.1958, loss_cls: 0.2860, acc: 87.8750, loss_bbox: 0.1558, loss_mask: 0.3435, loss: 1.1012
2020-08-13 18:54:05,643 - mmdet - INFO - Epoch [12][800/1350]   lr: 0.00250, eta: 15:30:41, time: 1.713, data_time: 0.886, memory: 6384, loss_rpn_cls: 0.1233, loss_rpn_bbox: 0.2011, loss_cls: 0.2868, acc: 88.0957, loss_bbox: 0.1605, loss_mask: 0.3339, loss: 1.1056
2020-08-13 18:55:35,821 - mmdet - INFO - Epoch [12][850/1350]   lr: 0.00250, eta: 15:29:27, time: 1.804, data_time: 0.975, memory: 6384, loss_rpn_cls: 0.1409, loss_rpn_bbox: 0.2080, loss_cls: 0.2870, acc: 87.8809, loss_bbox: 0.1593, loss_mask: 0.3408, loss: 1.1360
2020-08-13 18:56:58,420 - mmdet - INFO - Epoch [12][900/1350]   lr: 0.00250, eta: 15:27:58, time: 1.652, data_time: 0.807, memory: 6384, loss_rpn_cls: 0.1321, loss_rpn_bbox: 0.2060, loss_cls: 0.2826, acc: 88.1406, loss_bbox: 0.1579, loss_mask: 0.3422, loss: 1.1209
2020-08-13 18:58:25,445 - mmdet - INFO - Epoch [12][950/1350]   lr: 0.00250, eta: 15:26:38, time: 1.740, data_time: 0.893, memory: 6384, loss_rpn_cls: 0.1273, loss_rpn_bbox: 0.2044, loss_cls: 0.2857, acc: 88.0898, loss_bbox: 0.1579, loss_mask: 0.3504, loss: 1.1257
2020-08-13 18:59:47,797 - mmdet - INFO - Epoch [12][1000/1350]  lr: 0.00250, eta: 15:25:08, time: 1.647, data_time: 0.827, memory: 6384, loss_rpn_cls: 0.1183, loss_rpn_bbox: 0.1935, loss_cls: 0.2665, acc: 88.7637, loss_bbox: 0.1556, loss_mask: 0.3476, loss: 1.0815
2020-08-13 19:01:14,874 - mmdet - INFO - Epoch [12][1050/1350]  lr: 0.00250, eta: 15:23:48, time: 1.742, data_time: 0.910, memory: 6384, loss_rpn_cls: 0.1146, loss_rpn_bbox: 0.1967, loss_cls: 0.2866, acc: 87.8613, loss_bbox: 0.1636, loss_mask: 0.3409, loss: 1.1023
2020-08-13 19:02:38,354 - mmdet - INFO - Epoch [12][1100/1350]  lr: 0.00250, eta: 15:22:21, time: 1.670, data_time: 0.837, memory: 6384, loss_rpn_cls: 0.1205, loss_rpn_bbox: 0.1959, loss_cls: 0.2697, acc: 88.6289, loss_bbox: 0.1531, loss_mask: 0.3372, loss: 1.0764
2020-08-13 19:04:03,070 - mmdet - INFO - Epoch [12][1150/1350]  lr: 0.00250, eta: 15:20:56, time: 1.694, data_time: 0.862, memory: 6384, loss_rpn_cls: 0.1408, loss_rpn_bbox: 0.2123, loss_cls: 0.2942, acc: 87.7227, loss_bbox: 0.1649, loss_mask: 0.3438, loss: 1.1560
2020-08-13 19:05:28,245 - mmdet - INFO - Epoch [12][1200/1350]  lr: 0.00250, eta: 15:19:32, time: 1.703, data_time: 0.876, memory: 6384, loss_rpn_cls: 0.1072, loss_rpn_bbox: 0.1961, loss_cls: 0.2846, acc: 87.9648, loss_bbox: 0.1589, loss_mask: 0.3427, loss: 1.0895
2020-08-13 19:06:53,368 - mmdet - INFO - Epoch [12][1250/1350]  lr: 0.00250, eta: 15:18:08, time: 1.702, data_time: 0.867, memory: 6384, loss_rpn_cls: 0.1223, loss_rpn_bbox: 0.2056, loss_cls: 0.2994, acc: 87.2871, loss_bbox: 0.1668, loss_mask: 0.3423, loss: 1.1365
2020-08-13 19:08:18,845 - mmdet - INFO - Epoch [12][1300/1350]  lr: 0.00250, eta: 15:16:45, time: 1.710, data_time: 0.880, memory: 6384, loss_rpn_cls: 0.1256, loss_rpn_bbox: 0.2079, loss_cls: 0.2868, acc: 87.8672, loss_bbox: 0.1636, loss_mask: 0.3476, loss: 1.1314
2020-08-13 19:09:44,241 - mmdet - INFO - Epoch [12][1350/1350]  lr: 0.00250, eta: 15:15:21, time: 1.708, data_time: 0.893, memory: 6384, loss_rpn_cls: 0.1249, loss_rpn_bbox: 0.2008, loss_cls: 0.2970, acc: 87.5039, loss_bbox: 0.1621, loss_mask: 0.3442, loss: 1.1290
2020-08-13 19:58:59,723 - mmdet - INFO - Evaluating bbox...
2020-08-13 20:08:30,340 - mmdet - INFO - Epoch [12][1350/1350]  lr: 0.00250, bbox_mAP: 0.1630, bbox_mAP_50: 0.3380, bbox_mAP_75: 0.1360, bbox_mAP_s: 0.1320, bbox_mAP_m: 0.2910, bbox_mAP_l: 0.1990, bbox_mAP_copypaste: 0.163 0.338 0.136 0.132 0.291 0.199
2020-08-13 20:23:58,625 - mmdet - INFO - Epoch(train) [12][525] loss_rpn_cls: 0.1335, loss_rpn_bbox: 0.1912, loss_cls: 0.3140, acc: 86.9254, loss_bbox: 0.1590, loss_mask: 0.3314, loss: 1.1292
2020-08-13 20:25:30,697 - mmdet - INFO - Epoch [13][50/1350]    lr: 0.00250, eta: 15:14:11, time: 1.839, data_time: 0.998, memory: 6384, loss_rpn_cls: 0.1319, loss_rpn_bbox: 0.2122, loss_cls: 0.2991, acc: 87.4863, loss_bbox: 0.1661, loss_mask: 0.3438, loss: 1.1530
2020-08-13 20:26:56,741 - mmdet - INFO - Epoch [13][100/1350]   lr: 0.00250, eta: 15:12:49, time: 1.721, data_time: 0.913, memory: 6384, loss_rpn_cls: 0.1230, loss_rpn_bbox: 0.1985, loss_cls: 0.2785, acc: 88.3711, loss_bbox: 0.1568, loss_mask: 0.3490, loss: 1.1058
2020-08-13 20:28:26,690 - mmdet - INFO - Epoch [13][150/1350]   lr: 0.00250, eta: 15:11:34, time: 1.799, data_time: 0.983, memory: 6384, loss_rpn_cls: 0.1376, loss_rpn_bbox: 0.2097, loss_cls: 0.2971, acc: 87.5195, loss_bbox: 0.1616, loss_mask: 0.3438, loss: 1.1497
2020-08-13 20:29:50,060 - mmdet - INFO - Epoch [13][200/1350]   lr: 0.00250, eta: 15:10:07, time: 1.667, data_time: 0.851, memory: 6384, loss_rpn_cls: 0.1456, loss_rpn_bbox: 0.2137, loss_cls: 0.3102, acc: 86.8301, loss_bbox: 0.1739, loss_mask: 0.3430, loss: 1.1863
2020-08-13 20:31:18,423 - mmdet - INFO - Epoch [13][250/1350]   lr: 0.00250, eta: 15:08:49, time: 1.767, data_time: 0.950, memory: 6384, loss_rpn_cls: 0.1387, loss_rpn_bbox: 0.2103, loss_cls: 0.2953, acc: 87.8086, loss_bbox: 0.1670, loss_mask: 0.3508, loss: 1.1622
2020-08-13 20:32:40,882 - mmdet - INFO - Epoch [13][300/1350]   lr: 0.00250, eta: 15:07:19, time: 1.649, data_time: 0.831, memory: 6384, loss_rpn_cls: 0.1272, loss_rpn_bbox: 0.2113, loss_cls: 0.2984, acc: 87.2461, loss_bbox: 0.1656, loss_mask: 0.3571, loss: 1.1595
2020-08-13 20:34:04,865 - mmdet - INFO - Epoch [13][350/1350]   lr: 0.00250, eta: 15:05:53, time: 1.680, data_time: 0.864, memory: 6384, loss_rpn_cls: 0.1424, loss_rpn_bbox: 0.2167, loss_cls: 0.3119, acc: 86.6602, loss_bbox: 0.1698, loss_mask: 0.3571, loss: 1.1978
2020-08-13 20:35:32,274 - mmdet - INFO - Epoch [13][400/1350]   lr: 0.00250, eta: 15:04:33, time: 1.748, data_time: 0.948, memory: 6384, loss_rpn_cls: 0.1229, loss_rpn_bbox: 0.2053, loss_cls: 0.2932, acc: 87.7578, loss_bbox: 0.1604, loss_mask: 0.3450, loss: 1.1268
2020-08-13 20:37:00,372 - mmdet - INFO - Epoch [13][450/1350]   lr: 0.00250, eta: 15:03:15, time: 1.762, data_time: 0.928, memory: 6384, loss_rpn_cls: 0.1277, loss_rpn_bbox: 0.2098, loss_cls: 0.3046, acc: 87.1738, loss_bbox: 0.1674, loss_mask: 0.3488, loss: 1.1583
2020-08-13 20:38:24,133 - mmdet - INFO - Epoch [13][500/1350]   lr: 0.00250, eta: 15:01:48, time: 1.675, data_time: 0.832, memory: 6384, loss_rpn_cls: 0.1418, loss_rpn_bbox: 0.2154, loss_cls: 0.3077, acc: 86.9668, loss_bbox: 0.1675, loss_mask: 0.3485, loss: 1.1809
2020-08-13 20:39:52,707 - mmdet - INFO - Epoch [13][550/1350]   lr: 0.00250, eta: 15:00:30, time: 1.771, data_time: 0.925, memory: 6384, loss_rpn_cls: 0.1478, loss_rpn_bbox: 0.2245, loss_cls: 0.3073, acc: 87.0488, loss_bbox: 0.1719, loss_mask: 0.3518, loss: 1.2033
2020-08-13 20:41:18,099 - mmdet - INFO - Epoch [13][600/1350]   lr: 0.00250, eta: 14:59:07, time: 1.708, data_time: 0.867, memory: 6384, loss_rpn_cls: 0.1215, loss_rpn_bbox: 0.2086, loss_cls: 0.2845, acc: 87.9648, loss_bbox: 0.1547, loss_mask: 0.3578, loss: 1.1272
2020-08-13 20:42:43,760 - mmdet - INFO - Epoch [13][650/1350]   lr: 0.00250, eta: 14:57:43, time: 1.713, data_time: 0.864, memory: 6384, loss_rpn_cls: 0.1698, loss_rpn_bbox: 0.2231, loss_cls: 0.3146, acc: 86.8145, loss_bbox: 0.1750, loss_mask: 0.3437, loss: 1.2262
2020-08-13 20:44:06,089 - mmdet - INFO - Epoch [13][700/1350]   lr: 0.00250, eta: 14:56:14, time: 1.647, data_time: 0.817, memory: 6384, loss_rpn_cls: 0.1340, loss_rpn_bbox: 0.2094, loss_cls: 0.3070, acc: 87.2500, loss_bbox: 0.1611, loss_mask: 0.3404, loss: 1.1520
2020-08-13 20:45:35,552 - mmdet - INFO - Epoch [13][750/1350]   lr: 0.00250, eta: 14:54:58, time: 1.789, data_time: 0.948, memory: 6384, loss_rpn_cls: 0.1244, loss_rpn_bbox: 0.2079, loss_cls: 0.2875, acc: 87.9590, loss_bbox: 0.1572, loss_mask: 0.3465, loss: 1.1235
2020-08-13 20:46:59,976 - mmdet - INFO - Epoch [13][800/1350]   lr: 0.00250, eta: 14:53:32, time: 1.688, data_time: 0.838, memory: 6384, loss_rpn_cls: 0.1362, loss_rpn_bbox: 0.2109, loss_cls: 0.2974, acc: 87.3398, loss_bbox: 0.1620, loss_mask: 0.3482, loss: 1.1547
2020-08-13 20:48:28,271 - mmdet - INFO - Epoch [13][850/1350]   lr: 0.00250, eta: 14:52:14, time: 1.766, data_time: 0.918, memory: 6384, loss_rpn_cls: 0.1488, loss_rpn_bbox: 0.2174, loss_cls: 0.3041, acc: 87.1758, loss_bbox: 0.1628, loss_mask: 0.3567, loss: 1.1898
2020-08-13 20:49:50,533 - mmdet - INFO - Epoch [13][900/1350]   lr: 0.00250, eta: 14:50:44, time: 1.645, data_time: 0.832, memory: 6384, loss_rpn_cls: 0.1308, loss_rpn_bbox: 0.2074, loss_cls: 0.2935, acc: 87.5664, loss_bbox: 0.1619, loss_mask: 0.3398, loss: 1.1333
2020-08-13 20:51:16,200 - mmdet - INFO - Epoch [13][950/1350]   lr: 0.00250, eta: 14:49:21, time: 1.713, data_time: 0.904, memory: 6384, loss_rpn_cls: 0.1281, loss_rpn_bbox: 0.2091, loss_cls: 0.2878, acc: 87.8359, loss_bbox: 0.1607, loss_mask: 0.3524, loss: 1.1382
2020-08-13 20:52:44,133 - mmdet - INFO - Epoch [13][1000/1350]  lr: 0.00250, eta: 14:48:02, time: 1.759, data_time: 0.928, memory: 6384, loss_rpn_cls: 0.1423, loss_rpn_bbox: 0.2133, loss_cls: 0.3063, acc: 87.0859, loss_bbox: 0.1607, loss_mask: 0.3508, loss: 1.1734
2020-08-13 20:54:11,027 - mmdet - INFO - Epoch [13][1050/1350]  lr: 0.00250, eta: 14:46:40, time: 1.738, data_time: 0.912, memory: 6384, loss_rpn_cls: 0.1277, loss_rpn_bbox: 0.2094, loss_cls: 0.2926, acc: 87.6426, loss_bbox: 0.1598, loss_mask: 0.3470, loss: 1.1366
2020-08-13 20:55:35,828 - mmdet - INFO - Epoch [13][1100/1350]  lr: 0.00250, eta: 14:45:15, time: 1.696, data_time: 0.866, memory: 6384, loss_rpn_cls: 0.1170, loss_rpn_bbox: 0.2110, loss_cls: 0.2938, acc: 87.5039, loss_bbox: 0.1640, loss_mask: 0.3436, loss: 1.1294
2020-08-13 20:57:02,841 - mmdet - INFO - Epoch [13][1150/1350]  lr: 0.00250, eta: 14:43:55, time: 1.740, data_time: 0.904, memory: 6384, loss_rpn_cls: 0.1459, loss_rpn_bbox: 0.2200, loss_cls: 0.2970, acc: 87.7090, loss_bbox: 0.1602, loss_mask: 0.3399, loss: 1.1629
2020-08-13 20:58:28,863 - mmdet - INFO - Epoch [13][1200/1350]  lr: 0.00250, eta: 14:42:32, time: 1.720, data_time: 0.872, memory: 6384, loss_rpn_cls: 0.1367, loss_rpn_bbox: 0.2104, loss_cls: 0.2928, acc: 87.6074, loss_bbox: 0.1589, loss_mask: 0.3379, loss: 1.1368
2020-08-13 20:59:53,857 - mmdet - INFO - Epoch [13][1250/1350]  lr: 0.00250, eta: 14:41:07, time: 1.700, data_time: 0.896, memory: 6384, loss_rpn_cls: 0.1098, loss_rpn_bbox: 0.2045, loss_cls: 0.2846, acc: 87.8770, loss_bbox: 0.1581, loss_mask: 0.3561, loss: 1.1131
2020-08-13 21:01:23,786 - mmdet - INFO - Epoch [13][1300/1350]  lr: 0.00250, eta: 14:39:51, time: 1.799, data_time: 0.953, memory: 6384, loss_rpn_cls: 0.1165, loss_rpn_bbox: 0.2015, loss_cls: 0.2845, acc: 87.9688, loss_bbox: 0.1509, loss_mask: 0.3359, loss: 1.0892
2020-08-13 21:02:55,644 - mmdet - INFO - Epoch [13][1350/1350]  lr: 0.00250, eta: 14:38:39, time: 1.837, data_time: 0.900, memory: 6384, loss_rpn_cls: 0.1369, loss_rpn_bbox: 0.2158, loss_cls: 0.2975, acc: 87.5410, loss_bbox: 0.1574, loss_mask: 0.3379, loss: 1.1455
2020-08-13 21:52:28,909 - mmdet - INFO - Evaluating bbox...
2020-08-13 22:02:09,054 - mmdet - INFO - Epoch [13][1350/1350]  lr: 0.00250, bbox_mAP: 0.1550, bbox_mAP_50: 0.3370, bbox_mAP_75: 0.1260, bbox_mAP_s: 0.1270, bbox_mAP_m: 0.2820, bbox_mAP_l: 0.1990, bbox_mAP_copypaste: 0.155 0.337 0.126 0.127 0.282 0.199
2020-08-13 22:17:40,631 - mmdet - INFO - Epoch(train) [13][525] loss_rpn_cls: 0.1274, loss_rpn_bbox: 0.1991, loss_cls: 0.3295, acc: 85.9883, loss_bbox: 0.1689, loss_mask: 0.3525, loss: 1.1774
2020-08-13 22:19:16,448 - mmdet - INFO - Epoch [14][50/1350]    lr: 0.00250, eta: 14:37:33, time: 1.914, data_time: 1.096, memory: 6384, loss_rpn_cls: 0.1224, loss_rpn_bbox: 0.2054, loss_cls: 0.2928, acc: 87.3770, loss_bbox: 0.1577, loss_mask: 0.3542, loss: 1.1325
2020-08-13 22:20:41,617 - mmdet - INFO - Epoch [14][100/1350]   lr: 0.00250, eta: 14:36:08, time: 1.703, data_time: 0.881, memory: 6384, loss_rpn_cls: 0.1437, loss_rpn_bbox: 0.2169, loss_cls: 0.3011, acc: 87.3691, loss_bbox: 0.1624, loss_mask: 0.3442, loss: 1.1683
2020-08-13 22:22:06,700 - mmdet - INFO - Epoch [14][150/1350]   lr: 0.00250, eta: 14:34:44, time: 1.702, data_time: 0.877, memory: 6384, loss_rpn_cls: 0.1344, loss_rpn_bbox: 0.2127, loss_cls: 0.3041, acc: 87.2656, loss_bbox: 0.1639, loss_mask: 0.3501, loss: 1.1651
2020-08-13 22:23:33,226 - mmdet - INFO - Epoch [14][200/1350]   lr: 0.00250, eta: 14:33:22, time: 1.731, data_time: 0.916, memory: 6384, loss_rpn_cls: 0.1245, loss_rpn_bbox: 0.2067, loss_cls: 0.2935, acc: 87.6758, loss_bbox: 0.1548, loss_mask: 0.3535, loss: 1.1330
2020-08-13 22:24:59,665 - mmdet - INFO - Epoch [14][250/1350]   lr: 0.00250, eta: 14:31:59, time: 1.729, data_time: 0.893, memory: 6384, loss_rpn_cls: 0.1282, loss_rpn_bbox: 0.2062, loss_cls: 0.2838, acc: 88.0859, loss_bbox: 0.1576, loss_mask: 0.3373, loss: 1.1130
2020-08-13 22:26:23,035 - mmdet - INFO - Epoch [14][300/1350]   lr: 0.00250, eta: 14:30:32, time: 1.667, data_time: 0.834, memory: 6384, loss_rpn_cls: 0.1170, loss_rpn_bbox: 0.2074, loss_cls: 0.2964, acc: 87.4004, loss_bbox: 0.1599, loss_mask: 0.3453, loss: 1.1260
2020-08-13 22:27:51,351 - mmdet - INFO - Epoch [14][350/1350]   lr: 0.00250, eta: 14:29:13, time: 1.766, data_time: 0.927, memory: 6384, loss_rpn_cls: 0.1273, loss_rpn_bbox: 0.2125, loss_cls: 0.2928, acc: 87.6211, loss_bbox: 0.1578, loss_mask: 0.3428, loss: 1.1332
2020-08-13 22:29:12,856 - mmdet - INFO - Epoch [14][400/1350]   lr: 0.00250, eta: 14:27:42, time: 1.630, data_time: 0.805, memory: 6384, loss_rpn_cls: 0.1307, loss_rpn_bbox: 0.2099, loss_cls: 0.2950, acc: 87.4941, loss_bbox: 0.1617, loss_mask: 0.3435, loss: 1.1408
2020-08-13 22:30:38,078 - mmdet - INFO - Epoch [14][450/1350]   lr: 0.00250, eta: 14:26:17, time: 1.704, data_time: 0.892, memory: 6384, loss_rpn_cls: 0.1267, loss_rpn_bbox: 0.2051, loss_cls: 0.2880, acc: 88.0547, loss_bbox: 0.1614, loss_mask: 0.3431, loss: 1.1244
2020-08-13 22:32:02,929 - mmdet - INFO - Epoch [14][500/1350]   lr: 0.00250, eta: 14:24:52, time: 1.697, data_time: 0.907, memory: 6384, loss_rpn_cls: 0.1355, loss_rpn_bbox: 0.2188, loss_cls: 0.3093, acc: 86.6777, loss_bbox: 0.1641, loss_mask: 0.3458, loss: 1.1735
2020-08-13 22:33:28,799 - mmdet - INFO - Epoch [14][550/1350]   lr: 0.00250, eta: 14:23:29, time: 1.717, data_time: 0.900, memory: 6384, loss_rpn_cls: 0.1416, loss_rpn_bbox: 0.2117, loss_cls: 0.2920, acc: 87.7559, loss_bbox: 0.1608, loss_mask: 0.3360, loss: 1.1419
2020-08-13 22:34:56,923 - mmdet - INFO - Epoch [14][600/1350]   lr: 0.00250, eta: 14:22:09, time: 1.762, data_time: 0.938, memory: 6384, loss_rpn_cls: 0.1343, loss_rpn_bbox: 0.2075, loss_cls: 0.2922, acc: 88.0234, loss_bbox: 0.1542, loss_mask: 0.3381, loss: 1.1262
2020-08-13 22:36:22,164 - mmdet - INFO - Epoch [14][650/1350]   lr: 0.00250, eta: 14:20:45, time: 1.705, data_time: 0.885, memory: 6384, loss_rpn_cls: 0.1226, loss_rpn_bbox: 0.2066, loss_cls: 0.2788, acc: 88.2656, loss_bbox: 0.1500, loss_mask: 0.3446, loss: 1.1026
2020-08-13 22:37:48,208 - mmdet - INFO - Epoch [14][700/1350]   lr: 0.00250, eta: 14:19:22, time: 1.721, data_time: 0.896, memory: 6384, loss_rpn_cls: 0.1193, loss_rpn_bbox: 0.2021, loss_cls: 0.2860, acc: 87.9023, loss_bbox: 0.1585, loss_mask: 0.3374, loss: 1.1033
2020-08-13 22:39:14,844 - mmdet - INFO - Epoch [14][750/1350]   lr: 0.00250, eta: 14:18:00, time: 1.733, data_time: 0.905, memory: 6384, loss_rpn_cls: 0.1274, loss_rpn_bbox: 0.2161, loss_cls: 0.3007, acc: 87.3047, loss_bbox: 0.1620, loss_mask: 0.3502, loss: 1.1564
2020-08-13 22:40:39,247 - mmdet - INFO - Epoch [14][800/1350]   lr: 0.00250, eta: 14:16:34, time: 1.688, data_time: 0.857, memory: 6384, loss_rpn_cls: 0.1442, loss_rpn_bbox: 0.2240, loss_cls: 0.3119, acc: 86.7695, loss_bbox: 0.1690, loss_mask: 0.3555, loss: 1.2045
2020-08-13 22:42:06,684 - mmdet - INFO - Epoch [14][850/1350]   lr: 0.00250, eta: 14:15:13, time: 1.749, data_time: 0.905, memory: 6384, loss_rpn_cls: 0.1277, loss_rpn_bbox: 0.2062, loss_cls: 0.2836, acc: 88.1797, loss_bbox: 0.1560, loss_mask: 0.3409, loss: 1.1144
2020-08-13 22:43:29,223 - mmdet - INFO - Epoch [14][900/1350]   lr: 0.00250, eta: 14:13:44, time: 1.651, data_time: 0.835, memory: 6384, loss_rpn_cls: 0.1272, loss_rpn_bbox: 0.2074, loss_cls: 0.2945, acc: 87.5684, loss_bbox: 0.1589, loss_mask: 0.3444, loss: 1.1325
2020-08-13 22:44:57,133 - mmdet - INFO - Epoch [14][950/1350]   lr: 0.00250, eta: 14:12:24, time: 1.758, data_time: 0.939, memory: 6384, loss_rpn_cls: 0.1413, loss_rpn_bbox: 0.2135, loss_cls: 0.2935, acc: 87.6953, loss_bbox: 0.1572, loss_mask: 0.3485, loss: 1.1541
2020-08-13 22:46:23,511 - mmdet - INFO - Epoch [14][1000/1350]  lr: 0.00250, eta: 14:11:01, time: 1.728, data_time: 0.913, memory: 6384, loss_rpn_cls: 0.1468, loss_rpn_bbox: 0.2191, loss_cls: 0.2881, acc: 87.9609, loss_bbox: 0.1553, loss_mask: 0.3349, loss: 1.1441
2020-08-13 22:47:55,188 - mmdet - INFO - Epoch [14][1050/1350]  lr: 0.00250, eta: 14:09:47, time: 1.834, data_time: 0.906, memory: 6384, loss_rpn_cls: 0.1131, loss_rpn_bbox: 0.2086, loss_cls: 0.2894, acc: 87.7676, loss_bbox: 0.1578, loss_mask: 0.3490, loss: 1.1179
2020-08-13 22:49:16,676 - mmdet - INFO - Epoch [14][1100/1350]  lr: 0.00250, eta: 14:08:16, time: 1.630, data_time: 0.816, memory: 6384, loss_rpn_cls: 0.1402, loss_rpn_bbox: 0.2213, loss_cls: 0.3103, acc: 87.0371, loss_bbox: 0.1611, loss_mask: 0.3521, loss: 1.1848
2020-08-13 22:50:41,817 - mmdet - INFO - Epoch [14][1150/1350]  lr: 0.00250, eta: 14:06:52, time: 1.703, data_time: 0.909, memory: 6384, loss_rpn_cls: 0.1204, loss_rpn_bbox: 0.2125, loss_cls: 0.2829, acc: 88.1484, loss_bbox: 0.1490, loss_mask: 0.3480, loss: 1.1128
2020-08-13 22:52:06,646 - mmdet - INFO - Epoch [14][1200/1350]  lr: 0.00250, eta: 14:05:27, time: 1.697, data_time: 0.898, memory: 6384, loss_rpn_cls: 0.1356, loss_rpn_bbox: 0.2119, loss_cls: 0.2980, acc: 87.3926, loss_bbox: 0.1604, loss_mask: 0.3460, loss: 1.1519
2020-08-13 22:53:34,999 - mmdet - INFO - Epoch [14][1250/1350]  lr: 0.00250, eta: 14:04:07, time: 1.767, data_time: 0.943, memory: 6384, loss_rpn_cls: 0.1353, loss_rpn_bbox: 0.2187, loss_cls: 0.2895, acc: 87.8145, loss_bbox: 0.1546, loss_mask: 0.3312, loss: 1.1293
2020-08-13 22:55:01,154 - mmdet - INFO - Epoch [14][1300/1350]  lr: 0.00250, eta: 14:02:44, time: 1.723, data_time: 0.887, memory: 6384, loss_rpn_cls: 0.1270, loss_rpn_bbox: 0.2207, loss_cls: 0.3047, acc: 87.1367, loss_bbox: 0.1576, loss_mask: 0.3329, loss: 1.1429
2020-08-13 22:56:26,438 - mmdet - INFO - Epoch [14][1350/1350]  lr: 0.00250, eta: 14:01:19, time: 1.706, data_time: 0.873, memory: 6384, loss_rpn_cls: 0.1271, loss_rpn_bbox: 0.2076, loss_cls: 0.2919, acc: 87.7500, loss_bbox: 0.1589, loss_mask: 0.3458, loss: 1.1313
2020-08-13 23:49:35,105 - mmdet - INFO - Evaluating bbox...
2020-08-13 23:58:45,324 - mmdet - INFO - Epoch [14][1350/1350]  lr: 0.00250, bbox_mAP: 0.1560, bbox_mAP_50: 0.3400, bbox_mAP_75: 0.1240, bbox_mAP_s: 0.1220, bbox_mAP_m: 0.2860, bbox_mAP_l: 0.1830, bbox_mAP_copypaste: 0.156 0.340 0.124 0.122 0.286 0.183
2020-08-14 00:14:22,822 - mmdet - INFO - Epoch(train) [14][525] loss_rpn_cls: 0.1268, loss_rpn_bbox: 0.1975, loss_cls: 0.3091, acc: 87.2452, loss_bbox: 0.1616, loss_mask: 0.3475, loss: 1.1425
2020-08-14 00:15:54,912 - mmdet - INFO - Epoch [15][50/1350]    lr: 0.00250, eta: 14:00:05, time: 1.839, data_time: 1.027, memory: 6384, loss_rpn_cls: 0.1300, loss_rpn_bbox: 0.2058, loss_cls: 0.2800, acc: 88.0840, loss_bbox: 0.1540, loss_mask: 0.3560, loss: 1.1259
2020-08-14 00:17:22,280 - mmdet - INFO - Epoch [15][100/1350]   lr: 0.00250, eta: 13:58:44, time: 1.747, data_time: 0.947, memory: 6384, loss_rpn_cls: 0.1263, loss_rpn_bbox: 0.2084, loss_cls: 0.2845, acc: 87.8945, loss_bbox: 0.1524, loss_mask: 0.3458, loss: 1.1173
2020-08-14 00:18:49,825 - mmdet - INFO - Epoch [15][150/1350]   lr: 0.00250, eta: 13:57:23, time: 1.751, data_time: 0.924, memory: 6384, loss_rpn_cls: 0.1391, loss_rpn_bbox: 0.2195, loss_cls: 0.2964, acc: 87.3652, loss_bbox: 0.1614, loss_mask: 0.3430, loss: 1.1593
2020-08-14 00:20:14,029 - mmdet - INFO - Epoch [15][200/1350]   lr: 0.00250, eta: 13:55:57, time: 1.684, data_time: 0.848, memory: 6384, loss_rpn_cls: 0.1214, loss_rpn_bbox: 0.2136, loss_cls: 0.2962, acc: 87.5215, loss_bbox: 0.1557, loss_mask: 0.3483, loss: 1.1352
2020-08-14 00:21:41,383 - mmdet - INFO - Epoch [15][250/1350]   lr: 0.00250, eta: 13:54:35, time: 1.747, data_time: 0.920, memory: 6384, loss_rpn_cls: 0.1243, loss_rpn_bbox: 0.2084, loss_cls: 0.2885, acc: 88.0605, loss_bbox: 0.1526, loss_mask: 0.3361, loss: 1.1099
2020-08-14 00:23:08,238 - mmdet - INFO - Epoch [15][300/1350]   lr: 0.00250, eta: 13:53:13, time: 1.737, data_time: 0.929, memory: 6384, loss_rpn_cls: 0.1169, loss_rpn_bbox: 0.2052, loss_cls: 0.2765, acc: 88.3359, loss_bbox: 0.1495, loss_mask: 0.3452, loss: 1.0932
2020-08-14 00:24:35,212 - mmdet - INFO - Epoch [15][350/1350]   lr: 0.00250, eta: 13:51:51, time: 1.739, data_time: 0.902, memory: 6384, loss_rpn_cls: 0.1371, loss_rpn_bbox: 0.2085, loss_cls: 0.2977, acc: 87.5020, loss_bbox: 0.1638, loss_mask: 0.3353, loss: 1.1425
2020-08-14 00:26:00,755 - mmdet - INFO - Epoch [15][400/1350]   lr: 0.00250, eta: 13:50:27, time: 1.711, data_time: 0.884, memory: 6384, loss_rpn_cls: 0.1360, loss_rpn_bbox: 0.2267, loss_cls: 0.3107, acc: 86.9668, loss_bbox: 0.1593, loss_mask: 0.3425, loss: 1.1752
2020-08-14 00:27:24,768 - mmdet - INFO - Epoch [15][450/1350]   lr: 0.00250, eta: 13:49:00, time: 1.680, data_time: 0.906, memory: 6384, loss_rpn_cls: 0.1280, loss_rpn_bbox: 0.2207, loss_cls: 0.2972, acc: 87.4961, loss_bbox: 0.1591, loss_mask: 0.3436, loss: 1.1485
2020-08-14 00:28:50,663 - mmdet - INFO - Epoch [15][500/1350]   lr: 0.00250, eta: 13:47:36, time: 1.718, data_time: 0.927, memory: 6384, loss_rpn_cls: 0.1453, loss_rpn_bbox: 0.2296, loss_cls: 0.3220, acc: 86.5176, loss_bbox: 0.1690, loss_mask: 0.3462, loss: 1.2121
2020-08-14 00:30:14,914 - mmdet - INFO - Epoch [15][550/1350]   lr: 0.00250, eta: 13:46:10, time: 1.685, data_time: 0.879, memory: 6384, loss_rpn_cls: 0.1295, loss_rpn_bbox: 0.2116, loss_cls: 0.2943, acc: 87.8320, loss_bbox: 0.1584, loss_mask: 0.3404, loss: 1.1342
2020-08-14 00:31:42,044 - mmdet - INFO - Epoch [15][600/1350]   lr: 0.00250, eta: 13:44:48, time: 1.743, data_time: 0.943, memory: 6384, loss_rpn_cls: 0.1447, loss_rpn_bbox: 0.2216, loss_cls: 0.2993, acc: 87.4453, loss_bbox: 0.1634, loss_mask: 0.3487, loss: 1.1776
2020-08-14 00:33:07,371 - mmdet - INFO - Epoch [15][650/1350]   lr: 0.00250, eta: 13:43:24, time: 1.707, data_time: 0.908, memory: 6384, loss_rpn_cls: 0.1327, loss_rpn_bbox: 0.2188, loss_cls: 0.3002, acc: 87.2988, loss_bbox: 0.1594, loss_mask: 0.3508, loss: 1.1619
2020-08-14 00:34:34,078 - mmdet - INFO - Epoch [15][700/1350]   lr: 0.00250, eta: 13:42:01, time: 1.734, data_time: 0.930, memory: 6384, loss_rpn_cls: 0.1359, loss_rpn_bbox: 0.2108, loss_cls: 0.3046, acc: 86.9805, loss_bbox: 0.1573, loss_mask: 0.3532, loss: 1.1619
2020-08-14 00:36:01,689 - mmdet - INFO - Epoch [15][750/1350]   lr: 0.00250, eta: 13:40:40, time: 1.752, data_time: 0.931, memory: 6384, loss_rpn_cls: 0.1375, loss_rpn_bbox: 0.2200, loss_cls: 0.2972, acc: 87.3906, loss_bbox: 0.1553, loss_mask: 0.3509, loss: 1.1609
2020-08-14 00:37:31,297 - mmdet - INFO - Epoch [15][800/1350]   lr: 0.00250, eta: 13:39:22, time: 1.792, data_time: 0.970, memory: 6384, loss_rpn_cls: 0.1355, loss_rpn_bbox: 0.2303, loss_cls: 0.3080, acc: 86.8008, loss_bbox: 0.1621, loss_mask: 0.3451, loss: 1.1811
2020-08-14 00:38:57,594 - mmdet - INFO - Epoch [15][850/1350]   lr: 0.00250, eta: 13:37:58, time: 1.726, data_time: 0.897, memory: 6384, loss_rpn_cls: 0.1518, loss_rpn_bbox: 0.2347, loss_cls: 0.3090, acc: 86.6250, loss_bbox: 0.1590, loss_mask: 0.3467, loss: 1.2012
2020-08-14 00:40:22,985 - mmdet - INFO - Epoch [15][900/1350]   lr: 0.00250, eta: 13:36:34, time: 1.708, data_time: 0.884, memory: 6384, loss_rpn_cls: 0.1100, loss_rpn_bbox: 0.2114, loss_cls: 0.2888, acc: 87.8066, loss_bbox: 0.1563, loss_mask: 0.3430, loss: 1.1095
2020-08-14 00:41:49,530 - mmdet - INFO - Epoch [15][950/1350]   lr: 0.00250, eta: 13:35:11, time: 1.731, data_time: 0.896, memory: 6384, loss_rpn_cls: 0.1334, loss_rpn_bbox: 0.2215, loss_cls: 0.3064, acc: 87.0137, loss_bbox: 0.1637, loss_mask: 0.3431, loss: 1.1681
2020-08-14 00:43:21,737 - mmdet - INFO - Epoch [15][1000/1350]  lr: 0.00250, eta: 13:33:56, time: 1.844, data_time: 1.001, memory: 6384, loss_rpn_cls: 0.1465, loss_rpn_bbox: 0.2340, loss_cls: 0.3000, acc: 87.3809, loss_bbox: 0.1555, loss_mask: 0.3478, loss: 1.1838
2020-08-14 00:44:45,814 - mmdet - INFO - Epoch [15][1050/1350]  lr: 0.00250, eta: 13:32:30, time: 1.682, data_time: 0.829, memory: 6384, loss_rpn_cls: 0.1407, loss_rpn_bbox: 0.2205, loss_cls: 0.2992, acc: 87.3809, loss_bbox: 0.1587, loss_mask: 0.3459, loss: 1.1649
2020-08-14 00:46:16,169 - mmdet - INFO - Epoch [15][1100/1350]  lr: 0.00250, eta: 13:31:12, time: 1.807, data_time: 0.955, memory: 6384, loss_rpn_cls: 0.1359, loss_rpn_bbox: 0.2271, loss_cls: 0.2977, acc: 87.6016, loss_bbox: 0.1565, loss_mask: 0.3524, loss: 1.1696
2020-08-14 00:47:37,865 - mmdet - INFO - Epoch [15][1150/1350]  lr: 0.00250, eta: 13:29:42, time: 1.634, data_time: 0.785, memory: 6384, loss_rpn_cls: 0.1345, loss_rpn_bbox: 0.2220, loss_cls: 0.3085, acc: 86.8828, loss_bbox: 0.1630, loss_mask: 0.3531, loss: 1.1811
2020-08-14 00:49:04,822 - mmdet - INFO - Epoch [15][1200/1350]  lr: 0.00250, eta: 13:28:20, time: 1.739, data_time: 0.894, memory: 6384, loss_rpn_cls: 0.1372, loss_rpn_bbox: 0.2302, loss_cls: 0.2993, acc: 87.2031, loss_bbox: 0.1521, loss_mask: 0.3418, loss: 1.1606
2020-08-14 00:50:31,243 - mmdet - INFO - Epoch [15][1250/1350]  lr: 0.00250, eta: 13:26:56, time: 1.728, data_time: 0.884, memory: 6384, loss_rpn_cls: 0.1096, loss_rpn_bbox: 0.2091, loss_cls: 0.2792, acc: 88.2910, loss_bbox: 0.1545, loss_mask: 0.3472, loss: 1.0997
2020-08-14 00:51:57,526 - mmdet - INFO - Epoch [15][1300/1350]  lr: 0.00250, eta: 13:25:33, time: 1.726, data_time: 0.882, memory: 6384, loss_rpn_cls: 0.1335, loss_rpn_bbox: 0.2324, loss_cls: 0.3111, acc: 86.7949, loss_bbox: 0.1632, loss_mask: 0.3632, loss: 1.2034
2020-08-14 00:53:20,879 - mmdet - INFO - Epoch [15][1350/1350]  lr: 0.00250, eta: 13:24:05, time: 1.667, data_time: 0.833, memory: 6384, loss_rpn_cls: 0.1196, loss_rpn_bbox: 0.2146, loss_cls: 0.2919, acc: 87.6602, loss_bbox: 0.1512, loss_mask: 0.3444, loss: 1.1217
2020-08-14 01:29:51,477 - mmdet - INFO - Evaluating bbox...
2020-08-14 01:38:43,986 - mmdet - INFO - Epoch [15][1350/1350]  lr: 0.00250, bbox_mAP: 0.1360, bbox_mAP_50: 0.3070, bbox_mAP_75: 0.0990, bbox_mAP_s: 0.1110, bbox_mAP_m: 0.2430, bbox_mAP_l: 0.1190, bbox_mAP_copypaste: 0.136 0.307 0.099 0.111 0.243 0.119
2020-08-14 01:54:19,092 - mmdet - INFO - Epoch(train) [15][525] loss_rpn_cls: 0.1481, loss_rpn_bbox: 0.2183, loss_cls: 0.3378, acc: 86.6815, loss_bbox: 0.1508, loss_mask: 0.3681, loss: 1.2232
2020-08-14 01:55:55,223 - mmdet - INFO - Epoch [16][50/1350]    lr: 0.00250, eta: 13:22:56, time: 1.920, data_time: 1.087, memory: 6384, loss_rpn_cls: 0.1552, loss_rpn_bbox: 0.2383, loss_cls: 0.3246, acc: 86.0781, loss_bbox: 0.1660, loss_mask: 0.3491, loss: 1.2332
2020-08-14 01:57:20,977 - mmdet - INFO - Epoch [16][100/1350]   lr: 0.00250, eta: 13:21:31, time: 1.715, data_time: 0.904, memory: 6384, loss_rpn_cls: 0.1351, loss_rpn_bbox: 0.2287, loss_cls: 0.3039, acc: 87.1934, loss_bbox: 0.1561, loss_mask: 0.3415, loss: 1.1653
2020-08-14 01:58:45,855 - mmdet - INFO - Epoch [16][150/1350]   lr: 0.00250, eta: 13:20:06, time: 1.698, data_time: 0.897, memory: 6384, loss_rpn_cls: 0.1425, loss_rpn_bbox: 0.2273, loss_cls: 0.3038, acc: 87.2871, loss_bbox: 0.1597, loss_mask: 0.3468, loss: 1.1801
2020-08-14 02:00:11,807 - mmdet - INFO - Epoch [16][200/1350]   lr: 0.00250, eta: 13:18:42, time: 1.719, data_time: 0.914, memory: 6384, loss_rpn_cls: 0.1353, loss_rpn_bbox: 0.2299, loss_cls: 0.3078, acc: 86.8887, loss_bbox: 0.1599, loss_mask: 0.3634, loss: 1.1962
2020-08-14 02:01:41,963 - mmdet - INFO - Epoch [16][250/1350]   lr: 0.00250, eta: 13:17:24, time: 1.803, data_time: 0.985, memory: 6384, loss_rpn_cls: 0.1354, loss_rpn_bbox: 0.2333, loss_cls: 0.3039, acc: 87.1484, loss_bbox: 0.1545, loss_mask: 0.3434, loss: 1.1705
2020-08-14 02:03:07,176 - mmdet - INFO - Epoch [16][300/1350]   lr: 0.00250, eta: 13:15:59, time: 1.704, data_time: 0.872, memory: 6384, loss_rpn_cls: 0.1302, loss_rpn_bbox: 0.2205, loss_cls: 0.2948, acc: 87.4102, loss_bbox: 0.1544, loss_mask: 0.3377, loss: 1.1376
2020-08-14 02:04:33,942 - mmdet - INFO - Epoch [16][350/1350]   lr: 0.00250, eta: 13:14:36, time: 1.735, data_time: 0.904, memory: 6384, loss_rpn_cls: 0.1229, loss_rpn_bbox: 0.2161, loss_cls: 0.2869, acc: 87.9258, loss_bbox: 0.1521, loss_mask: 0.3550, loss: 1.1330
2020-08-14 02:05:59,703 - mmdet - INFO - Epoch [16][400/1350]   lr: 0.00250, eta: 13:13:12, time: 1.715, data_time: 0.882, memory: 6384, loss_rpn_cls: 0.1327, loss_rpn_bbox: 0.2305, loss_cls: 0.2967, acc: 87.4355, loss_bbox: 0.1560, loss_mask: 0.3566, loss: 1.1724
2020-08-14 02:07:26,179 - mmdet - INFO - Epoch [16][450/1350]   lr: 0.00250, eta: 13:11:48, time: 1.730, data_time: 0.897, memory: 6384, loss_rpn_cls: 0.1248, loss_rpn_bbox: 0.2170, loss_cls: 0.2963, acc: 87.4922, loss_bbox: 0.1557, loss_mask: 0.3351, loss: 1.1288
2020-08-14 02:08:50,889 - mmdet - INFO - Epoch [16][500/1350]   lr: 0.00250, eta: 13:10:22, time: 1.694, data_time: 0.857, memory: 6384, loss_rpn_cls: 0.1264, loss_rpn_bbox: 0.2154, loss_cls: 0.2942, acc: 87.5879, loss_bbox: 0.1545, loss_mask: 0.3461, loss: 1.1365
2020-08-14 02:10:16,030 - mmdet - INFO - Epoch [16][550/1350]   lr: 0.00250, eta: 13:08:57, time: 1.703, data_time: 0.885, memory: 6384, loss_rpn_cls: 0.1337, loss_rpn_bbox: 0.2316, loss_cls: 0.3121, acc: 86.8379, loss_bbox: 0.1602, loss_mask: 0.3465, loss: 1.1841
2020-08-14 02:11:43,752 - mmdet - INFO - Epoch [16][600/1350]   lr: 0.00250, eta: 13:07:36, time: 1.754, data_time: 0.921, memory: 6384, loss_rpn_cls: 0.1123, loss_rpn_bbox: 0.2116, loss_cls: 0.2854, acc: 87.9844, loss_bbox: 0.1518, loss_mask: 0.3452, loss: 1.1062
2020-08-14 02:13:10,726 - mmdet - INFO - Epoch [16][650/1350]   lr: 0.00250, eta: 13:06:13, time: 1.739, data_time: 0.904, memory: 6384, loss_rpn_cls: 0.1329, loss_rpn_bbox: 0.2217, loss_cls: 0.3048, acc: 87.0449, loss_bbox: 0.1574, loss_mask: 0.3506, loss: 1.1674
2020-08-14 02:14:38,489 - mmdet - INFO - Epoch [16][700/1350]   lr: 0.00250, eta: 13:04:51, time: 1.755, data_time: 0.912, memory: 6384, loss_rpn_cls: 0.1496, loss_rpn_bbox: 0.2441, loss_cls: 0.3238, acc: 86.2852, loss_bbox: 0.1621, loss_mask: 0.3529, loss: 1.2325
2020-08-14 02:16:08,926 - mmdet - INFO - Epoch [16][750/1350]   lr: 0.00250, eta: 13:03:33, time: 1.809, data_time: 0.863, memory: 6384, loss_rpn_cls: 0.1170, loss_rpn_bbox: 0.2226, loss_cls: 0.3101, acc: 86.7559, loss_bbox: 0.1564, loss_mask: 0.3400, loss: 1.1461
2020-08-14 02:17:35,699 - mmdet - INFO - Epoch [16][800/1350]   lr: 0.00250, eta: 13:02:10, time: 1.735, data_time: 0.922, memory: 6384, loss_rpn_cls: 0.1482, loss_rpn_bbox: 0.2306, loss_cls: 0.3093, acc: 86.8398, loss_bbox: 0.1588, loss_mask: 0.3473, loss: 1.1942
2020-08-14 02:19:04,735 - mmdet - INFO - Epoch [16][850/1350]   lr: 0.00250, eta: 13:00:50, time: 1.781, data_time: 0.969, memory: 6384, loss_rpn_cls: 0.1349, loss_rpn_bbox: 0.2144, loss_cls: 0.3064, acc: 86.9629, loss_bbox: 0.1558, loss_mask: 0.3475, loss: 1.1591
2020-08-14 02:20:29,684 - mmdet - INFO - Epoch [16][900/1350]   lr: 0.00250, eta: 12:59:24, time: 1.699, data_time: 0.857, memory: 6384, loss_rpn_cls: 0.1446, loss_rpn_bbox: 0.2331, loss_cls: 0.3121, acc: 86.5391, loss_bbox: 0.1593, loss_mask: 0.3409, loss: 1.1901
2020-08-14 02:21:54,437 - mmdet - INFO - Epoch [16][950/1350]   lr: 0.00250, eta: 12:57:59, time: 1.695, data_time: 0.859, memory: 6384, loss_rpn_cls: 0.1355, loss_rpn_bbox: 0.2224, loss_cls: 0.3024, acc: 87.1953, loss_bbox: 0.1581, loss_mask: 0.3345, loss: 1.1529
2020-08-14 02:23:22,051 - mmdet - INFO - Epoch [16][1000/1350]  lr: 0.00250, eta: 12:56:36, time: 1.752, data_time: 0.918, memory: 6384, loss_rpn_cls: 0.1498, loss_rpn_bbox: 0.2302, loss_cls: 0.3033, acc: 87.2305, loss_bbox: 0.1595, loss_mask: 0.3439, loss: 1.1866
2020-08-14 02:24:49,314 - mmdet - INFO - Epoch [16][1050/1350]  lr: 0.00250, eta: 12:55:14, time: 1.745, data_time: 0.938, memory: 6384, loss_rpn_cls: 0.1129, loss_rpn_bbox: 0.2142, loss_cls: 0.2909, acc: 87.6172, loss_bbox: 0.1498, loss_mask: 0.3473, loss: 1.1152
2020-08-14 02:26:13,684 - mmdet - INFO - Epoch [16][1100/1350]  lr: 0.00250, eta: 12:53:48, time: 1.687, data_time: 0.857, memory: 6384, loss_rpn_cls: 0.1335, loss_rpn_bbox: 0.2190, loss_cls: 0.2988, acc: 87.4492, loss_bbox: 0.1556, loss_mask: 0.3376, loss: 1.1445
2020-08-14 02:27:42,332 - mmdet - INFO - Epoch [16][1150/1350]  lr: 0.00250, eta: 12:52:27, time: 1.773, data_time: 0.945, memory: 6384, loss_rpn_cls: 0.1368, loss_rpn_bbox: 0.2181, loss_cls: 0.2911, acc: 87.8027, loss_bbox: 0.1541, loss_mask: 0.3455, loss: 1.1456
2020-08-14 02:29:06,932 - mmdet - INFO - Epoch [16][1200/1350]  lr: 0.00250, eta: 12:51:01, time: 1.692, data_time: 0.857, memory: 6384, loss_rpn_cls: 0.1322, loss_rpn_bbox: 0.2143, loss_cls: 0.2949, acc: 87.5898, loss_bbox: 0.1568, loss_mask: 0.3279, loss: 1.1260
2020-08-14 02:30:37,086 - mmdet - INFO - Epoch [16][1250/1350]  lr: 0.00250, eta: 12:49:42, time: 1.803, data_time: 0.970, memory: 6384, loss_rpn_cls: 0.1328, loss_rpn_bbox: 0.2178, loss_cls: 0.2876, acc: 87.9238, loss_bbox: 0.1516, loss_mask: 0.3296, loss: 1.1194
2020-08-14 02:32:01,909 - mmdet - INFO - Epoch [16][1300/1350]  lr: 0.00250, eta: 12:48:16, time: 1.696, data_time: 0.866, memory: 6384, loss_rpn_cls: 0.1256, loss_rpn_bbox: 0.2282, loss_cls: 0.2880, acc: 87.7109, loss_bbox: 0.1522, loss_mask: 0.3515, loss: 1.1455
2020-08-14 02:33:26,954 - mmdet - INFO - Epoch [16][1350/1350]  lr: 0.00250, eta: 12:46:51, time: 1.701, data_time: 0.878, memory: 6384, loss_rpn_cls: 0.1421, loss_rpn_bbox: 0.2290, loss_cls: 0.3050, acc: 86.8555, loss_bbox: 0.1589, loss_mask: 0.3481, loss: 1.1832
2020-08-14 03:17:12,066 - mmdet - INFO - Evaluating bbox...
2020-08-14 03:26:51,728 - mmdet - INFO - Epoch [16][1350/1350]  lr: 0.00250, bbox_mAP: 0.1480, bbox_mAP_50: 0.3210, bbox_mAP_75: 0.1160, bbox_mAP_s: 0.1110, bbox_mAP_m: 0.3140, bbox_mAP_l: 0.1450, bbox_mAP_copypaste: 0.148 0.321 0.116 0.111 0.314 0.145
2020-08-14 03:42:32,232 - mmdet - INFO - Epoch(train) [16][525] loss_rpn_cls: 0.1301, loss_rpn_bbox: 0.2049, loss_cls: 0.3303, acc: 86.2197, loss_bbox: 0.1582, loss_mask: 0.3450, loss: 1.1684
2020-08-14 03:44:07,382 - mmdet - INFO - Epoch [17][50/1350]    lr: 0.00250, eta: 12:45:38, time: 1.901, data_time: 1.082, memory: 6384, loss_rpn_cls: 0.1259, loss_rpn_bbox: 0.2071, loss_cls: 0.2777, acc: 88.2598, loss_bbox: 0.1503, loss_mask: 0.3588, loss: 1.1197
2020-08-14 03:45:31,005 - mmdet - INFO - Epoch [17][100/1350]   lr: 0.00250, eta: 12:44:11, time: 1.672, data_time: 0.887, memory: 6384, loss_rpn_cls: 0.1306, loss_rpn_bbox: 0.2239, loss_cls: 0.2966, acc: 87.3496, loss_bbox: 0.1547, loss_mask: 0.3456, loss: 1.1514
2020-08-14 03:47:00,717 - mmdet - INFO - Epoch [17][150/1350]   lr: 0.00250, eta: 12:42:51, time: 1.794, data_time: 0.979, memory: 6384, loss_rpn_cls: 0.1267, loss_rpn_bbox: 0.2163, loss_cls: 0.2893, acc: 87.7285, loss_bbox: 0.1533, loss_mask: 0.3412, loss: 1.1269
2020-08-14 03:48:26,634 - mmdet - INFO - Epoch [17][200/1350]   lr: 0.00250, eta: 12:41:27, time: 1.718, data_time: 0.881, memory: 6384, loss_rpn_cls: 0.1397, loss_rpn_bbox: 0.2237, loss_cls: 0.3028, acc: 87.1211, loss_bbox: 0.1579, loss_mask: 0.3457, loss: 1.1699
2020-08-14 03:49:53,299 - mmdet - INFO - Epoch [17][250/1350]   lr: 0.00250, eta: 12:40:03, time: 1.733, data_time: 0.909, memory: 6384, loss_rpn_cls: 0.1328, loss_rpn_bbox: 0.2232, loss_cls: 0.3109, acc: 86.8750, loss_bbox: 0.1641, loss_mask: 0.3534, loss: 1.1844
2020-08-14 03:51:19,386 - mmdet - INFO - Epoch [17][300/1350]   lr: 0.00250, eta: 12:38:39, time: 1.722, data_time: 0.894, memory: 6384, loss_rpn_cls: 0.1570, loss_rpn_bbox: 0.2384, loss_cls: 0.3229, acc: 86.3301, loss_bbox: 0.1636, loss_mask: 0.3546, loss: 1.2364
2020-08-14 03:52:47,651 - mmdet - INFO - Epoch [17][350/1350]   lr: 0.00250, eta: 12:37:17, time: 1.765, data_time: 0.929, memory: 6384, loss_rpn_cls: 0.1510, loss_rpn_bbox: 0.2385, loss_cls: 0.3255, acc: 86.1035, loss_bbox: 0.1654, loss_mask: 0.3451, loss: 1.2255
2020-08-14 03:54:14,337 - mmdet - INFO - Epoch [17][400/1350]   lr: 0.00250, eta: 12:35:54, time: 1.734, data_time: 0.898, memory: 6384, loss_rpn_cls: 0.1632, loss_rpn_bbox: 0.2442, loss_cls: 0.3199, acc: 86.4453, loss_bbox: 0.1611, loss_mask: 0.3449, loss: 1.2333
2020-08-14 03:55:45,742 - mmdet - INFO - Epoch [17][450/1350]   lr: 0.00250, eta: 12:34:36, time: 1.828, data_time: 0.989, memory: 6384, loss_rpn_cls: 0.1625, loss_rpn_bbox: 0.2366, loss_cls: 0.3176, acc: 86.5273, loss_bbox: 0.1612, loss_mask: 0.3501, loss: 1.2281
2020-08-14 03:57:09,629 - mmdet - INFO - Epoch [17][500/1350]   lr: 0.00250, eta: 12:33:09, time: 1.678, data_time: 0.845, memory: 6384, loss_rpn_cls: 0.1298, loss_rpn_bbox: 0.2303, loss_cls: 0.3009, acc: 87.3047, loss_bbox: 0.1557, loss_mask: 0.3537, loss: 1.1703
2020-08-14 03:58:38,797 - mmdet - INFO - Epoch [17][550/1350]   lr: 0.00250, eta: 12:31:48, time: 1.783, data_time: 0.952, memory: 6384, loss_rpn_cls: 0.1418, loss_rpn_bbox: 0.2403, loss_cls: 0.3116, acc: 86.6250, loss_bbox: 0.1607, loss_mask: 0.3516, loss: 1.2060
2020-08-14 04:00:03,739 - mmdet - INFO - Epoch [17][600/1350]   lr: 0.00250, eta: 12:30:23, time: 1.699, data_time: 0.861, memory: 6384, loss_rpn_cls: 0.1478, loss_rpn_bbox: 0.2353, loss_cls: 0.3213, acc: 85.9980, loss_bbox: 0.1616, loss_mask: 0.3536, loss: 1.2197
2020-08-14 04:01:35,676 - mmdet - INFO - Epoch [17][650/1350]   lr: 0.00250, eta: 12:29:05, time: 1.839, data_time: 1.001, memory: 6384, loss_rpn_cls: 0.1388, loss_rpn_bbox: 0.2482, loss_cls: 0.3103, acc: 86.6777, loss_bbox: 0.1555, loss_mask: 0.3596, loss: 1.2125
2020-08-14 04:03:00,404 - mmdet - INFO - Epoch [17][700/1350]   lr: 0.00250, eta: 12:27:39, time: 1.695, data_time: 0.871, memory: 6384, loss_rpn_cls: 0.1500, loss_rpn_bbox: 0.2573, loss_cls: 0.3199, acc: 86.3418, loss_bbox: 0.1566, loss_mask: 0.3656, loss: 1.2494
2020-08-14 04:04:29,812 - mmdet - INFO - Epoch [17][750/1350]   lr: 0.00250, eta: 12:26:19, time: 1.788, data_time: 0.950, memory: 6384, loss_rpn_cls: 0.1639, loss_rpn_bbox: 0.2631, loss_cls: 0.3267, acc: 86.1328, loss_bbox: 0.1599, loss_mask: 0.3490, loss: 1.2625
2020-08-14 04:05:57,484 - mmdet - INFO - Epoch [17][800/1350]   lr: 0.00250, eta: 12:24:57, time: 1.753, data_time: 0.912, memory: 6384, loss_rpn_cls: 0.1422, loss_rpn_bbox: 0.2470, loss_cls: 0.3054, acc: 87.0977, loss_bbox: 0.1531, loss_mask: 0.3443, loss: 1.1920
2020-08-14 04:07:27,185 - mmdet - INFO - Epoch [17][850/1350]   lr: 0.00250, eta: 12:23:36, time: 1.794, data_time: 0.950, memory: 6384, loss_rpn_cls: 0.1189, loss_rpn_bbox: 0.2344, loss_cls: 0.2977, acc: 87.6348, loss_bbox: 0.1451, loss_mask: 0.3445, loss: 1.1405
2020-08-14 04:08:51,084 - mmdet - INFO - Epoch [17][900/1350]   lr: 0.00250, eta: 12:22:09, time: 1.678, data_time: 0.836, memory: 6384, loss_rpn_cls: 0.1429, loss_rpn_bbox: 0.2482, loss_cls: 0.2960, acc: 87.5195, loss_bbox: 0.1477, loss_mask: 0.3440, loss: 1.1788
2020-08-14 04:10:19,984 - mmdet - INFO - Epoch [17][950/1350]   lr: 0.00250, eta: 12:20:48, time: 1.778, data_time: 0.955, memory: 6384, loss_rpn_cls: 0.1666, loss_rpn_bbox: 0.2556, loss_cls: 0.3338, acc: 85.6621, loss_bbox: 0.1574, loss_mask: 0.3594, loss: 1.2728
2020-08-14 04:11:44,407 - mmdet - INFO - Epoch [17][1000/1350]  lr: 0.00250, eta: 12:19:22, time: 1.688, data_time: 0.876, memory: 6384, loss_rpn_cls: 0.1407, loss_rpn_bbox: 0.2486, loss_cls: 0.3055, acc: 87.1113, loss_bbox: 0.1519, loss_mask: 0.3500, loss: 1.1968
2020-08-14 04:13:09,762 - mmdet - INFO - Epoch [17][1050/1350]  lr: 0.00250, eta: 12:17:57, time: 1.707, data_time: 0.890, memory: 6384, loss_rpn_cls: 0.1482, loss_rpn_bbox: 0.2712, loss_cls: 0.3067, acc: 87.2812, loss_bbox: 0.1534, loss_mask: 0.3589, loss: 1.2385
2020-08-14 04:14:35,636 - mmdet - INFO - Epoch [17][1100/1350]  lr: 0.00250, eta: 12:16:32, time: 1.717, data_time: 0.887, memory: 6384, loss_rpn_cls: 0.1680, loss_rpn_bbox: 0.2653, loss_cls: 0.3199, acc: 86.7051, loss_bbox: 0.1583, loss_mask: 0.3498, loss: 1.2612
2020-08-14 04:16:06,931 - mmdet - INFO - Epoch [17][1150/1350]  lr: 0.00250, eta: 12:15:13, time: 1.826, data_time: 0.996, memory: 6384, loss_rpn_cls: 0.1556, loss_rpn_bbox: 0.2523, loss_cls: 0.3005, acc: 87.5840, loss_bbox: 0.1490, loss_mask: 0.3331, loss: 1.1905
2020-08-14 04:17:30,714 - mmdet - INFO - Epoch [17][1200/1350]  lr: 0.00250, eta: 12:13:46, time: 1.676, data_time: 0.836, memory: 6384, loss_rpn_cls: 0.1299, loss_rpn_bbox: 0.2501, loss_cls: 0.3062, acc: 86.9609, loss_bbox: 0.1485, loss_mask: 0.3559, loss: 1.1907
2020-08-14 04:18:59,371 - mmdet - INFO - Epoch [17][1250/1350]  lr: 0.00250, eta: 12:12:25, time: 1.773, data_time: 0.928, memory: 6384, loss_rpn_cls: 0.1454, loss_rpn_bbox: 0.2466, loss_cls: 0.3036, acc: 87.0234, loss_bbox: 0.1504, loss_mask: 0.3428, loss: 1.1889
2020-08-14 04:20:27,954 - mmdet - INFO - Epoch [17][1300/1350]  lr: 0.00250, eta: 12:11:03, time: 1.772, data_time: 0.915, memory: 6384, loss_rpn_cls: 0.1544, loss_rpn_bbox: 0.2339, loss_cls: 0.3053, acc: 87.1719, loss_bbox: 0.1486, loss_mask: 0.3343, loss: 1.1765
2020-08-14 04:21:52,242 - mmdet - INFO - Epoch [17][1350/1350]  lr: 0.00250, eta: 12:09:37, time: 1.686, data_time: 0.852, memory: 6384, loss_rpn_cls: 0.1234, loss_rpn_bbox: 0.2449, loss_cls: 0.3161, acc: 86.5488, loss_bbox: 0.1537, loss_mask: 0.3530, loss: 1.1911
2020-08-14 05:11:08,689 - mmdet - INFO - Evaluating bbox...
2020-08-14 05:20:54,545 - mmdet - INFO - Epoch [17][1350/1350]  lr: 0.00250, bbox_mAP: 0.1290, bbox_mAP_50: 0.3000, bbox_mAP_75: 0.0890, bbox_mAP_s: 0.1150, bbox_mAP_m: 0.2360, bbox_mAP_l: 0.1120, bbox_mAP_copypaste: 0.129 0.300 0.089 0.115 0.236 0.112
2020-08-14 05:36:30,466 - mmdet - INFO - Epoch(train) [17][525] loss_rpn_cls: 0.1534, loss_rpn_bbox: 0.2284, loss_cls: 0.3639, acc: 84.6895, loss_bbox: 0.1640, loss_mask: 0.3502, loss: 1.2599
2020-08-14 05:38:02,526 - mmdet - INFO - Epoch [18][50/1350]    lr: 0.00250, eta: 12:08:19, time: 1.839, data_time: 1.024, memory: 6384, loss_rpn_cls: 0.1567, loss_rpn_bbox: 0.2523, loss_cls: 0.3143, acc: 86.8457, loss_bbox: 0.1502, loss_mask: 0.3375, loss: 1.2109
2020-08-14 05:39:30,046 - mmdet - INFO - Epoch [18][100/1350]   lr: 0.00250, eta: 12:06:56, time: 1.750, data_time: 0.979, memory: 6384, loss_rpn_cls: 0.1370, loss_rpn_bbox: 0.2582, loss_cls: 0.3048, acc: 87.0078, loss_bbox: 0.1514, loss_mask: 0.3465, loss: 1.1980
2020-08-14 05:40:55,571 - mmdet - INFO - Epoch [18][150/1350]   lr: 0.00250, eta: 12:05:31, time: 1.710, data_time: 0.918, memory: 6384, loss_rpn_cls: 0.1260, loss_rpn_bbox: 0.2493, loss_cls: 0.3071, acc: 87.0020, loss_bbox: 0.1536, loss_mask: 0.3556, loss: 1.1917
2020-08-14 05:42:19,474 - mmdet - INFO - Epoch [18][200/1350]   lr: 0.00250, eta: 12:04:04, time: 1.678, data_time: 0.907, memory: 6384, loss_rpn_cls: 0.1462, loss_rpn_bbox: 0.2593, loss_cls: 0.3086, acc: 87.2520, loss_bbox: 0.1495, loss_mask: 0.3342, loss: 1.1978
2020-08-14 05:43:48,412 - mmdet - INFO - Epoch [18][250/1350]   lr: 0.00250, eta: 12:02:42, time: 1.779, data_time: 0.982, memory: 6384, loss_rpn_cls: 0.1526, loss_rpn_bbox: 0.2555, loss_cls: 0.3030, acc: 87.3164, loss_bbox: 0.1497, loss_mask: 0.3374, loss: 1.1983
2020-08-14 05:45:14,367 - mmdet - INFO - Epoch [18][300/1350]   lr: 0.00250, eta: 12:01:17, time: 1.719, data_time: 0.887, memory: 6384, loss_rpn_cls: 0.1402, loss_rpn_bbox: 0.2475, loss_cls: 0.3111, acc: 86.8594, loss_bbox: 0.1543, loss_mask: 0.3536, loss: 1.2066
2020-08-14 05:46:41,798 - mmdet - INFO - Epoch [18][350/1350]   lr: 0.00250, eta: 11:59:54, time: 1.748, data_time: 0.963, memory: 6384, loss_rpn_cls: 0.1351, loss_rpn_bbox: 0.2552, loss_cls: 0.2998, acc: 87.3906, loss_bbox: 0.1506, loss_mask: 0.3536, loss: 1.1944
2020-08-14 05:48:08,876 - mmdet - INFO - Epoch [18][400/1350]   lr: 0.00250, eta: 11:58:31, time: 1.742, data_time: 0.944, memory: 6384, loss_rpn_cls: 0.1506, loss_rpn_bbox: 0.2491, loss_cls: 0.3092, acc: 86.9180, loss_bbox: 0.1540, loss_mask: 0.3492, loss: 1.2121
2020-08-14 05:49:36,043 - mmdet - INFO - Epoch [18][450/1350]   lr: 0.00250, eta: 11:57:07, time: 1.743, data_time: 0.933, memory: 6384, loss_rpn_cls: 0.1319, loss_rpn_bbox: 0.2301, loss_cls: 0.2962, acc: 87.5273, loss_bbox: 0.1480, loss_mask: 0.3427, loss: 1.1489
2020-08-14 05:51:01,920 - mmdet - INFO - Epoch [18][500/1350]   lr: 0.00250, eta: 11:55:43, time: 1.718, data_time: 0.910, memory: 6384, loss_rpn_cls: 0.1217, loss_rpn_bbox: 0.2388, loss_cls: 0.3125, acc: 86.6094, loss_bbox: 0.1537, loss_mask: 0.3588, loss: 1.1855
2020-08-14 05:52:26,917 - mmdet - INFO - Epoch [18][550/1350]   lr: 0.00250, eta: 11:54:17, time: 1.700, data_time: 0.906, memory: 6384, loss_rpn_cls: 0.1255, loss_rpn_bbox: 0.2345, loss_cls: 0.3058, acc: 87.1172, loss_bbox: 0.1545, loss_mask: 0.3388, loss: 1.1592
2020-08-14 05:53:55,442 - mmdet - INFO - Epoch [18][600/1350]   lr: 0.00250, eta: 11:52:55, time: 1.770, data_time: 0.963, memory: 6384, loss_rpn_cls: 0.1251, loss_rpn_bbox: 0.2264, loss_cls: 0.2884, acc: 87.7969, loss_bbox: 0.1460, loss_mask: 0.3406, loss: 1.1265
2020-08-14 05:55:25,409 - mmdet - INFO - Epoch [18][650/1350]   lr: 0.00250, eta: 11:51:34, time: 1.799, data_time: 0.997, memory: 6384, loss_rpn_cls: 0.1324, loss_rpn_bbox: 0.2230, loss_cls: 0.2919, acc: 87.6738, loss_bbox: 0.1449, loss_mask: 0.3367, loss: 1.1289
2020-08-14 05:56:50,768 - mmdet - INFO - Epoch [18][700/1350]   lr: 0.00250, eta: 11:50:09, time: 1.707, data_time: 0.867, memory: 6384, loss_rpn_cls: 0.1475, loss_rpn_bbox: 0.2505, loss_cls: 0.3096, acc: 86.9570, loss_bbox: 0.1553, loss_mask: 0.3525, loss: 1.2155
2020-08-14 05:58:18,621 - mmdet - INFO - Epoch [18][750/1350]   lr: 0.00250, eta: 11:48:46, time: 1.757, data_time: 0.917, memory: 6384, loss_rpn_cls: 0.1288, loss_rpn_bbox: 0.2326, loss_cls: 0.2891, acc: 88.0605, loss_bbox: 0.1498, loss_mask: 0.3433, loss: 1.1436
2020-08-14 05:59:41,920 - mmdet - INFO - Epoch [18][800/1350]   lr: 0.00250, eta: 11:47:18, time: 1.666, data_time: 0.867, memory: 6384, loss_rpn_cls: 0.1382, loss_rpn_bbox: 0.2514, loss_cls: 0.3093, acc: 86.6660, loss_bbox: 0.1520, loss_mask: 0.3447, loss: 1.1957
2020-08-14 06:01:09,989 - mmdet - INFO - Epoch [18][850/1350]   lr: 0.00250, eta: 11:45:56, time: 1.761, data_time: 0.940, memory: 6384, loss_rpn_cls: 0.1333, loss_rpn_bbox: 0.2365, loss_cls: 0.2864, acc: 88.0332, loss_bbox: 0.1488, loss_mask: 0.3386, loss: 1.1435
2020-08-14 06:02:39,947 - mmdet - INFO - Epoch [18][900/1350]   lr: 0.00250, eta: 11:44:35, time: 1.799, data_time: 0.975, memory: 6384, loss_rpn_cls: 0.1397, loss_rpn_bbox: 0.2426, loss_cls: 0.2878, acc: 87.8711, loss_bbox: 0.1509, loss_mask: 0.3300, loss: 1.1511
2020-08-14 06:04:07,357 - mmdet - INFO - Epoch [18][950/1350]   lr: 0.00250, eta: 11:43:12, time: 1.748, data_time: 0.926, memory: 6384, loss_rpn_cls: 0.1371, loss_rpn_bbox: 0.2471, loss_cls: 0.2966, acc: 87.3730, loss_bbox: 0.1444, loss_mask: 0.3472, loss: 1.1724
2020-08-14 06:05:33,913 - mmdet - INFO - Epoch [18][1000/1350]  lr: 0.00250, eta: 11:41:48, time: 1.731, data_time: 0.903, memory: 6384, loss_rpn_cls: 0.1262, loss_rpn_bbox: 0.2302, loss_cls: 0.2966, acc: 87.4883, loss_bbox: 0.1493, loss_mask: 0.3397, loss: 1.1421
2020-08-14 06:07:03,076 - mmdet - INFO - Epoch [18][1050/1350]  lr: 0.00250, eta: 11:40:26, time: 1.783, data_time: 0.953, memory: 6384, loss_rpn_cls: 0.1392, loss_rpn_bbox: 0.2370, loss_cls: 0.2959, acc: 87.7090, loss_bbox: 0.1470, loss_mask: 0.3331, loss: 1.1522
2020-08-14 06:08:26,981 - mmdet - INFO - Epoch [18][1100/1350]  lr: 0.00250, eta: 11:38:59, time: 1.678, data_time: 0.849, memory: 6384, loss_rpn_cls: 0.1311, loss_rpn_bbox: 0.2432, loss_cls: 0.2994, acc: 87.2129, loss_bbox: 0.1460, loss_mask: 0.3377, loss: 1.1575
2020-08-14 06:09:55,174 - mmdet - INFO - Epoch [18][1150/1350]  lr: 0.00250, eta: 11:37:36, time: 1.764, data_time: 0.936, memory: 6384, loss_rpn_cls: 0.1485, loss_rpn_bbox: 0.2395, loss_cls: 0.2839, acc: 88.1973, loss_bbox: 0.1475, loss_mask: 0.3319, loss: 1.1514
2020-08-14 06:11:23,475 - mmdet - INFO - Epoch [18][1200/1350]  lr: 0.00250, eta: 11:36:14, time: 1.766, data_time: 0.923, memory: 6384, loss_rpn_cls: 0.1265, loss_rpn_bbox: 0.2434, loss_cls: 0.2850, acc: 87.9062, loss_bbox: 0.1442, loss_mask: 0.3438, loss: 1.1431
2020-08-14 06:12:50,104 - mmdet - INFO - Epoch [18][1250/1350]  lr: 0.00250, eta: 11:34:50, time: 1.733, data_time: 0.912, memory: 6384, loss_rpn_cls: 0.1209, loss_rpn_bbox: 0.2334, loss_cls: 0.2854, acc: 87.8848, loss_bbox: 0.1430, loss_mask: 0.3540, loss: 1.1366
2020-08-14 06:14:15,075 - mmdet - INFO - Epoch [18][1300/1350]  lr: 0.00250, eta: 11:33:24, time: 1.699, data_time: 0.863, memory: 6384, loss_rpn_cls: 0.1197, loss_rpn_bbox: 0.2292, loss_cls: 0.2900, acc: 87.6816, loss_bbox: 0.1408, loss_mask: 0.3435, loss: 1.1233
2020-08-14 06:15:41,915 - mmdet - INFO - Epoch [18][1350/1350]  lr: 0.00250, eta: 11:32:00, time: 1.737, data_time: 0.921, memory: 6384, loss_rpn_cls: 0.1295, loss_rpn_bbox: 0.2231, loss_cls: 0.2961, acc: 87.4141, loss_bbox: 0.1511, loss_mask: 0.3402, loss: 1.1401
2020-08-14 07:05:22,018 - mmdet - INFO - Evaluating bbox...
2020-08-14 07:15:04,232 - mmdet - INFO - Epoch [18][1350/1350]  lr: 0.00250, bbox_mAP: 0.1650, bbox_mAP_50: 0.3450, bbox_mAP_75: 0.1360, bbox_mAP_s: 0.1340, bbox_mAP_m: 0.2940, bbox_mAP_l: 0.1170, bbox_mAP_copypaste: 0.165 0.345 0.136 0.134 0.294 0.117
2020-08-14 07:30:41,771 - mmdet - INFO - Epoch(train) [18][525] loss_rpn_cls: 0.1371, loss_rpn_bbox: 0.2160, loss_cls: 0.3033, acc: 87.5355, loss_bbox: 0.1472, loss_mask: 0.3367, loss: 1.1403
2020-08-14 07:32:15,230 - mmdet - INFO - Epoch [19][50/1350]    lr: 0.00250, eta: 11:30:42, time: 1.867, data_time: 1.036, memory: 6384, loss_rpn_cls: 0.1173, loss_rpn_bbox: 0.2293, loss_cls: 0.2908, acc: 87.7402, loss_bbox: 0.1450, loss_mask: 0.3417, loss: 1.1242
2020-08-14 07:33:39,318 - mmdet - INFO - Epoch [19][100/1350]   lr: 0.00250, eta: 11:29:15, time: 1.682, data_time: 0.879, memory: 6384, loss_rpn_cls: 0.1166, loss_rpn_bbox: 0.2186, loss_cls: 0.2923, acc: 87.6328, loss_bbox: 0.1449, loss_mask: 0.3405, loss: 1.1129
2020-08-14 07:35:09,589 - mmdet - INFO - Epoch [19][150/1350]   lr: 0.00250, eta: 11:27:55, time: 1.805, data_time: 1.009, memory: 6384, loss_rpn_cls: 0.1343, loss_rpn_bbox: 0.2441, loss_cls: 0.2989, acc: 87.2441, loss_bbox: 0.1486, loss_mask: 0.3406, loss: 1.1665
2020-08-14 07:36:34,265 - mmdet - INFO - Epoch [19][200/1350]   lr: 0.00250, eta: 11:26:29, time: 1.694, data_time: 0.867, memory: 6384, loss_rpn_cls: 0.1376, loss_rpn_bbox: 0.2331, loss_cls: 0.2994, acc: 87.3965, loss_bbox: 0.1510, loss_mask: 0.3362, loss: 1.1574
2020-08-14 07:37:58,881 - mmdet - INFO - Epoch [19][250/1350]   lr: 0.00250, eta: 11:25:02, time: 1.692, data_time: 0.883, memory: 6384, loss_rpn_cls: 0.1244, loss_rpn_bbox: 0.2477, loss_cls: 0.2894, acc: 87.9297, loss_bbox: 0.1464, loss_mask: 0.3523, loss: 1.1603
2020-08-14 07:39:26,150 - mmdet - INFO - Epoch [19][300/1350]   lr: 0.00250, eta: 11:23:39, time: 1.745, data_time: 0.937, memory: 6384, loss_rpn_cls: 0.1378, loss_rpn_bbox: 0.2404, loss_cls: 0.3026, acc: 86.9902, loss_bbox: 0.1497, loss_mask: 0.3480, loss: 1.1785
2020-08-14 07:40:51,748 - mmdet - INFO - Epoch [19][350/1350]   lr: 0.00250, eta: 11:22:13, time: 1.712, data_time: 0.905, memory: 6384, loss_rpn_cls: 0.1227, loss_rpn_bbox: 0.2474, loss_cls: 0.2952, acc: 87.4121, loss_bbox: 0.1483, loss_mask: 0.3547, loss: 1.1683
2020-08-14 07:42:20,964 - mmdet - INFO - Epoch [19][400/1350]   lr: 0.00250, eta: 11:20:51, time: 1.784, data_time: 0.976, memory: 6384, loss_rpn_cls: 0.1332, loss_rpn_bbox: 0.2445, loss_cls: 0.2998, acc: 87.4219, loss_bbox: 0.1527, loss_mask: 0.3521, loss: 1.1823
2020-08-14 07:43:48,655 - mmdet - INFO - Epoch [19][450/1350]   lr: 0.00250, eta: 11:19:28, time: 1.754, data_time: 0.939, memory: 6384, loss_rpn_cls: 0.1656, loss_rpn_bbox: 0.2497, loss_cls: 0.3025, acc: 87.3027, loss_bbox: 0.1494, loss_mask: 0.3360, loss: 1.2033
2020-08-14 07:45:15,201 - mmdet - INFO - Epoch [19][500/1350]   lr: 0.00250, eta: 11:18:04, time: 1.731, data_time: 0.902, memory: 6384, loss_rpn_cls: 0.1400, loss_rpn_bbox: 0.2575, loss_cls: 0.2946, acc: 87.6641, loss_bbox: 0.1448, loss_mask: 0.3413, loss: 1.1782
2020-08-14 07:46:44,058 - mmdet - INFO - Epoch [19][550/1350]   lr: 0.00250, eta: 11:16:41, time: 1.777, data_time: 0.942, memory: 6384, loss_rpn_cls: 0.1480, loss_rpn_bbox: 0.2490, loss_cls: 0.2970, acc: 87.5273, loss_bbox: 0.1481, loss_mask: 0.3424, loss: 1.1845
2020-08-14 07:48:07,808 - mmdet - INFO - Epoch [19][600/1350]   lr: 0.00250, eta: 11:15:14, time: 1.675, data_time: 0.840, memory: 6384, loss_rpn_cls: 0.1477, loss_rpn_bbox: 0.2540, loss_cls: 0.3131, acc: 86.7285, loss_bbox: 0.1544, loss_mask: 0.3442, loss: 1.2135
2020-08-14 07:49:36,640 - mmdet - INFO - Epoch [19][650/1350]   lr: 0.00250, eta: 11:13:52, time: 1.777, data_time: 0.944, memory: 6384, loss_rpn_cls: 0.1509, loss_rpn_bbox: 0.2549, loss_cls: 0.2944, acc: 87.6816, loss_bbox: 0.1393, loss_mask: 0.3466, loss: 1.1862
2020-08-14 07:51:02,056 - mmdet - INFO - Epoch [19][700/1350]   lr: 0.00250, eta: 11:12:26, time: 1.708, data_time: 0.880, memory: 6384, loss_rpn_cls: 0.2109, loss_rpn_bbox: 0.2963, loss_cls: 0.3819, acc: 84.8672, loss_bbox: 0.1568, loss_mask: 0.3653, loss: 1.4111
2020-08-14 07:52:30,943 - mmdet - INFO - Epoch [19][750/1350]   lr: 0.00250, eta: 11:11:04, time: 1.778, data_time: 0.945, memory: 6384, loss_rpn_cls: 0.1730, loss_rpn_bbox: 0.2747, loss_cls: 0.3137, acc: 86.5645, loss_bbox: 0.1537, loss_mask: 0.3432, loss: 1.2583
2020-08-14 07:53:56,525 - mmdet - INFO - Epoch [19][800/1350]   lr: 0.00250, eta: 11:09:39, time: 1.712, data_time: 0.869, memory: 6384, loss_rpn_cls: 0.1578, loss_rpn_bbox: 0.2596, loss_cls: 0.3037, acc: 87.2812, loss_bbox: 0.1415, loss_mask: 0.3234, loss: 1.1861
2020-08-14 07:55:24,579 - mmdet - INFO - Epoch [19][850/1350]   lr: 0.00250, eta: 11:08:16, time: 1.761, data_time: 0.931, memory: 6384, loss_rpn_cls: 0.1297, loss_rpn_bbox: 0.2503, loss_cls: 0.2967, acc: 87.6270, loss_bbox: 0.1464, loss_mask: 0.3478, loss: 1.1710
2020-08-14 07:56:48,547 - mmdet - INFO - Epoch [19][900/1350]   lr: 0.00250, eta: 11:06:49, time: 1.679, data_time: 0.863, memory: 6384, loss_rpn_cls: 0.1387, loss_rpn_bbox: 0.2546, loss_cls: 0.3090, acc: 86.7207, loss_bbox: 0.1491, loss_mask: 0.3339, loss: 1.1853
2020-08-14 07:58:15,405 - mmdet - INFO - Epoch [19][950/1350]   lr: 0.00250, eta: 11:05:25, time: 1.737, data_time: 0.938, memory: 6384, loss_rpn_cls: 0.1620, loss_rpn_bbox: 0.2592, loss_cls: 0.3130, acc: 86.8047, loss_bbox: 0.1538, loss_mask: 0.3423, loss: 1.2303
2020-08-14 07:59:42,249 - mmdet - INFO - Epoch [19][1000/1350]  lr: 0.00250, eta: 11:04:00, time: 1.737, data_time: 0.911, memory: 6384, loss_rpn_cls: 0.1397, loss_rpn_bbox: 0.2477, loss_cls: 0.3072, acc: 87.1035, loss_bbox: 0.1446, loss_mask: 0.3381, loss: 1.1774
2020-08-14 08:01:08,808 - mmdet - INFO - Epoch [19][1050/1350]  lr: 0.00250, eta: 11:02:36, time: 1.731, data_time: 0.922, memory: 6384, loss_rpn_cls: 0.1417, loss_rpn_bbox: 0.2562, loss_cls: 0.2963, acc: 87.3477, loss_bbox: 0.1478, loss_mask: 0.3428, loss: 1.1849
2020-08-14 08:02:36,989 - mmdet - INFO - Epoch [19][1100/1350]  lr: 0.00250, eta: 11:01:13, time: 1.764, data_time: 0.967, memory: 6384, loss_rpn_cls: 0.1389, loss_rpn_bbox: 0.2511, loss_cls: 0.2939, acc: 87.5566, loss_bbox: 0.1438, loss_mask: 0.3365, loss: 1.1641
2020-08-14 08:04:04,021 - mmdet - INFO - Epoch [19][1150/1350]  lr: 0.00250, eta: 10:59:49, time: 1.741, data_time: 0.926, memory: 6384, loss_rpn_cls: 0.1578, loss_rpn_bbox: 0.2495, loss_cls: 0.2970, acc: 87.4551, loss_bbox: 0.1431, loss_mask: 0.3410, loss: 1.1885
2020-08-14 08:05:31,286 - mmdet - INFO - Epoch [19][1200/1350]  lr: 0.00250, eta: 10:58:25, time: 1.745, data_time: 0.932, memory: 6384, loss_rpn_cls: 0.1519, loss_rpn_bbox: 0.2656, loss_cls: 0.3110, acc: 86.7812, loss_bbox: 0.1472, loss_mask: 0.3321, loss: 1.2078
2020-08-14 08:06:56,577 - mmdet - INFO - Epoch [19][1250/1350]  lr: 0.00250, eta: 10:56:59, time: 1.706, data_time: 0.925, memory: 6384, loss_rpn_cls: 0.1251, loss_rpn_bbox: 0.2519, loss_cls: 0.2993, acc: 87.3047, loss_bbox: 0.1454, loss_mask: 0.3438, loss: 1.1655
2020-08-14 08:08:25,291 - mmdet - INFO - Epoch [19][1300/1350]  lr: 0.00250, eta: 10:55:36, time: 1.774, data_time: 0.869, memory: 6384, loss_rpn_cls: 0.1217, loss_rpn_bbox: 0.2439, loss_cls: 0.2765, acc: 88.4395, loss_bbox: 0.1376, loss_mask: 0.3469, loss: 1.1266
2020-08-14 08:09:50,954 - mmdet - INFO - Epoch [19][1350/1350]  lr: 0.00250, eta: 10:54:11, time: 1.713, data_time: 0.930, memory: 6384, loss_rpn_cls: 0.1316, loss_rpn_bbox: 0.2385, loss_cls: 0.2916, acc: 87.5859, loss_bbox: 0.1459, loss_mask: 0.3356, loss: 1.1431
2020-08-14 08:59:32,867 - mmdet - INFO - Evaluating bbox...
2020-08-14 09:09:18,882 - mmdet - INFO - Epoch [19][1350/1350]  lr: 0.00250, bbox_mAP: 0.1470, bbox_mAP_50: 0.3270, bbox_mAP_75: 0.1120, bbox_mAP_s: 0.1350, bbox_mAP_m: 0.2110, bbox_mAP_l: 0.0740, bbox_mAP_copypaste: 0.147 0.327 0.112 0.135 0.211 0.074
2020-08-14 09:25:02,172 - mmdet - INFO - Epoch(train) [19][525] loss_rpn_cls: 0.1459, loss_rpn_bbox: 0.2350, loss_cls: 0.3281, acc: 86.4818, loss_bbox: 0.1530, loss_mask: 0.3418, loss: 1.2039
2020-08-14 09:26:35,667 - mmdet - INFO - Epoch [20][50/1350]    lr: 0.00250, eta: 10:52:53, time: 1.867, data_time: 1.040, memory: 6384, loss_rpn_cls: 0.1481, loss_rpn_bbox: 0.2617, loss_cls: 0.2929, acc: 87.5098, loss_bbox: 0.1407, loss_mask: 0.3458, loss: 1.1892
2020-08-14 09:28:01,419 - mmdet - INFO - Epoch [20][100/1350]   lr: 0.00250, eta: 10:51:27, time: 1.715, data_time: 0.910, memory: 6384, loss_rpn_cls: 0.1431, loss_rpn_bbox: 0.2508, loss_cls: 0.2977, acc: 87.3223, loss_bbox: 0.1462, loss_mask: 0.3520, loss: 1.1898
2020-08-14 09:29:27,482 - mmdet - INFO - Epoch [20][150/1350]   lr: 0.00250, eta: 10:50:02, time: 1.721, data_time: 0.926, memory: 6384, loss_rpn_cls: 0.1324, loss_rpn_bbox: 0.2553, loss_cls: 0.2938, acc: 87.6914, loss_bbox: 0.1419, loss_mask: 0.3533, loss: 1.1767
2020-08-14 09:30:53,982 - mmdet - INFO - Epoch [20][200/1350]   lr: 0.00250, eta: 10:48:37, time: 1.730, data_time: 0.929, memory: 6384, loss_rpn_cls: 0.1863, loss_rpn_bbox: 0.2810, loss_cls: 0.3403, acc: 86.2676, loss_bbox: 0.1492, loss_mask: 0.3537, loss: 1.3105
2020-08-14 09:32:19,513 - mmdet - INFO - Epoch [20][250/1350]   lr: 0.00250, eta: 10:47:12, time: 1.711, data_time: 0.914, memory: 6384, loss_rpn_cls: 0.1557, loss_rpn_bbox: 0.2807, loss_cls: 0.3055, acc: 87.0645, loss_bbox: 0.1456, loss_mask: 0.3391, loss: 1.2266
2020-08-14 09:33:47,028 - mmdet - INFO - Epoch [20][300/1350]   lr: 0.00250, eta: 10:45:48, time: 1.750, data_time: 0.973, memory: 6384, loss_rpn_cls: 0.1536, loss_rpn_bbox: 0.2691, loss_cls: 0.2906, acc: 87.6016, loss_bbox: 0.1439, loss_mask: 0.3405, loss: 1.1977
2020-08-14 09:35:13,548 - mmdet - INFO - Epoch [20][350/1350]   lr: 0.00250, eta: 10:44:23, time: 1.730, data_time: 0.958, memory: 6384, loss_rpn_cls: 0.1568, loss_rpn_bbox: 0.2704, loss_cls: 0.3095, acc: 86.8711, loss_bbox: 0.1474, loss_mask: 0.3447, loss: 1.2288
2020-08-14 09:36:40,416 - mmdet - INFO - Epoch [20][400/1350]   lr: 0.00250, eta: 10:42:59, time: 1.737, data_time: 0.948, memory: 6384, loss_rpn_cls: 0.1431, loss_rpn_bbox: 0.2579, loss_cls: 0.2954, acc: 87.4258, loss_bbox: 0.1436, loss_mask: 0.3458, loss: 1.1858
2020-08-14 09:38:09,790 - mmdet - INFO - Epoch [20][450/1350]   lr: 0.00250, eta: 10:41:37, time: 1.787, data_time: 0.999, memory: 6384, loss_rpn_cls: 0.1504, loss_rpn_bbox: 0.2652, loss_cls: 0.3297, acc: 85.7715, loss_bbox: 0.1448, loss_mask: 0.3379, loss: 1.2281
2020-08-14 09:39:35,085 - mmdet - INFO - Epoch [20][500/1350]   lr: 0.00250, eta: 10:40:11, time: 1.706, data_time: 0.905, memory: 6384, loss_rpn_cls: 0.1406, loss_rpn_bbox: 0.2716, loss_cls: 0.3062, acc: 86.8496, loss_bbox: 0.1435, loss_mask: 0.3441, loss: 1.2060
2020-08-14 09:41:02,325 - mmdet - INFO - Epoch [20][550/1350]   lr: 0.00250, eta: 10:38:47, time: 1.745, data_time: 0.948, memory: 6384, loss_rpn_cls: 0.1493, loss_rpn_bbox: 0.2743, loss_cls: 0.3019, acc: 87.2617, loss_bbox: 0.1440, loss_mask: 0.3409, loss: 1.2105
2020-08-14 09:42:33,280 - mmdet - INFO - Epoch [20][600/1350]   lr: 0.00250, eta: 10:37:26, time: 1.819, data_time: 0.991, memory: 6384, loss_rpn_cls: 0.1606, loss_rpn_bbox: 0.2694, loss_cls: 0.3068, acc: 86.9707, loss_bbox: 0.1373, loss_mask: 0.3596, loss: 1.2336
2020-08-14 09:44:01,213 - mmdet - INFO - Epoch [20][650/1350]   lr: 0.00250, eta: 10:36:02, time: 1.759, data_time: 0.921, memory: 6384, loss_rpn_cls: 0.1538, loss_rpn_bbox: 0.2706, loss_cls: 0.3042, acc: 87.0547, loss_bbox: 0.1422, loss_mask: 0.3538, loss: 1.2246

[ZwwWayne]

I have no idea about the divergence.

[ecm200]

@ZwwWayne a further update to this issue.

When I use the same training dataset, and seed for image selection, but use a ResNeXt101 based backbone, as opposed to the previously mentioned ResNet architectures, I appear to get more stable training.

The above image shows various models used with ResNet50 and 100 architectures. All but 1 of the ResNet models used the standard learning rate of 0.02 scaled by 1/8 factor (0.0025) due to me using only a single GPU (following the Linear Scaling rule by Goyal et al (2018)). The orange model (ResNet50) that eventually fails with NAN loss at ~42k minibatches and has a loss convergence the same as the ResNeXt101 model (grey) that doens't diverge, both had a learing rate of 0.01 (0.00125).

[ZwwWayne]

This case some times indicates that your data might not be very clean, thus the model simply blows up at some cases. There are several things you could try: 1. Add gradient clip to restrict the gradient to be smaller than a number, e.g. 35. In this case, you could observe the gradient norm of each iteration and you could get some hints when loss goes Nan. 2. Use smaller learning rate, some times these hyper parameters need to be tuned. 3. Check the data by visualization. You need to make sure that there is no isues (e.g., out of range box, zero area box) in the annotation and data.

[ecm200]

@ZwwWayne thank very much for the suggestions, these are very helpful.

I am glad to say that my investigations and thoughts are along the lines you have suggested (bar the clipped gradient idea). I have suspected that this is either a data issue or a gradient stability issue (which of course could be manifest of the data issue, these are not mutually exclusive).

With regards to the data, I have made an effort to ensure that no label data has points outside the image frame, this is something I experienced as an earlier issue with more simple geometries of the input image objects (spheroids as opposed to cuboids). However, I did not explicitly consider the possibility of zero area bounding boxes / polygons.

I will report back with my findings, so I will leave this issue open for the moment.

Thanks again.

[ecm200]

@ZwwWayne an update for you.

Firstly, with regards to the input data quality, I made an additional check on the training data regarding labelling, paying particular close attention to the smallest bounding box and masks. Whilst I had previously checked to ensure that no bounding boxes were 0.0, I had failed to ensure that the resulting objects were not smaller than a single pixel, which resulted in approximately 1600 object annotations (out of 1.3 million) that had non zero bounding box areas but 0 pixel area masks. These were quite well spread amongst the training set, with a maximum of 10 of these in any one image (with there being between 350 and 500 objects per image). So I have removed these labels from the training set.

Running a ResNeXt101 Mask R-CNN model, this time using 2X GPUs (deep blue), has resulted in better performance of stable training over 72 epochs, which was also improved over the 1X GPU (deep green) training run of 36 epochs. (Note the default learning rate of 0.01@8XGPU was linearly scaled with the relative mini-batch size of 16 (2/16 or 4/16 factor for the 1X and 2X GPU respectively). This is a significant improvement over the behaviour encountered when training with the dataset with these 0 pixel area masks present (red, orange and grey).

Combined Loss

BBOX mAP

[ZwwWayne]

Hi @ecm200 , Thanks for your feedback and congrats! We are planning to reorganize our documentation with more detailed tutorials for users to debug and begin their own projects. Your experience and report here are very valuable to this.

Since the issue seems to be resolved, this issue will be closed. The valuable experience will be included in the documentation in the future. Feel free to reopen this issue if you have any further questions.

[ecm200]

Hi @ecm200 , Thanks for your feedback and congrats! We are planning to reorganize our documentation with more detailed tutorials for users to debug and begin their own projects. Your experience and report here are very valuable to this.

Since the issue seems to be resolved, this issue will be closed. The valuable experience will be included in the documentation in the future. Feel free to reopen this issue if you have any further questions.

@ZwwWayne you are most welcome, and it is an absolute pleasure to provide feedback on this fantastic project. I should be thanking you and the rest of development team!

Please let me know if there's anything more I can do to assist.