search

open-mmlab / mmdetection

OpenMMLab Detection Toolbox and Benchmark
https://mmdetection.readthedocs.io
Apache License 2.0
29.56k stars 9.46k forks source link

Training speed of mmdetection v1 is much slower than detectron2 #2672

Closed serend1p1ty closed 4 years ago

serend1p1ty commented 4 years ago

Describe the bug

The training speed of Faster-R-CNN-Caffe-c4 on pascal VOC 2007 is too slow.

Mmdetection log:

2020-05-08 18:32:34,938 - mmdet - INFO - Start running, host: aistudio@jupyter-282778-426164, work_dir: /home/aistudio/work/mmdet-ps/work_dirs/faster_rcnn_r50_caffe_c4_1x_voc07
2020-05-08 18:32:34,938 - mmdet - INFO - workflow: [('train', 1)], max: 4 epochs
2020-05-08 18:33:04,627 - mmdet - INFO - Epoch [1][50/7517] lr: 0.00125, eta: 4:57:01, time: 0.594, data_time: 0.033, memory: 6836, loss_rpn_cls: 0.3086, loss_rpn_bbox: 0.0766, loss_cls: 0.5470, acc: 91.7441, loss_bbox: 0.1292, loss: 1.0614
2020-05-08 18:33:34,077 - mmdet - INFO - Epoch [1][100/7517]    lr: 0.00125, eta: 4:55:21, time: 0.589, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.1468, loss_rpn_bbox: 0.1463, loss_cls: 0.2872, acc: 94.6328, loss_bbox: 0.1335, loss: 0.7137
2020-05-08 18:34:03,320 - mmdet - INFO - Epoch [1][150/7517]    lr: 0.00125, eta: 4:53:47, time: 0.585, data_time: 0.030, memory: 6836, loss_rpn_cls: 0.1537, loss_rpn_bbox: 0.1198, loss_cls: 0.2554, acc: 94.9531, loss_bbox: 0.1237, loss: 0.6526

*speed = 4:57:01 / (7517 4) = 0.01 min / iter**

Detectron2 log (18000 iteration):

[05/08 23:59:49] d2.engine.train_loop INFO: Starting training from iteration 0
[05/08 23:59:59] d2.utils.events INFO:  eta: 2:26:45  iter: 19  total_loss: 2.304  loss_cls: 1.515  loss_box_reg: 0.130  loss_rpn_cls: 0.688  loss_rpn_loc: 0.127  time: 0.4915  data_time: 0.0343  lr: 0.000477  max_mem: 4529M

speed = 2:26:45 / 18000 = 0.008 min / iter

That means that if you run 4 epochs on pascal VOC 2007(RepeatDataset, times=3, that is 3w iteration), mmdetection takes an hour more than detectron2.

Reproduction

Mmdetection config:

# model settings
norm_cfg = dict(type='BN', requires_grad=False)
model = dict(
    type='FasterRCNN',
    pretrained='open-mmlab://resnet50_caffe',
    backbone=dict(
        type='ResNet',
        depth=50,
        num_stages=3,
        strides=(1, 2, 2),
        dilations=(1, 1, 1),
        out_indices=(2, ),
        frozen_stages=1,
        norm_cfg=norm_cfg,
        norm_eval=True,
        style='caffe'),
    shared_head=dict(
        type='ResLayer',
        depth=50,
        stage=3,
        stride=2,
        dilation=1,
        style='caffe',
        norm_cfg=norm_cfg,
        norm_eval=True),
    rpn_head=dict(
        type='RPNHead',
        in_channels=1024,
        feat_channels=1024,
        anchor_scales=[2, 4, 8, 16, 32],
        anchor_ratios=[0.5, 1.0, 2.0],
        anchor_strides=[16],
        target_means=[.0, .0, .0, .0],
        target_stds=[1.0, 1.0, 1.0, 1.0],
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0)),
    bbox_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=14, sample_num=2),
        out_channels=1024,
        featmap_strides=[16]),
    bbox_head=dict(
        type='BBoxHead',
        with_avg_pool=True,
        roi_feat_size=7,
        in_channels=2048,
        num_classes=21,
        target_means=[0., 0., 0., 0.],
        target_stds=[0.1, 0.1, 0.2, 0.2],
        reg_class_agnostic=False,
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)))
# model training and testing settings
train_cfg = dict(
    rpn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.7,
            neg_iou_thr=0.3,
            min_pos_iou=0.3,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=256,
            pos_fraction=0.5,
            neg_pos_ub=-1,
            add_gt_as_proposals=False),
        allowed_border=0,
        pos_weight=-1,
        debug=False),
    rpn_proposal=dict(
        nms_across_levels=False,
        nms_pre=12000,
        nms_post=2000,
        max_num=2000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.5,
            neg_iou_thr=0.5,
            min_pos_iou=0.5,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=512,
            pos_fraction=0.25,
            neg_pos_ub=-1,
            add_gt_as_proposals=True),
        pos_weight=-1,
        debug=False))
test_cfg = dict(
    rpn=dict(
        nms_across_levels=False,
        nms_pre=6000,
        nms_post=1000,
        max_num=1000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        score_thr=0.05, nms=dict(type='nms', iou_thr=0.5), max_per_img=100))
# dataset settings
dataset_type = 'VOCDataset'
data_root = 'data/VOCdevkit/'
img_norm_cfg = dict(
    mean=[102.9801, 115.9465, 122.7717], std=[1.0, 1.0, 1.0], to_rgb=False)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='Resize', img_scale=(1000, 600), keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1000, 600),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
data = dict(
    imgs_per_gpu=2,
    workers_per_gpu=0,
    train=dict(
        type='RepeatDataset',
        times=3,
        dataset=dict(
            type=dataset_type,
            ann_file=data_root + 'VOC2007/ImageSets/Main/trainval.txt',
            img_prefix=data_root + 'VOC2007/',
            pipeline=train_pipeline)),
    val=dict(
        type=dataset_type,
        ann_file=data_root + 'VOC2007/ImageSets/Main/test.txt',
        img_prefix=data_root + 'VOC2007/',
        pipeline=test_pipeline),
    test=dict(
        type=dataset_type,
        ann_file=data_root + 'VOC2007/ImageSets/Main/test.txt',
        img_prefix=data_root + 'VOC2007/',
        pipeline=test_pipeline))
evaluation = dict(interval=1, metric='mAP')
# optimizer
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
# learning policy
lr_config = dict(policy='step', step=[3])  # actual epoch = 3 * 3 = 9
checkpoint_config = dict(interval=1)
# yapf:disable
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        # dict(type='TensorboardLoggerHook')
    ])
# yapf:enable
# runtime settings
total_epochs = 4  # actual epoch = 4 * 3 = 12
dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = './work_dirs/faster_rcnn_r50_caffe_c4_1x_voc07'
load_from = None
resume_from = None
workflow = [('train', 1)]

Detectron2 official config:

_BASE_: "../Base-RCNN-C4.yaml"
MODEL:
  WEIGHTS: "detectron2://ImageNetPretrained/MSRA/R-50.pkl"
  MASK_ON: False
  RESNETS:
    DEPTH: 50
  ROI_HEADS:
    NUM_CLASSES: 20
INPUT:
  MIN_SIZE_TRAIN: (480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800)
  MIN_SIZE_TEST: 800
DATASETS:
  TRAIN: ('voc_2007_trainval', 'voc_2012_trainval')
  TEST: ('voc_2007_test',)
SOLVER:
  STEPS: (12000, 16000)
  MAX_ITER: 18000  # 17.4 epochs
  WARMUP_ITERS: 100
  1. What command or script did you run?

mmdetection:

python tools/train.py faster_rcnn_r50_caffe_c4_1x_voc07.py --gpus 1 --autoscale-lr

detectron2:

python tools/train_net.py \
    --config-file configs/PascalVOC-Detection/faster_rcnn_R_50_C4.yaml \
    --num-gpus 1 SOLVER.IMS_PER_BATCH 2 SOLVER.BASE_LR 0.0025
  1. Did you make any modifications on the code or config? Did you understand what you have modified?

No modifications

  1. What dataset did you use?

Pascal VOC 2007

  1. Please run python mmdet/utils/collect_env.py to collect necessary environment infomation and paste it here. 
    
------------------------------------------------------------
sys.platform: linux
Python: 3.7.4 (default, Aug 13 2019, 20:35:49) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 9.2, V9.2.148
GPU 0: Tesla V100-SXM2-32GB
GCC: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
PyTorch: 1.4.0+cu92
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CUDA Runtime 9.2
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_37,code=compute_37
- CuDNN 7.6.3
- Magma 2.5.1
- Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.5.0+cu92 OpenCV: 4.2.0 MMCV: 0.4.4 MMDetection: 1.1.0+d672cf7 MMDetection Compiler: GCC 5.4 MMDetection CUDA Compiler: 9.2

``

hellock commented 4 years ago

The estimating time of the first several iters is inaccurate since it includes the initial loading time.

serend1p1ty commented 4 years ago

@hellock Hi, thanks for your response. But it actually spends almost 5 hours:

2020-05-08 18:32:34,938 - mmdet - INFO - workflow: [('train', 1)], max: 4 epochs
2020-05-08 18:33:04,627 - mmdet - INFO - Epoch [1][50/7517] lr: 0.00125, eta: 4:57:01, time: 0.594, data_time: 0.033, memory: 6836, loss_rpn_cls: 0.3086, loss_rpn_bbox: 0.0766, loss_cls: 0.5470, acc: 91.7441, loss_bbox: 0.1292, loss: 1.0614
2020-05-08 18:33:34,077 - mmdet - INFO - Epoch [1][100/7517]    lr: 0.00125, eta: 4:55:21, time: 0.589, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.1468, loss_rpn_bbox: 0.1463, loss_cls: 0.2872, acc: 94.6328, loss_bbox: 0.1335, loss: 0.7137
2020-05-08 18:34:03,320 - mmdet - INFO - Epoch [1][150/7517]    lr: 0.00125, eta: 4:53:47, time: 0.585, data_time: 0.030, memory: 6836, loss_rpn_cls: 0.1537, loss_rpn_bbox: 0.1198, loss_cls: 0.2554, acc: 94.9531, loss_bbox: 0.1237, loss: 0.6526
2020-05-08 18:34:32,633 - mmdet - INFO - Epoch [1][200/7517]    lr: 0.00125, eta: 4:52:55, time: 0.586, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1289, loss_rpn_bbox: 0.0683, loss_cls: 0.2415, acc: 94.9922, loss_bbox: 0.1241, loss: 0.5628
2020-05-08 18:35:02,085 - mmdet - INFO - Epoch [1][250/7517]    lr: 0.00125, eta: 4:52:29, time: 0.589, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1439, loss_rpn_bbox: 0.0613, loss_cls: 0.2529, acc: 94.4062, loss_bbox: 0.1308, loss: 0.5889
2020-05-08 18:35:31,811 - mmdet - INFO - Epoch [1][300/7517]    lr: 0.00125, eta: 4:52:29, time: 0.595, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.1290, loss_rpn_bbox: 0.0780, loss_cls: 0.2648, acc: 94.3487, loss_bbox: 0.1392, loss: 0.6111
2020-05-08 18:36:01,235 - mmdet - INFO - Epoch [1][350/7517]    lr: 0.00125, eta: 4:51:55, time: 0.588, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1301, loss_rpn_bbox: 0.0739, loss_cls: 0.2452, acc: 94.1113, loss_bbox: 0.1299, loss: 0.5791
2020-05-08 18:36:30,588 - mmdet - INFO - Epoch [1][400/7517]    lr: 0.00125, eta: 4:51:17, time: 0.587, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1174, loss_rpn_bbox: 0.0661, loss_cls: 0.2548, acc: 94.3906, loss_bbox: 0.1362, loss: 0.5744
2020-05-08 18:36:59,943 - mmdet - INFO - Epoch [1][450/7517]    lr: 0.00125, eta: 4:50:41, time: 0.587, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1254, loss_rpn_bbox: 0.1008, loss_cls: 0.2280, acc: 94.5273, loss_bbox: 0.1307, loss: 0.5850
2020-05-08 18:37:29,554 - mmdet - INFO - Epoch [1][500/7517]    lr: 0.00125, eta: 4:50:21, time: 0.592, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.1385, loss_rpn_bbox: 0.0708, loss_cls: 0.2370, acc: 93.8926, loss_bbox: 0.1409, loss: 0.5873
2020-05-08 18:37:59,120 - mmdet - INFO - Epoch [1][550/7517]    lr: 0.00125, eta: 4:49:57, time: 0.591, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1109, loss_rpn_bbox: 0.0648, loss_cls: 0.2329, acc: 94.0371, loss_bbox: 0.1420, loss: 0.5506
2020-05-08 18:38:28,641 - mmdet - INFO - Epoch [1][600/7517]    lr: 0.00125, eta: 4:49:30, time: 0.590, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1036, loss_rpn_bbox: 0.0646, loss_cls: 0.2208, acc: 94.3379, loss_bbox: 0.1347, loss: 0.5236
2020-05-08 18:38:58,091 - mmdet - INFO - Epoch [1][650/7517]    lr: 0.00125, eta: 4:49:00, time: 0.589, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1225, loss_rpn_bbox: 0.0601, loss_cls: 0.2496, acc: 93.3784, loss_bbox: 0.1515, loss: 0.5837
2020-05-08 18:39:27,558 - mmdet - INFO - Epoch [1][700/7517]    lr: 0.00125, eta: 4:48:30, time: 0.589, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0939, loss_rpn_bbox: 0.0558, loss_cls: 0.1830, acc: 95.0195, loss_bbox: 0.1176, loss: 0.4503
2020-05-08 18:39:57,179 - mmdet - INFO - Epoch [1][750/7517]    lr: 0.00125, eta: 4:48:06, time: 0.592, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.1137, loss_rpn_bbox: 0.0634, loss_cls: 0.2146, acc: 94.0254, loss_bbox: 0.1425, loss: 0.5342
2020-05-08 18:40:26,743 - mmdet - INFO - Epoch [1][800/7517]    lr: 0.00125, eta: 4:47:40, time: 0.591, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1161, loss_rpn_bbox: 0.0592, loss_cls: 0.1985, acc: 94.2051, loss_bbox: 0.1309, loss: 0.5047
2020-05-08 18:40:56,273 - mmdet - INFO - Epoch [1][850/7517]    lr: 0.00125, eta: 4:47:12, time: 0.591, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1338, loss_rpn_bbox: 0.0609, loss_cls: 0.2544, acc: 92.3555, loss_bbox: 0.1733, loss: 0.6223
2020-05-08 18:41:25,922 - mmdet - INFO - Epoch [1][900/7517]    lr: 0.00125, eta: 4:46:48, time: 0.593, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1110, loss_rpn_bbox: 0.0541, loss_cls: 0.2304, acc: 93.3945, loss_bbox: 0.1416, loss: 0.5371
2020-05-08 18:41:55,510 - mmdet - INFO - Epoch [1][950/7517]    lr: 0.00125, eta: 4:46:21, time: 0.592, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1003, loss_rpn_bbox: 0.0659, loss_cls: 0.1871, acc: 94.5059, loss_bbox: 0.1290, loss: 0.4824
2020-05-08 18:42:25,100 - mmdet - INFO - Epoch [1][1000/7517]   lr: 0.00125, eta: 4:45:54, time: 0.592, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0904, loss_rpn_bbox: 0.0529, loss_cls: 0.1743, acc: 94.7188, loss_bbox: 0.1221, loss: 0.4396
2020-05-08 18:42:54,750 - mmdet - INFO - Epoch [1][1050/7517]   lr: 0.00125, eta: 4:45:28, time: 0.593, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0963, loss_rpn_bbox: 0.0641, loss_cls: 0.2276, acc: 93.2070, loss_bbox: 0.1531, loss: 0.5411
2020-05-08 18:43:24,253 - mmdet - INFO - Epoch [1][1100/7517]   lr: 0.00125, eta: 4:44:58, time: 0.590, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1108, loss_rpn_bbox: 0.0506, loss_cls: 0.1976, acc: 94.0332, loss_bbox: 0.1384, loss: 0.4974
2020-05-08 18:43:53,852 - mmdet - INFO - Epoch [1][1150/7517]   lr: 0.00125, eta: 4:44:31, time: 0.592, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0980, loss_rpn_bbox: 0.0506, loss_cls: 0.1732, acc: 94.4434, loss_bbox: 0.1340, loss: 0.4558
2020-05-08 18:44:23,494 - mmdet - INFO - Epoch [1][1200/7517]   lr: 0.00125, eta: 4:44:04, time: 0.593, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0927, loss_rpn_bbox: 0.0444, loss_cls: 0.1861, acc: 94.0625, loss_bbox: 0.1375, loss: 0.4606
2020-05-08 18:44:53,197 - mmdet - INFO - Epoch [1][1250/7517]   lr: 0.00125, eta: 4:43:39, time: 0.594, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1082, loss_rpn_bbox: 0.0504, loss_cls: 0.2261, acc: 93.0137, loss_bbox: 0.1602, loss: 0.5449
2020-05-08 18:45:22,794 - mmdet - INFO - Epoch [1][1300/7517]   lr: 0.00125, eta: 4:43:11, time: 0.592, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0974, loss_rpn_bbox: 0.0577, loss_cls: 0.1936, acc: 93.9199, loss_bbox: 0.1372, loss: 0.4859
2020-05-08 18:45:52,463 - mmdet - INFO - Epoch [1][1350/7517]   lr: 0.00125, eta: 4:42:44, time: 0.593, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0930, loss_rpn_bbox: 0.0446, loss_cls: 0.1968, acc: 93.9453, loss_bbox: 0.1500, loss: 0.4844
2020-05-08 18:46:22,177 - mmdet - INFO - Epoch [1][1400/7517]   lr: 0.00125, eta: 4:42:18, time: 0.594, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0934, loss_rpn_bbox: 0.0514, loss_cls: 0.1839, acc: 93.8477, loss_bbox: 0.1384, loss: 0.4670
2020-05-08 18:46:51,858 - mmdet - INFO - Epoch [1][1450/7517]   lr: 0.00125, eta: 4:41:52, time: 0.594, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0984, loss_rpn_bbox: 0.0549, loss_cls: 0.2000, acc: 93.8301, loss_bbox: 0.1417, loss: 0.4950
2020-05-08 18:47:21,486 - mmdet - INFO - Epoch [1][1500/7517]   lr: 0.00125, eta: 4:41:24, time: 0.593, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0985, loss_rpn_bbox: 0.0498, loss_cls: 0.1882, acc: 93.4492, loss_bbox: 0.1489, loss: 0.4854
2020-05-08 18:47:51,164 - mmdet - INFO - Epoch [1][1550/7517]   lr: 0.00125, eta: 4:40:56, time: 0.594, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0859, loss_rpn_bbox: 0.0547, loss_cls: 0.1944, acc: 93.6719, loss_bbox: 0.1437, loss: 0.4787
2020-05-08 18:48:20,803 - mmdet - INFO - Epoch [1][1600/7517]   lr: 0.00125, eta: 4:40:28, time: 0.593, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0971, loss_rpn_bbox: 0.0553, loss_cls: 0.2034, acc: 93.7070, loss_bbox: 0.1464, loss: 0.5022
2020-05-08 18:48:50,516 - mmdet - INFO - Epoch [1][1650/7517]   lr: 0.00125, eta: 4:40:01, time: 0.594, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0905, loss_rpn_bbox: 0.0429, loss_cls: 0.1954, acc: 93.6035, loss_bbox: 0.1453, loss: 0.4740
2020-05-08 18:49:20,057 - mmdet - INFO - Epoch [1][1700/7517]   lr: 0.00125, eta: 4:39:31, time: 0.591, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0867, loss_rpn_bbox: 0.0558, loss_cls: 0.1975, acc: 93.4764, loss_bbox: 0.1494, loss: 0.4893
2020-05-08 18:49:49,665 - mmdet - INFO - Epoch [1][1750/7517]   lr: 0.00125, eta: 4:39:03, time: 0.592, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0923, loss_rpn_bbox: 0.0606, loss_cls: 0.1920, acc: 93.2070, loss_bbox: 0.1531, loss: 0.4980
2020-05-08 18:50:19,288 - mmdet - INFO - Epoch [1][1800/7517]   lr: 0.00125, eta: 4:38:34, time: 0.592, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0941, loss_rpn_bbox: 0.0545, loss_cls: 0.2239, acc: 92.7461, loss_bbox: 0.1515, loss: 0.5239
2020-05-08 18:50:48,893 - mmdet - INFO - Epoch [1][1850/7517]   lr: 0.00125, eta: 4:38:05, time: 0.592, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0923, loss_rpn_bbox: 0.0611, loss_cls: 0.1953, acc: 93.6113, loss_bbox: 0.1501, loss: 0.4988
2020-05-08 18:51:18,551 - mmdet - INFO - Epoch [1][1900/7517]   lr: 0.00125, eta: 4:37:37, time: 0.593, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.1114, loss_rpn_bbox: 0.0510, loss_cls: 0.2244, acc: 92.2227, loss_bbox: 0.1783, loss: 0.5650
2020-05-08 18:51:48,137 - mmdet - INFO - Epoch [1][1950/7517]   lr: 0.00125, eta: 4:37:08, time: 0.592, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0935, loss_rpn_bbox: 0.0532, loss_cls: 0.1668, acc: 94.3151, loss_bbox: 0.1293, loss: 0.4427
2020-05-08 18:52:17,706 - mmdet - INFO - Epoch [1][2000/7517]   lr: 0.00125, eta: 4:36:38, time: 0.591, data_time: 0.030, memory: 6836, loss_rpn_cls: 0.0826, loss_rpn_bbox: 0.0385, loss_cls: 0.1624, acc: 94.5071, loss_bbox: 0.1228, loss: 0.4063
2020-05-08 18:52:47,270 - mmdet - INFO - Epoch [1][2050/7517]   lr: 0.00125, eta: 4:36:08, time: 0.591, data_time: 0.030, memory: 6836, loss_rpn_cls: 0.0973, loss_rpn_bbox: 0.0421, loss_cls: 0.1756, acc: 94.4219, loss_bbox: 0.1282, loss: 0.4433
2020-05-08 18:53:16,902 - mmdet - INFO - Epoch [1][2100/7517]   lr: 0.00125, eta: 4:35:40, time: 0.593, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0908, loss_rpn_bbox: 0.0676, loss_cls: 0.1859, acc: 93.8945, loss_bbox: 0.1352, loss: 0.4794
2020-05-08 18:53:47,582 - mmdet - INFO - Epoch [1][2150/7517]   lr: 0.00125, eta: 4:35:24, time: 0.614, data_time: 0.038, memory: 6836, loss_rpn_cls: 0.0943, loss_rpn_bbox: 0.0446, loss_cls: 0.1971, acc: 93.5371, loss_bbox: 0.1427, loss: 0.4788
2020-05-08 18:54:17,782 - mmdet - INFO - Epoch [1][2200/7517]   lr: 0.00125, eta: 4:35:03, time: 0.604, data_time: 0.034, memory: 6836, loss_rpn_cls: 0.0894, loss_rpn_bbox: 0.0565, loss_cls: 0.2019, acc: 93.4089, loss_bbox: 0.1487, loss: 0.4965
2020-05-08 18:54:47,474 - mmdet - INFO - Epoch [1][2250/7517]   lr: 0.00125, eta: 4:34:34, time: 0.594, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0942, loss_rpn_bbox: 0.0521, loss_cls: 0.1924, acc: 93.8984, loss_bbox: 0.1376, loss: 0.4763
2020-05-08 18:55:17,131 - mmdet - INFO - Epoch [1][2300/7517]   lr: 0.00125, eta: 4:34:05, time: 0.593, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0903, loss_rpn_bbox: 0.0708, loss_cls: 0.1835, acc: 94.2520, loss_bbox: 0.1314, loss: 0.4761
2020-05-08 18:55:46,904 - mmdet - INFO - Epoch [1][2350/7517]   lr: 0.00125, eta: 4:33:37, time: 0.595, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0865, loss_rpn_bbox: 0.0491, loss_cls: 0.1859, acc: 93.4395, loss_bbox: 0.1570, loss: 0.4786
2020-05-08 18:56:16,649 - mmdet - INFO - Epoch [1][2400/7517]   lr: 0.00125, eta: 4:33:09, time: 0.595, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0989, loss_rpn_bbox: 0.0477, loss_cls: 0.1939, acc: 93.4961, loss_bbox: 0.1484, loss: 0.4890
2020-05-08 18:56:46,439 - mmdet - INFO - Epoch [1][2450/7517]   lr: 0.00125, eta: 4:32:41, time: 0.596, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0875, loss_rpn_bbox: 0.0440, loss_cls: 0.1996, acc: 93.3379, loss_bbox: 0.1483, loss: 0.4794
2020-05-08 18:57:16,177 - mmdet - INFO - Epoch [1][2500/7517]   lr: 0.00125, eta: 4:32:13, time: 0.595, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0846, loss_rpn_bbox: 0.0401, loss_cls: 0.1984, acc: 93.3535, loss_bbox: 0.1474, loss: 0.4704
2020-05-08 18:57:45,944 - mmdet - INFO - Epoch [1][2550/7517]   lr: 0.00125, eta: 4:31:45, time: 0.595, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0738, loss_rpn_bbox: 0.0554, loss_cls: 0.1680, acc: 94.2461, loss_bbox: 0.1295, loss: 0.4268
2020-05-08 18:58:15,692 - mmdet - INFO - Epoch [1][2600/7517]   lr: 0.00125, eta: 4:31:17, time: 0.595, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0842, loss_rpn_bbox: 0.0448, loss_cls: 0.1766, acc: 94.0488, loss_bbox: 0.1314, loss: 0.4369
2020-05-08 18:58:45,411 - mmdet - INFO - Epoch [1][2650/7517]   lr: 0.00125, eta: 4:30:48, time: 0.594, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0805, loss_rpn_bbox: 0.0493, loss_cls: 0.1815, acc: 93.8945, loss_bbox: 0.1426, loss: 0.4538
2020-05-08 18:59:15,232 - mmdet - INFO - Epoch [1][2700/7517]   lr: 0.00125, eta: 4:30:20, time: 0.596, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0810, loss_rpn_bbox: 0.0430, loss_cls: 0.1615, acc: 94.2578, loss_bbox: 0.1459, loss: 0.4315
2020-05-08 18:59:44,999 - mmdet - INFO - Epoch [1][2750/7517]   lr: 0.00125, eta: 4:29:52, time: 0.595, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0911, loss_rpn_bbox: 0.0521, loss_cls: 0.1714, acc: 93.9961, loss_bbox: 0.1377, loss: 0.4523
2020-05-08 19:00:14,871 - mmdet - INFO - Epoch [1][2800/7517]   lr: 0.00125, eta: 4:29:24, time: 0.597, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0912, loss_rpn_bbox: 0.0518, loss_cls: 0.1645, acc: 94.2871, loss_bbox: 0.1247, loss: 0.4322
2020-05-08 19:00:44,768 - mmdet - INFO - Epoch [1][2850/7517]   lr: 0.00125, eta: 4:28:57, time: 0.598, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0784, loss_rpn_bbox: 0.0450, loss_cls: 0.1670, acc: 94.3574, loss_bbox: 0.1320, loss: 0.4223
2020-05-08 19:01:14,560 - mmdet - INFO - Epoch [1][2900/7517]   lr: 0.00125, eta: 4:28:29, time: 0.596, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0739, loss_rpn_bbox: 0.0490, loss_cls: 0.1477, acc: 95.0176, loss_bbox: 0.1148, loss: 0.3854
2020-05-08 19:01:44,278 - mmdet - INFO - Epoch [1][2950/7517]   lr: 0.00125, eta: 4:28:00, time: 0.594, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0686, loss_rpn_bbox: 0.0597, loss_cls: 0.1359, acc: 95.3145, loss_bbox: 0.1058, loss: 0.3700
2020-05-08 19:02:14,081 - mmdet - INFO - Epoch [1][3000/7517]   lr: 0.00125, eta: 4:27:32, time: 0.596, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0853, loss_rpn_bbox: 0.0439, loss_cls: 0.1640, acc: 94.3730, loss_bbox: 0.1284, loss: 0.4216
2020-05-08 19:02:43,866 - mmdet - INFO - Epoch [1][3050/7517]   lr: 0.00125, eta: 4:27:03, time: 0.596, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0771, loss_rpn_bbox: 0.0471, loss_cls: 0.1945, acc: 93.0462, loss_bbox: 0.1557, loss: 0.4744
2020-05-08 19:03:13,665 - mmdet - INFO - Epoch [1][3100/7517]   lr: 0.00125, eta: 4:26:35, time: 0.596, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0767, loss_rpn_bbox: 0.0489, loss_cls: 0.1698, acc: 94.0155, loss_bbox: 0.1407, loss: 0.4361
2020-05-08 19:03:43,494 - mmdet - INFO - Epoch [1][3150/7517]   lr: 0.00125, eta: 4:26:07, time: 0.597, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0811, loss_rpn_bbox: 0.0406, loss_cls: 0.1900, acc: 93.7637, loss_bbox: 0.1365, loss: 0.4482
2020-05-08 19:04:13,333 - mmdet - INFO - Epoch [1][3200/7517]   lr: 0.00125, eta: 4:25:38, time: 0.597, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0757, loss_rpn_bbox: 0.0448, loss_cls: 0.1551, acc: 94.5488, loss_bbox: 0.1264, loss: 0.4019
2020-05-08 19:04:43,118 - mmdet - INFO - Epoch [1][3250/7517]   lr: 0.00125, eta: 4:25:10, time: 0.596, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0892, loss_rpn_bbox: 0.0514, loss_cls: 0.1474, acc: 94.9336, loss_bbox: 0.1216, loss: 0.4096
2020-05-08 19:05:12,868 - mmdet - INFO - Epoch [1][3300/7517]   lr: 0.00125, eta: 4:24:41, time: 0.595, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0849, loss_rpn_bbox: 0.0498, loss_cls: 0.1687, acc: 94.1736, loss_bbox: 0.1292, loss: 0.4326
2020-05-08 19:05:42,746 - mmdet - INFO - Epoch [1][3350/7517]   lr: 0.00125, eta: 4:24:13, time: 0.598, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0859, loss_rpn_bbox: 0.0626, loss_cls: 0.1725, acc: 94.1172, loss_bbox: 0.1374, loss: 0.4584
2020-05-08 19:06:12,606 - mmdet - INFO - Epoch [1][3400/7517]   lr: 0.00125, eta: 4:23:45, time: 0.597, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0897, loss_rpn_bbox: 0.0513, loss_cls: 0.1917, acc: 93.1367, loss_bbox: 0.1591, loss: 0.4917
2020-05-08 19:06:42,322 - mmdet - INFO - Epoch [1][3450/7517]   lr: 0.00125, eta: 4:23:15, time: 0.594, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0824, loss_rpn_bbox: 0.0405, loss_cls: 0.1667, acc: 94.3950, loss_bbox: 0.1385, loss: 0.4281
2020-05-08 19:07:12,140 - mmdet - INFO - Epoch [1][3500/7517]   lr: 0.00125, eta: 4:22:47, time: 0.596, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0838, loss_rpn_bbox: 0.0398, loss_cls: 0.1619, acc: 94.4992, loss_bbox: 0.1238, loss: 0.4092
2020-05-08 19:07:42,005 - mmdet - INFO - Epoch [1][3550/7517]   lr: 0.00125, eta: 4:22:19, time: 0.597, data_time: 0.033, memory: 6836, loss_rpn_cls: 0.0705, loss_rpn_bbox: 0.0565, loss_cls: 0.1381, acc: 95.0566, loss_bbox: 0.1226, loss: 0.3877
2020-05-08 19:08:11,797 - mmdet - INFO - Epoch [1][3600/7517]   lr: 0.00125, eta: 4:21:50, time: 0.596, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0783, loss_rpn_bbox: 0.0564, loss_cls: 0.1743, acc: 94.1113, loss_bbox: 0.1298, loss: 0.4388
2020-05-08 19:08:41,587 - mmdet - INFO - Epoch [1][3650/7517]   lr: 0.00125, eta: 4:21:21, time: 0.596, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0677, loss_rpn_bbox: 0.0459, loss_cls: 0.1438, acc: 94.7852, loss_bbox: 0.1179, loss: 0.3753
2020-05-08 19:09:11,449 - mmdet - INFO - Epoch [1][3700/7517]   lr: 0.00125, eta: 4:20:52, time: 0.597, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0686, loss_rpn_bbox: 0.0447, loss_cls: 0.1380, acc: 95.4746, loss_bbox: 0.1051, loss: 0.3563
2020-05-08 19:09:41,355 - mmdet - INFO - Epoch [1][3750/7517]   lr: 0.00125, eta: 4:20:24, time: 0.598, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0723, loss_rpn_bbox: 0.0551, loss_cls: 0.1667, acc: 94.2285, loss_bbox: 0.1325, loss: 0.4267
2020-05-08 19:10:11,282 - mmdet - INFO - Epoch [1][3800/7517]   lr: 0.00125, eta: 4:19:56, time: 0.599, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0799, loss_rpn_bbox: 0.0539, loss_cls: 0.1657, acc: 94.1719, loss_bbox: 0.1320, loss: 0.4314
2020-05-08 19:10:41,140 - mmdet - INFO - Epoch [1][3850/7517]   lr: 0.00125, eta: 4:19:28, time: 0.597, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0699, loss_rpn_bbox: 0.0449, loss_cls: 0.1236, acc: 95.5137, loss_bbox: 0.1059, loss: 0.3442
2020-05-08 19:11:11,019 - mmdet - INFO - Epoch [1][3900/7517]   lr: 0.00125, eta: 4:18:59, time: 0.598, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0880, loss_rpn_bbox: 0.0423, loss_cls: 0.1669, acc: 93.9512, loss_bbox: 0.1360, loss: 0.4332
2020-05-08 19:11:40,695 - mmdet - INFO - Epoch [1][3950/7517]   lr: 0.00125, eta: 4:18:30, time: 0.593, data_time: 0.033, memory: 6836, loss_rpn_cls: 0.1018, loss_rpn_bbox: 0.0408, loss_cls: 0.1757, acc: 93.8169, loss_bbox: 0.1296, loss: 0.4478
2020-05-08 19:12:10,532 - mmdet - INFO - Epoch [1][4000/7517]   lr: 0.00125, eta: 4:18:01, time: 0.597, data_time: 0.033, memory: 6836, loss_rpn_cls: 0.0833, loss_rpn_bbox: 0.0519, loss_cls: 0.1747, acc: 93.9035, loss_bbox: 0.1265, loss: 0.4364
2020-05-08 19:12:40,406 - mmdet - INFO - Epoch [1][4050/7517]   lr: 0.00125, eta: 4:17:32, time: 0.597, data_time: 0.033, memory: 6836, loss_rpn_cls: 0.0880, loss_rpn_bbox: 0.0503, loss_cls: 0.1858, acc: 93.2878, loss_bbox: 0.1528, loss: 0.4768
2020-05-08 19:13:10,193 - mmdet - INFO - Epoch [1][4100/7517]   lr: 0.00125, eta: 4:17:03, time: 0.596, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0675, loss_rpn_bbox: 0.0397, loss_cls: 0.1543, acc: 94.6758, loss_bbox: 0.1087, loss: 0.3701
2020-05-08 19:13:39,963 - mmdet - INFO - Epoch [1][4150/7517]   lr: 0.00125, eta: 4:16:34, time: 0.595, data_time: 0.031, memory: 6836, loss_rpn_cls: 0.0825, loss_rpn_bbox: 0.0450, loss_cls: 0.1579, acc: 94.4629, loss_bbox: 0.1206, loss: 0.4060
2020-05-08 19:14:09,798 - mmdet - INFO - Epoch [1][4200/7517]   lr: 0.00125, eta: 4:16:05, time: 0.597, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0813, loss_rpn_bbox: 0.0501, loss_cls: 0.1761, acc: 93.9102, loss_bbox: 0.1374, loss: 0.4450
2020-05-08 19:14:39,571 - mmdet - INFO - Epoch [1][4250/7517]   lr: 0.00125, eta: 4:15:36, time: 0.595, data_time: 0.032, memory: 6836, loss_rpn_cls: 0.0818, loss_rpn_bbox: 0.0490, loss_cls: 0.1750, acc: 94.1836, loss_bbox: 0.1283, loss: 0.4341

Fortunately, although the speed is slow, the accuracy is good (mAP = 72.4)

Is it normal to cost 5 hours to train a Faster-RCNN-C4 network on VOC 2007? Did I miss something?

serend1p1ty commented 4 years ago

I also noticed gpu memory is a little larger than detectron2 (6836 VS 4529, image per batch = 2)

hellock commented 4 years ago

Oh I didn't notice that you were using MMDetection v.1.1. You may try v2.0 which is faster than v1.1.

serend1p1ty commented 4 years ago

@hellock OK, I'll test V2 first. Thank you for your reply.

serend1p1ty commented 4 years ago

Yes, the training speed of v2 is similar to detectron2. @hellock