gap in map

[x-x110]

My experimental equipment is 3xTitan, and according to the rules of Detectron2, set the learning rate to 0.045. Without modifying any parameters, the resulting map is about 35.6. why?

[x-x110]

batch size 16 / per

[chensnathan]

Hi, Could you post your training log?

[x-x110]

CUDNN_BENCHMARK: false DATALOADER: ASPECT_RATIO_GROUPING: true FILTER_EMPTY_ANNOTATIONS: true NUM_WORKERS: 8 REPEAT_THRESHOLD: 0.0 SAMPLER_TRAIN: TrainingSampler DATASETS: PRECOMPUTED_PROPOSAL_TOPK_TEST: 1000 PRECOMPUTED_PROPOSAL_TOPK_TRAIN: 2000 PROPOSAL_FILES_TEST: [] PROPOSAL_FILES_TRAIN: [] TEST:

• coco_2017_val TRAIN:
• coco_2017_train GLOBAL: HACK: 1.0 INPUT: CROP: ENABLED: false SIZE:
  • 0.9
  • 0.9 TYPE: relative_range DISTORTION: ENABLED: false EXPOSURE: 1.5 HUE: 0.1 SATURATION: 1.5 FORMAT: BGR JITTER_CROP: ENABLED: false JITTER_RATIO: 0.3 MASK_FORMAT: polygon MAX_SIZE_TEST: 1333 MAX_SIZE_TRAIN: 1333 MIN_SIZE_TEST: 800 MIN_SIZE_TRAIN:
• 800 MIN_SIZE_TRAIN_SAMPLING: choice MOSAIC: ENABLED: false MIN_OFFSET: 0.2 MOSAIC_HEIGHT: 640 MOSAIC_WIDTH: 640 NUM_IMAGES: 4 POOL_CAPACITY: 1000 RANDOM_FLIP: horizontal RESIZE: ENABLED: false SCALE_JITTER:
  • 0.8
  • 1.2 SHAPE:
  • 640
  • 640 TEST_SHAPE:
  • 608
  • 608 SHIFT: SHIFT_PIXELS: 32 MODEL: ANCHOR_GENERATOR: ANGLES:
  • • -90
    • 0
    • 90 ASPECT_RATIOS:
  • • 1.0 NAME: DefaultAnchorGenerator OFFSET: 0.0 SIZES:
  • • 32
    • 64
    • 128
    • 256
    • 512 BACKBONE: FREEZE_AT: 2 NAME: build_resnet_backbone DARKNET: DEPTH: 53 NORM: BN OUT_FEATURES:
  • res5 RES5_DILATION: 1 WITH_CSP: true DEVICE: cuda FPN: FUSE_TYPE: sum IN_FEATURES: [] NORM: '' OUT_CHANNELS: 256 KEYPOINT_ON: false LOAD_PROPOSALS: false MASK_ON: false META_ARCHITECTURE: YOLOF PANOPTIC_FPN: COMBINE: ENABLED: true INSTANCES_CONFIDENCE_THRESH: 0.5 OVERLAP_THRESH: 0.5 STUFF_AREA_LIMIT: 4096 INSTANCE_LOSS_WEIGHT: 1.0 PIXEL_MEAN:
• 103.53
• 116.28
• 123.675 PIXEL_STD:
• 1.0
• 1.0
• 1.0 PROPOSAL_GENERATOR: MIN_SIZE: 0 NAME: RPN RESNETS: DEFORM_MODULATED: false DEFORM_NUM_GROUPS: 1 DEFORM_ON_PER_STAGE:
  • false
  • false
  • false
  • false DEPTH: 50 NORM: FrozenBN NUM_GROUPS: 1 OUT_FEATURES:
  • res5 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: true WIDTH_PER_GROUP: 64 RETINANET: BBOX_REG_LOSS_TYPE: smooth_l1 BBOX_REG_WEIGHTS: &id001
  • 1.0
  • 1.0
  • 1.0
  • 1.0 FOCAL_LOSS_ALPHA: 0.25 FOCAL_LOSS_GAMMA: 2.0 IN_FEATURES:
  • p3
  • p4
  • p5
  • p6
  • p7 IOU_LABELS:
  • 0
  • -1
  • 1 IOU_THRESHOLDS:
  • 0.4
  • 0.5 NMS_THRESH_TEST: 0.5 NORM: '' NUM_CLASSES: 80 NUM_CONVS: 4 PRIOR_PROB: 0.01 SCORE_THRESH_TEST: 0.05 SMOOTH_L1_LOSS_BETA: 0.1 TOPK_CANDIDATES_TEST: 1000 ROI_BOX_CASCADE_HEAD: BBOX_REG_WEIGHTS:
  • • 10.0
    • 10.0
    • 5.0
    • 5.0
  • • 20.0
    • 20.0
    • 10.0
    • 10.0
  • • 30.0
    • 30.0
    • 15.0
    • 15.0 IOUS:
  • 0.5
  • 0.6
  • 0.7 ROI_BOX_HEAD: BBOX_REG_LOSS_TYPE: smooth_l1 BBOX_REG_LOSS_WEIGHT: 1.0 BBOX_REG_WEIGHTS:
  • 10.0
  • 10.0
  • 5.0
  • 5.0 CLS_AGNOSTIC_BBOX_REG: false CONV_DIM: 256 FC_DIM: 1024 NAME: '' NORM: '' NUM_CONV: 0 NUM_FC: 0 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 SMOOTH_L1_BETA: 0.0 TRAIN_ON_PRED_BOXES: false ROI_HEADS: BATCH_SIZE_PER_IMAGE: 512 IN_FEATURES:
  • res4 IOU_LABELS:
  • 0
  • 1 IOU_THRESHOLDS:
  • 0.5 NAME: Res5ROIHeads NMS_THRESH_TEST: 0.5 NUM_CLASSES: 80 POSITIVE_FRACTION: 0.25 PROPOSAL_APPEND_GT: true SCORE_THRESH_TEST: 0.05 ROI_KEYPOINT_HEAD: CONV_DIMS:
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512 LOSS_WEIGHT: 1.0 MIN_KEYPOINTS_PER_IMAGE: 1 NAME: KRCNNConvDeconvUpsampleHead NORMALIZE_LOSS_BY_VISIBLE_KEYPOINTS: true NUM_KEYPOINTS: 17 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 ROI_MASK_HEAD: CLS_AGNOSTIC_MASK: false CONV_DIM: 256 NAME: MaskRCNNConvUpsampleHead NORM: '' NUM_CONV: 0 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 RPN: BATCH_SIZE_PER_IMAGE: 256 BBOX_REG_LOSS_TYPE: smooth_l1 BBOX_REG_LOSS_WEIGHT: 1.0 BBOX_REG_WEIGHTS: *id001 BOUNDARY_THRESH: -1 CONV_DIMS:
  • -1 HEAD_NAME: StandardRPNHead IN_FEATURES:
  • res4 IOU_LABELS:
  • 0
  • -1
  • 1 IOU_THRESHOLDS:
  • 0.3
  • 0.7 LOSS_WEIGHT: 1.0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOPK_TEST: 1000 POST_NMS_TOPK_TRAIN: 2000 PRE_NMS_TOPK_TEST: 6000 PRE_NMS_TOPK_TRAIN: 12000 SMOOTH_L1_BETA: 0.0 SEM_SEG_HEAD: COMMON_STRIDE: 4 CONVS_DIM: 128 IGNORE_VALUE: 255 IN_FEATURES:
  • p2
  • p3
  • p4
  • p5 LOSS_WEIGHT: 1.0 NAME: SemSegFPNHead NORM: GN NUM_CLASSES: 54 WEIGHTS: detectron2://ImageNetPretrained/MSRA/R-50.pkl YOLOF: BOX_TRANSFORM: ADD_CTR_CLAMP: true BBOX_REG_WEIGHTS:
    • 1.0
    • 1.0
    • 1.0
    • 1.0 CTR_CLAMP: 32 DECODER: ACTIVATION: ReLU CLS_NUM_CONVS: 2 IN_CHANNELS: 512 NORM: BN NUM_ANCHORS: 5 NUM_CLASSES: 80 PRIOR_PROB: 0.01 REG_NUM_CONVS: 4 DETECTIONS_PER_IMAGE: 100 ENCODER: ACTIVATION: ReLU BACKBONE_LEVEL: res5 BLOCK_DILATIONS:
    • 2
    • 4
    • 6
    • 8 BLOCK_MID_CHANNELS: 128 IN_CHANNELS: 2048 NORM: BN NUM_CHANNELS: 512 NUM_RESIDUAL_BLOCKS: 4 LOSSES: BBOX_REG_LOSS_TYPE: giou FOCAL_LOSS_ALPHA: 0.25 FOCAL_LOSS_GAMMA: 2.0 MATCHER: TOPK: 4 NEG_IGNORE_THRESHOLD: 0.7 NMS_THRESH_TEST: 0.6 POS_IGNORE_THRESHOLD: 0.15 SCORE_THRESH_TEST: 0.05 TOPK_CANDIDATES_TEST: 1000 OUTPUT_DIR: output/yolof/R_50_C5_1x SEED: -1 SOLVER: AMP: ENABLED: false BACKBONE_MULTIPLIER: 0.334 BASE_LR: 0.045 BIAS_LR_FACTOR: 1.0 CHECKPOINT_PERIOD: 2500 CLIP_GRADIENTS: CLIP_TYPE: value CLIP_VALUE: 1.0 ENABLED: false NORM_TYPE: 2.0 GAMMA: 0.1 IMS_PER_BATCH: 48 LR_SCHEDULER_NAME: WarmupMultiStepLR MAX_ITER: 22500 MOMENTUM: 0.9 NESTEROV: false REFERENCE_WORLD_SIZE: 0 STEPS:
• 15000
• 20000 WARMUP_FACTOR: 0.00066667 WARMUP_ITERS: 1500 WARMUP_METHOD: linear WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0.0001 WEIGHT_DECAY_NORM: 0.0 TEST: AUG: ENABLED: false FLIP: true MAX_SIZE: 4000 MIN_SIZES:
  • 400
  • 500
  • 600
  • 700
  • 800
  • 900
  • 1000
  • 1100
  • 1200 DETECTIONS_PER_IMAGE: 100 EVAL_PERIOD: 0 EXPECTED_RESULTS: [] KEYPOINT_OKS_SIGMAS: [] PRECISE_BN: ENABLED: false NUM_ITER: 200 VERSION: 2 VIS_PERIOD: 0

[chensnathan]

There are several key points about how to modify the settings:

1. Batch size and learning rate. The default settings are set for 8 GPUs with a total batch size of 64 (8 images per GPU). You have 3 GPUs, and set the batch size as 48 (16 images per GPU). Thus, according to the scaling rule, your learning rate should be 0.12 * 48 / 64 = 0.09.
2. Training iterations and learning rate steps. We training a maximum iteration of 22500 for batch size 64, for batch size 48, you should also modify the maximum iteration from 22500 to 22500 64 / 48 = 30000. For the learning rate steps, they also should be re-calculated according to the rule, [15000 64 / 48, 20000 * 64 / 48]
3. The warmup iterations and the warmup factor. For batch size 64, we warm up the training for 1500 iterations. Thus, for batch size 48, you can modify it from 1500 iterations to 1500 * 64 / 48 = 2000 iterations. And for the warmup factor, it can be obtained by: 1. / 2000.

[x-x110]

I will modify these parameters and provide the result. Thank you for your reply

[x-x110]

After modifying these parameters, 37.39 can be obtained in the iteration times of 30000

[chensnathan]

This result is reasonable.

[x-x110]

thanks you reply

[shenhaibb]

There are several key points about how to modify the settings:

1. Batch size and learning rate. The default settings are set for 8 GPUs with a total batch size of 64 (8 images per GPU). You have 3 GPUs, and set the batch size as 48 (16 images per GPU). Thus, according to the scaling rule, your learning rate should be 0.12 * 48 / 64 = 0.09.
2. Training iterations and learning rate steps. We training a maximum iteration of 22500 for batch size 64, for batch size 48, you should also modify the maximum iteration from 22500 to 22500 64 / 48 = 30000. For the learning rate steps, they also should be re-calculated according to the rule, [15000 64 / 48, 20000 * 64 / 48]
3. The warmup iterations and the warmup factor. For batch size 64, we warm up the training for 1500 iterations. Thus, for batch size 48, you can modify it from 1500 iterations to 1500 * 64 / 48 = 2000 iterations. And for the warmup factor, it can be obtained by: 1. / 2000.

hi, i have only 1gpu(8gb) i set batch size as 8 learning rate: 0.12 8 / 64 = 0.0015 maximum iteration: 22500 64 / 8 = 180000 learning rate steps: [15000 64 / 8, 20000 64 / 48] == [120000, 160000] warm up iterations: 1500 * 64 / 8 = 12000 warmup factor: 1. / 2000 = 0.0005 Is it the way I calculated it？

[SelimSavas]

There are several key points about how to modify the settings:

1. Batch size and learning rate. The default settings are set for 8 GPUs with a total batch size of 64 (8 images per GPU). You have 3 GPUs, and set the batch size as 48 (16 images per GPU). Thus, according to the scaling rule, your learning rate should be 0.12 * 48 / 64 = 0.09.
2. Training iterations and learning rate steps. We training a maximum iteration of 22500 for batch size 64, for batch size 48, you should also modify the maximum iteration from 22500 to 22500 64 / 48 = 30000. For the learning rate steps, they also should be re-calculated according to the rule, [15000 64 / 48, 20000 * 64 / 48]
3. The warmup iterations and the warmup factor. For batch size 64, we warm up the training for 1500 iterations. Thus, for batch size 48, you can modify it from 1500 iterations to 1500 * 64 / 48 = 2000 iterations. And for the warmup factor, it can be obtained by: 1. / 2000.

There are several key points about how to modify the settings:

1. Batch size and learning rate. The default settings are set for 8 GPUs with a total batch size of 64 (8 images per GPU). You have 3 GPUs, and set the batch size as 48 (16 images per GPU). Thus, according to the scaling rule, your learning rate should be 0.12 * 48 / 64 = 0.09.
2. Training iterations and learning rate steps. We training a maximum iteration of 22500 for batch size 64, for batch size 48, you should also modify the maximum iteration from 22500 to 22500 64 / 48 = 30000. For the learning rate steps, they also should be re-calculated according to the rule, [15000 64 / 48, 20000 * 64 / 48]
3. The warmup iterations and the warmup factor. For batch size 64, we warm up the training for 1500 iterations. Thus, for batch size 48, you can modify it from 1500 iterations to 1500 * 64 / 48 = 2000 iterations. And for the warmup factor, it can be obtained by: 1. / 2000.

Is this calculation valid for every data set? Are we going to do the same calculation for datasets of different sizes? @chensnathan