Meet Problems on training

[ToAmadeus]

I successfully starting the training,but i meet some the problem which makes my accuracy very low. ''' Above is part of my training logs,obviously my training loss keeps very high. [05/11 11:05:31 d2.utils.events]: eta: 0:16:35 iter: 2839 total_loss: 13.81 loss_ce: 1.062 loss_bbox: 0.2362 loss_giou: 0.7347 loss_ce_0: 1.533 loss_bbox_0: 0.4347 loss_giou_0: 1.347 loss_ce_1: 1.076 loss_bbox_1: 0.3313 loss_giou_1: 0.9831 loss_ce_2: 0.983 loss_bbox_2: 0.2493 loss_giou_2: 0.8497 loss_ce_3: 1.095 loss_bbox_3: 0.2378 loss_giou_3: 0.8031 loss_ce_4: 1.037 loss_bbox_4: 0.2179 loss_giou_4: 0.7701 time: 0.4684 last_time: 0.4062 data_time: 0.0213 last_data_time: 0.0158 lr: 2.5e-07 max_mem: 4188M [05/11 11:05:41 d2.utils.events]: eta: 0:16:28 iter: 2859 total_loss: 15.24 loss_ce: 1.078 loss_bbox: 0.2586 loss_giou: 0.9109 loss_ce_0: 1.416 loss_bbox_0: 0.5326 loss_giou_0: 1.461 loss_ce_1: 1.054 loss_bbox_1: 0.3595 loss_giou_1: 1.137 loss_ce_2: 1.06 loss_bbox_2: 0.3113 loss_giou_2: 1.046 loss_ce_3: 1.061 loss_bbox_3: 0.2689 loss_giou_3: 0.9477 loss_ce_4: 1.049 loss_bbox_4: 0.249 loss_giou_4: 0.9952 time: 0.4684 last_time: 0.5241 data_time: 0.0168 last_data_time: 0.0031 lr: 2.5e-07 max_mem: 4188M [05/11 11:05:50 d2.utils.events]: eta: 0:16:21 iter: 2879 total_loss: 13.91 loss_ce: 0.9869 loss_bbox: 0.2141 loss_giou: 0.8199 loss_ce_0: 1.447 loss_bbox_0: 0.4575 loss_giou_0: 1.436 loss_ce_1: 1.119 loss_bbox_1: 0.3276 loss_giou_1: 1.095 loss_ce_2: 0.9777 loss_bbox_2: 0.257 loss_giou_2: 0.9116 loss_ce_3: 0.9843 loss_bbox_3: 0.2249 loss_giou_3: 0.8466 loss_ce_4: 0.9701 loss_bbox_4: 0.214 loss_giou_4: 0.8948 time: 0.4685 last_time: 0.4227 data_time: 0.0191 last_data_time: 0.0087 lr: 2.5e-07 max_mem: 4188M [05/11 11:06:00 d2.utils.events]: eta: 0:16:13 iter: 2899 total_loss: 15.19 loss_ce: 1.103 loss_bbox: 0.2365 loss_giou: 0.9256 loss_ce_0: 1.504 loss_bbox_0: 0.4395 loss_giou_0: 1.279 loss_ce_1: 1.222 loss_bbox_1: 0.3448 loss_giou_1: 1.117 loss_ce_2: 1.162 loss_bbox_2: 0.2908 loss_giou_2: 0.9648 loss_ce_3: 1.125 loss_bbox_3: 0.221 loss_giou_3: 0.8601 loss_ce_4: 1.114 loss_bbox_4: 0.2222 loss_giou_4: 0.8862 time: 0.4684 last_time: 0.3591 data_time: 0.0175 last_data_time: 0.0036 lr: 2.5e-07 max_mem: 4189M [05/11 11:06:09 d2.utils.events]: eta: 0:16:03 iter: 2919 total_loss: 16.14 loss_ce: 1.177 loss_bbox: 0.2488 loss_giou: 0.9901 loss_ce_0: 1.581 loss_bbox_0: 0.4737 loss_giou_0: 1.483 loss_ce_1: 1.259 loss_bbox_1: 0.3435 loss_giou_1: 1.139 loss_ce_2: 1.111 loss_bbox_2: 0.2833 loss_giou_2: 1.123 loss_ce_3: 1.209 loss_bbox_3: 0.2793 loss_giou_3: 1.077 loss_ce_4: 1.14 loss_bbox_4: 0.2468 loss_giou_4: 0.9795 time: 0.4683 last_time: 0.4238 data_time: 0.0216 last_data_time: 0.0247 lr: 2.5e-07 max_mem: 4189M [05/11 11:06:18 d2.utils.events]: eta: 0:15:52 iter: 2939 total_loss: 14.07 loss_ce: 1.022 loss_bbox: 0.2032 loss_giou: 0.8397 loss_ce_0: 1.466 loss_bbox_0: 0.4374 loss_giou_0: 1.336 loss_ce_1: 1.121 loss_bbox_1: 0.3034 loss_giou_1: 1.064 loss_ce_2: 1.03 loss_bbox_2: 0.2275 loss_giou_2: 0.8407 loss_ce_3: 1.004 loss_bbox_3: 0.2181 loss_giou_3: 0.8761 loss_ce_4: 1.032 loss_bbox_4: 0.2072 loss_giou_4: 0.8682 time: 0.4682 last_time: 0.6090 data_time: 0.0136 last_data_time: 0.0056 lr: 2.5e-07 max_mem: 4189M [05/11 11:06:27 d2.utils.events]: eta: 0:15:43 iter: 2959 total_loss: 14.88 loss_ce: 1.016 loss_bbox: 0.2088 loss_giou: 0.8485 loss_ce_0: 1.406 loss_bbox_0: 0.5361 loss_giou_0: 1.509 loss_ce_1: 1.142 loss_bbox_1: 0.359 loss_giou_1: 1.156 loss_ce_2: 1.006 loss_bbox_2: 0.3124 loss_giou_2: 0.988 loss_ce_3: 1.055 loss_bbox_3: 0.2405 loss_giou_3: 0.9191 loss_ce_4: 1.029 loss_bbox_4: 0.2193 loss_giou_4: 0.8009 time: 0.4682 last_time: 0.6083 data_time: 0.0237 last_data_time: 0.0375 lr: 2.5e-07 max_mem: 4189M [05/11 11:06:36 d2.utils.events]: eta: 0:15:29 iter: 2979 total_loss: 15.37 loss_ce: 1.096 loss_bbox: 0.2677 loss_giou: 0.8776 loss_ce_0: 1.494 loss_bbox_0: 0.4445 loss_giou_0: 1.361 loss_ce_1: 1.137 loss_bbox_1: 0.2767 loss_giou_1: 1.035 loss_ce_2: 1.104 loss_bbox_2: 0.2791 loss_giou_2: 1.006 loss_ce_3: 1.126 loss_bbox_3: 0.2635 loss_giou_3: 0.9505 loss_ce_4: 1.106 loss_bbox_4: 0.2859 loss_giou_4: 1.016 time: 0.4679 last_time: 0.4104 data_time: 0.0196 last_data_time: 0.0030 lr: 2.5e-07 max_mem: 4189M '''

And here is my config.yaml ''' CUDNN_BENCHMARK: false DATALOADER: ASPECT_RATIO_GROUPING: true FILTER_EMPTY_ANNOTATIONS: false NUM_WORKERS: 4 REPEAT_THRESHOLD: 0.0 SAMPLER_TRAIN: TrainingSampler DATASETS: PRECOMPUTED_PROPOSAL_TOPK_TEST: 1000 PRECOMPUTED_PROPOSAL_TOPK_TRAIN: 2000 PROPOSAL_FILES_TEST: [] PROPOSAL_FILES_TRAIN: [] TEST:

• coco_2017_val TRAIN:
• coco_2017_train GLOBAL: HACK: 1.0 INPUT: CROP: ENABLED: false SIZE:
  • 0.9
  • 0.9 TYPE: absolute_range FORMAT: RGB MASK_FORMAT: polygon MAX_SIZE_TEST: 1800 MAX_SIZE_TRAIN: 1800 MIN_SIZE_TEST: 800 MIN_SIZE_TRAIN:
• 800
• 1000
• 1200
• 1500 MIN_SIZE_TRAIN_SAMPLING: choice RANDOM_FLIP: horizontal MODEL: ANCHOR_GENERATOR: ANGLES:
  • • -90
    • 0
    • 90 ASPECT_RATIOS:
  • • 0.5
    • 1.0
    • 2.0 NAME: DefaultAnchorGenerator OFFSET: 0.0 SIZES:
  • • 32
    • 64
    • 128
    • 256
    • 512 BACKBONE: FREEZE_AT: 2 NAME: build_resnet_fpn_backbone DEVICE: cuda DiffusionDet: ACTIVATION: relu ALPHA: 0.25 CLASS_WEIGHT: 2.0 DEEP_SUPERVISION: true DIM_DYNAMIC: 64 DIM_FEEDFORWARD: 2048 DROPOUT: 0.0 GAMMA: 2.0 GIOU_WEIGHT: 2.0 HIDDEN_DIM: 256 L1_WEIGHT: 5.0 NHEADS: 8 NO_OBJECT_WEIGHT: 0.1 NUM_CLASSES: 7 NUM_CLS: 1 NUM_DYNAMIC: 2 NUM_HEADS: 6 NUM_PROPOSALS: 500 NUM_REG: 3 OTA_K: 5 PRIOR_PROB: 0.01 SAMPLE_STEP: 1 SNR_SCALE: 2.0 USE_FED_LOSS: false USE_FOCAL: true USE_NMS: true FPN: FUSE_TYPE: sum IN_FEATURES:
  • res2
  • res3
  • res4
  • res5 NORM: '' OUT_CHANNELS: 256 KEYPOINT_ON: false LOAD_PROPOSALS: false MASK_ON: false META_ARCHITECTURE: DiffusionDet PANOPTIC_FPN: COMBINE: ENABLED: true INSTANCES_CONFIDENCE_THRESH: 0.5 OVERLAP_THRESH: 0.5 STUFF_AREA_LIMIT: 4096 INSTANCE_LOSS_WEIGHT: 1.0 PIXEL_MEAN:
• 23.7354
• 23.7354
• 23.7354 PIXEL_STD:
• 27.256
• 27.256
• 27.256 PROPOSAL_GENERATOR: MIN_SIZE: 0 NAME: RPN RESNETS: DEFORM_MODULATED: false DEFORM_NUM_GROUPS: 1 DEFORM_ON_PER_STAGE:
  • false
  • false
  • false
  • false DEPTH: 50 NORM: FrozenBN NUM_GROUPS: 1 OUT_FEATURES:
  • res2
  • res3
  • res4
  • res5 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: false WIDTH_PER_GROUP: 64 RETINANET: BBOX_REG_LOSS_TYPE: smooth_l1 BBOX_REG_WEIGHTS: &id002
  • 1.0
  • 1.0
  • 1.0
  • 1.0 FOCAL_LOSS_ALPHA: 0.25 FOCAL_LOSS_GAMMA: 2.0 IN_FEATURES:
  • p3
  • p4
  • p5
  • p6
  • p7 IOU_LABELS:
  • 0
  • -1
  • 1 IOU_THRESHOLDS:
  • 0.4
  • 0.5 NMS_THRESH_TEST: 0.5 NORM: '' NUM_CLASSES: 7 NUM_CONVS: 4 PRIOR_PROB: 0.01 SCORE_THRESH_TEST: 0.05 SMOOTH_L1_LOSS_BETA: 0.1 TOPK_CANDIDATES_TEST: 1000 ROI_BOX_CASCADE_HEAD: BBOX_REG_WEIGHTS:
  • &id001
    • 10.0
    • 10.0
    • 5.0
    • 5.0
  • • 20.0
    • 20.0
    • 10.0
    • 10.0
  • • 30.0
    • 30.0
    • 15.0
    • 15.0 IOUS:
  • 0.5
  • 0.6
  • 0.7 ROI_BOX_HEAD: BBOX_REG_LOSS_TYPE: smooth_l1 BBOX_REG_LOSS_WEIGHT: 1.0 BBOX_REG_WEIGHTS: *id001 CLS_AGNOSTIC_BBOX_REG: false CONV_DIM: 256 FC_DIM: 1024 FED_LOSS_FREQ_WEIGHT_POWER: 0.5 FED_LOSS_NUM_CLASSES: 50 NAME: '' NORM: '' NUM_CONV: 0 NUM_FC: 0 POOLER_RESOLUTION: 7 POOLER_SAMPLING_RATIO: 2 POOLER_TYPE: ROIAlignV2 SMOOTH_L1_BETA: 0.0 TRAIN_ON_PRED_BOXES: false USE_FED_LOSS: false USE_SIGMOID_CE: false ROI_HEADS: BATCH_SIZE_PER_IMAGE: 512 IN_FEATURES:
  • p2
  • p3
  • p4
  • p5 IOU_LABELS:
  • 0
  • 1 IOU_THRESHOLDS:
  • 0.5 NAME: Res5ROIHeads NMS_THRESH_TEST: 0.5 NUM_CLASSES: 7 POSITIVE_FRACTION: 0.25 PROPOSAL_APPEND_GT: true SCORE_THRESH_TEST: 0.05 ROI_KEYPOINT_HEAD: CONV_DIMS:
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512 LOSS_WEIGHT: 1.0 MIN_KEYPOINTS_PER_IMAGE: 1 NAME: KRCNNConvDeconvUpsampleHead NORMALIZE_LOSS_BY_VISIBLE_KEYPOINTS: true NUM_KEYPOINTS: 17 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 ROI_MASK_HEAD: CLS_AGNOSTIC_MASK: false CONV_DIM: 256 NAME: MaskRCNNConvUpsampleHead NORM: '' NUM_CONV: 0 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 RPN: BATCH_SIZE_PER_IMAGE: 256 BBOX_REG_LOSS_TYPE: smooth_l1 BBOX_REG_LOSS_WEIGHT: 1.0 BBOX_REG_WEIGHTS: *id002 BOUNDARY_THRESH: -1 CONV_DIMS:
  • -1 HEAD_NAME: StandardRPNHead IN_FEATURES:
  • res4 IOU_LABELS:
  • 0
  • -1
  • 1 IOU_THRESHOLDS:
  • 0.3
  • 0.7 LOSS_WEIGHT: 1.0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOPK_TEST: 1000 POST_NMS_TOPK_TRAIN: 2000 PRE_NMS_TOPK_TEST: 6000 PRE_NMS_TOPK_TRAIN: 12000 SMOOTH_L1_BETA: 0.0 SEM_SEG_HEAD: COMMON_STRIDE: 4 CONVS_DIM: 128 IGNORE_VALUE: 255 IN_FEATURES:
  • p2
  • p3
  • p4
  • p5 LOSS_WEIGHT: 1.0 NAME: SemSegFPNHead NORM: GN NUM_CLASSES: 54 SWIN: OUT_FEATURES:
  • 0
  • 1
  • 2
  • 3 SIZE: B USE_CHECKPOINT: false WEIGHTS: detectron2://ImageNetPretrained/torchvision/R-50.pkl MODEL_EMA: DECAY: 0.999 DEVICE: '' ENABLED: false USE_EMA_WEIGHTS_FOR_EVAL_ONLY: false YOLOX: false OUTPUT_DIR: ./output SEED: 40244023 SOLVER: AMP: ENABLED: false BACKBONE_MULTIPLIER: 1.0 BASE_LR: 5.0e-06 BASE_LR_END: 0.0 BIAS_LR_FACTOR: 1.0 CHECKPOINT_PERIOD: 5000 CLIP_GRADIENTS: CLIP_TYPE: full_model CLIP_VALUE: 1.0 ENABLED: true NORM_TYPE: 2.0 GAMMA: 0.05 IMS_PER_BATCH: 1 LR_SCHEDULER_NAME: WarmupMultiStepLR MAX_ITER: 5000 MOMENTUM: 0.9 NESTEROV: false NUM_DECAYS: 3 OPTIMIZER: ADAMW REFERENCE_WORLD_SIZE: 0 RESCALE_INTERVAL: false STEPS:
• 2000
• 3999 WARMUP_FACTOR: 0.01 WARMUP_ITERS: 1000 WARMUP_METHOD: linear WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: null WEIGHT_DECAY_NORM: 0.0 TEST: AUG: CVPODS_TTA: true ENABLED: false FLIP: true MAX_SIZE: 4000 MIN_SIZES:
  • 400
  • 500
  • 600
  • 640
  • 700
  • 900
  • 1000
  • 1100
  • 1200
  • 1300
  • 1400
  • 1500
  • 1800
  • 800 SCALE_FILTER: true SCALE_RANGES:
  • • 96
    • 10000
  • • 96
    • 10000
  • • 64
    • 10000
  • • 64
    • 10000
  • • 64
    • 10000
  • • 0
    • 10000
  • • 0
    • 10000
  • • 0
    • 256
  • • 0
    • 256
  • • 0
    • 192
  • • 0
    • 192
  • • 0
    • 96
  • • 0
    • 10000 DETECTIONS_PER_IMAGE: 100 EVAL_PERIOD: 3000 EXPECTED_RESULTS: [] KEYPOINT_OKS_SIGMAS: [] PRECISE_BN: ENABLED: false NUM_ITER: 200 VERSION: 2 VIS_PERIOD: 0 '''

And my result is very terrible '''	category	AP	category	AP	category	AP
A220	0.533	A320/321	0.490	A330	0.000
ARJ21	0.102	Boeing737	0.171	Boeing787	0.120
other	0.813

'''