Can not reproduce the results of "cityscapes" to "foggy cityscapes".

[cwhgn]

Hi，I get some problems during the reproduction of the results of "cityscapes" to "foggy cityscapes".

During my reproduction, I caught an error as below:

The environment is followed by your prerequisites: detectron=0.3, pytorch=1.7, cuda=11.0, python=3.7. So I changed your code in below to solve the error:

From https://github.com/facebookresearch/adaptive_teacher/blob/cba3c59cadfc9f1a3a676a82bf63d76579ab552b/adapteacher/engine/trainer.py#L167-L168 to evaluator_list.append(COCOEvaluator(dataset_name, cfg, True, output_dir=output_folder))

This error is solved after my modification. Is this modification OK? However, besides this error, another error arise when I run 32979 iterations, shown as below.

I have changed the IMG_PER_BATCH_LABEL/IMG_PER_BATCH_UNLABEL to 8/8. But the error is still there. My GPU is 16G V100 and Memory is 377G. Due to this error, I can not reproduce the results of "cityscapes" to "foggy cityscapes". Do you know how to fix this error?

[yujheli]

I did not get this error. When you look into the class COCOEvaluator, these two parameters are not supposed to be required:

[cwhgn]

Did you use the wrong version of Detectron2? The COCOEvaluator in Detectron2=0.3 is as below:

[cwhgn]

I've changed my Detectron2 from v0.3 to v0.5. The code can be well run. But the performance is much lower than the reported, as below (city to foggy). It's hard for me to find the reason, as I do not changed anything in codes. Do you have any log files for me to find the problem?

[cwhgn]

Here is my config.yml:

CUDNN_BENCHMARK: false DATALOADER: ASPECT_RATIO_GROUPING: true FILTER_EMPTY_ANNOTATIONS: true NUM_WORKERS: 4 RANDOM_DATA_SEED: 0 RANDOM_DATA_SEED_PATH: dataseed/COCO_supervision.txt REPEAT_THRESHOLD: 0.0 SAMPLER_TRAIN: TrainingSampler SUP_PERCENT: 100.0 DATASETS: CROSS_DATASET: true PRECOMPUTED_PROPOSAL_TOPK_TEST: 1000 PRECOMPUTED_PROPOSAL_TOPK_TRAIN: 2000 PROPOSAL_FILES_TEST: [] PROPOSAL_FILES_TRAIN: [] TEST:

• cityscapes_foggy_val TRAIN:
• coco_2017_train TRAIN_LABEL:
• cityscapes_fine_instance_seg_train TRAIN_UNLABEL:
• cityscapes_foggy_train EMAMODEL: SUP_CONSIST: true GLOBAL: HACK: 1.0 INPUT: CROP: ENABLED: false SIZE:
  • 0.9
  • 0.9 TYPE: relative_range FORMAT: BGR MASK_FORMAT: polygon MAX_SIZE_TEST: 1333 MAX_SIZE_TRAIN: 1333 MIN_SIZE_TEST: 800 MIN_SIZE_TRAIN:
• 640
• 672
• 704
• 736
• 768
• 800 MIN_SIZE_TRAIN_SAMPLING: choice RANDOM_FLIP: horizontal MODEL: ANCHOR_GENERATOR: ANGLES:
  • • -90
    • 0
    • 90 ASPECT_RATIOS:
  • • 0.5
    • 1.0
    • 2.0 NAME: DefaultAnchorGenerator OFFSET: 0.0 SIZES:
  • • 32
    • 64
    • 128
    • 256
    • 512 BACKBONE: FREEZE_AT: 2 NAME: build_vgg_backbone DEVICE: cuda FPN: FUSE_TYPE: sum IN_FEATURES: [] NORM: '' OUT_CHANNELS: 256 KEYPOINT_ON: false LOAD_PROPOSALS: false MASK_ON: false META_ARCHITECTURE: DAobjTwoStagePseudoLabGeneralizedRCNN PANOPTIC_FPN: COMBINE: ENABLED: true INSTANCES_CONFIDENCE_THRESH: 0.5 OVERLAP_THRESH: 0.5 STUFF_AREA_LIMIT: 4096 INSTANCE_LOSS_WEIGHT: 1.0 PIXEL_MEAN:
• 103.53
• 116.28
• 123.675 PIXEL_STD:
• 1.0
• 1.0
• 1.0 PROPOSAL_GENERATOR: MIN_SIZE: 0 NAME: PseudoLabRPN RESNETS: DEFORM_MODULATED: false DEFORM_NUM_GROUPS: 1 DEFORM_ON_PER_STAGE:
  • false
  • false
  • false
  • false DEPTH: 101 NORM: FrozenBN NUM_GROUPS: 1 OUT_FEATURES:
  • res4 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: true WIDTH_PER_GROUP: 64 RETINANET: BBOX_REG_LOSS_TYPE: smooth_l1 BBOX_REG_WEIGHTS: &id001
  • 1.0
  • 1.0
  • 1.0
  • 1.0 FOCAL_LOSS_ALPHA: 0.25 FOCAL_LOSS_GAMMA: 2.0 IN_FEATURES:
  • p3
  • p4
  • p5
  • p6
  • p7 IOU_LABELS:
  • 0
  • -1
  • 1 IOU_THRESHOLDS:
  • 0.4
  • 0.5 NMS_THRESH_TEST: 0.5 NORM: '' NUM_CLASSES: 80 NUM_CONVS: 4 PRIOR_PROB: 0.01 SCORE_THRESH_TEST: 0.05 SMOOTH_L1_LOSS_BETA: 0.1 TOPK_CANDIDATES_TEST: 1000 ROI_BOX_CASCADE_HEAD: BBOX_REG_WEIGHTS:
  • • 10.0
    • 10.0
    • 5.0
    • 5.0
  • • 20.0
    • 20.0
    • 10.0
    • 10.0
  • • 30.0
    • 30.0
    • 15.0
    • 15.0 IOUS:
  • 0.5
  • 0.6
  • 0.7 ROI_BOX_HEAD: BBOX_REG_LOSS_TYPE: smooth_l1 BBOX_REG_LOSS_WEIGHT: 1.0 BBOX_REG_WEIGHTS:
  • 10.0
  • 10.0
  • 5.0
  • 5.0 CLS_AGNOSTIC_BBOX_REG: false CONV_DIM: 256 FC_DIM: 1024 NAME: FastRCNNConvFCHead NORM: '' NUM_CONV: 0 NUM_FC: 2 POOLER_RESOLUTION: 7 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 SMOOTH_L1_BETA: 0.0 TRAIN_ON_PRED_BOXES: false ROI_HEADS: BATCH_SIZE_PER_IMAGE: 512 IN_FEATURES:
  • vgg4 IOU_LABELS:
  • 0
  • 1 IOU_THRESHOLDS:
  • 0.5 LOSS: CrossEntropy NAME: StandardROIHeadsPseudoLab NMS_THRESH_TEST: 0.5 NUM_CLASSES: 8 POSITIVE_FRACTION: 0.25 PROPOSAL_APPEND_GT: true SCORE_THRESH_TEST: 0.05 ROI_KEYPOINT_HEAD: CONV_DIMS:
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512
  • 512 LOSS_WEIGHT: 1.0 MIN_KEYPOINTS_PER_IMAGE: 1 NAME: KRCNNConvDeconvUpsampleHead NORMALIZE_LOSS_BY_VISIBLE_KEYPOINTS: true NUM_KEYPOINTS: 17 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 ROI_MASK_HEAD: CLS_AGNOSTIC_MASK: false CONV_DIM: 256 NAME: MaskRCNNConvUpsampleHead NORM: '' NUM_CONV: 0 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_TYPE: ROIAlignV2 RPN: BATCH_SIZE_PER_IMAGE: 256 BBOX_REG_LOSS_TYPE: smooth_l1 BBOX_REG_LOSS_WEIGHT: 1.0 BBOX_REG_WEIGHTS: *id001 BOUNDARY_THRESH: -1 CONV_DIMS:
  • -1 HEAD_NAME: StandardRPNHead IN_FEATURES:
  • vgg4 IOU_LABELS:
  • 0
  • -1
  • 1 IOU_THRESHOLDS:
  • 0.3
  • 0.7 LOSS: CrossEntropy LOSS_WEIGHT: 1.0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOPK_TEST: 1000 POST_NMS_TOPK_TRAIN: 2000 PRE_NMS_TOPK_TEST: 6000 PRE_NMS_TOPK_TRAIN: 12000 SMOOTH_L1_BETA: 0.0 UNSUP_LOSS_WEIGHT: 1.0 SEM_SEG_HEAD: COMMON_STRIDE: 4 CONVS_DIM: 128 IGNORE_VALUE: 255 IN_FEATURES:
  • p2
  • p3
  • p4
  • p5 LOSS_WEIGHT: 1.0 NAME: SemSegFPNHead NORM: GN NUM_CLASSES: 54 WEIGHTS: '' OUTPUT_DIR: output/exp_city SEED: -1 SEMISUPNET: BBOX_THRESHOLD: 0.8 BURN_UP_STEP: 20000 DIS_LOSS_WEIGHT: 0.1 DIS_TYPE: vgg4 EMA_KEEP_RATE: 0.9996 LOSS_WEIGHT_TYPE: standard MLP_DIM: 128 PSEUDO_BBOX_SAMPLE: thresholding SUP_LOSS_WEIGHT: 1.0 TEACHER_UPDATE_ITER: 1 Trainer: ateacher UNSUP_LOSS_WEIGHT: 1.0 SOLVER: AMP: ENABLED: false BASE_LR: 0.04 BIAS_LR_FACTOR: 1.0 CHECKPOINT_PERIOD: 5000 CLIP_GRADIENTS: CLIP_TYPE: value CLIP_VALUE: 1.0 ENABLED: false NORM_TYPE: 2.0 FACTOR_LIST:
• 1
• 1
• 1
• 1
• 1 GAMMA: 0.1 IMG_PER_BATCH_LABEL: 8 IMG_PER_BATCH_UNLABEL: 8 IMS_PER_BATCH: 16 LR_SCHEDULER_NAME: WarmupTwoStageMultiStepLR MAX_ITER: 100000 MOMENTUM: 0.9 NESTEROV: false REFERENCE_WORLD_SIZE: 0 STEPS:
• 60000
• 80000
• 90000
• 360000 WARMUP_FACTOR: 0.001 WARMUP_ITERS: 1000 WARMUP_METHOD: linear WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0.0001 WEIGHT_DECAY_NORM: 0.0 TEST: AUG: ENABLED: false FLIP: true MAX_SIZE: 4000 MIN_SIZES:
  • 400
  • 500
  • 600
  • 700
  • 800
  • 900
  • 1000
  • 1100
  • 1200 DETECTIONS_PER_IMAGE: 100 EVALUATOR: COCOeval EVAL_PERIOD: 10000 EXPECTED_RESULTS: [] KEYPOINT_OKS_SIGMAS: [] PRECISE_BN: ENABLED: false NUM_ITER: 200 VAL_LOSS: true VERSION: 2 VIS_PERIOD: 0

[cwhgn]

By the way, when I run the code on city to foggy, the dataset information is listed as below. Do it the same with yours?

When I run the code on pascal voc to clipart, an error is received. I do not change any code, but set the dataset directory as you required. I get the clipart from (https://github.com/naoto0804/cross-domain-detection/tree/master/datasets), do I need to do some modification before running the code? Do you know how to fix it?

[yujheli]

@cwhgn check if comment this line works for you for the clipart? https://github.com/facebookresearch/adaptive_teacher/blob/main/adapteacher/data/dataset_mapper.py#L83

[cwhgn]

@cwhgn check if comment this line works for you for the clipart? https://github.com/facebookresearch/adaptive_teacher/blob/main/adapteacher/data/dataset_mapper.py#L83

Thanks, it works. By the way, can you give me some help on the low performance problem of city-to-foggy I reproduced above?

[yujheli]

@cwhgn Are you looking at AP50 instead of AP? For AP50 of each class, please find it in the tensorboard.

[cwhgn]

@cwhgn Are you looking at AP50 instead of AP? For AP50 of each class, please find it in the tensorboard.

Thanks, it's a great help for me! Another question, how to fix this problem that arises during my training on pascal voc to clipart.

[yujheli]

@cwhgn see https://github.com/facebookresearch/adaptive_teacher/issues/25

[onkarkris]

I've changed my Detectron2 from v0.3 to v0.5. The code can be well run. But the performance is much lower than the reported, as below (city to foggy). It's hard for me to find the reason, as I do not changed anything in codes. Do you have any log files for me to find the problem?

@cwhgn I am unable to reproduce results on smaller batch size (4) on 4 GPUs, did you try with with less GPUs? I can get around 41 AP@50

[yujheli]

For those who asked to reproduce the foggy cityscapes, I attached the parameters swift:

The best result requires the specific parameter while I'm still figuring out the difficulty.