Reproduced results on COCO val dataset are inferior than official results

[YHDang]

Hello, thanks for your excellent research. I have reproduced CID on the COCO dataset. However, the reproduced results on COCO val dataset are only 66.7% AP (Python 3.8.0, CUDA 11.4). And then I updated the experimental settings with Python 3.9.0, CUDA 11.4. Results are worse. The valid results on COCO are 55.9% AP. Why is there such a gap between the reproduced results and the official results?

[haowei2020]

Hello, thanks for your excellent research. I have reproduced CID on the COCO dataset. However, the reproduced results on COCO val dataset are only 66.7% AP (Python 3.8.0, CUDA 11.4). And then I updated the experimental settings with Python 3.9.0, CUDA 11.4. Results are worse. The valid results on COCO are 55.9% AP. Why is there such a gap between the reproduced results and the official results?

I have the same issue on CrowdPose, test result on CrowdPose is only 60.7% AP while official result is 71.3%.

[uyoung-jeong]

This might happen when you train this model with smaller batch size. Training with 2 gpus, each with 20 batch size will produce similar or slightly higher result.

[yzrs]

I have the same issue on CrowdPose (62.1% AP) and OChuman(47.9% AP) , my log of CrowdPose is as the follows. (Train with 2 gpus,each with 20 batch size). I would appreciate it if you could point out the problem in my settings.

2023-12-15 19:47:19,113 Namespace(cfg='experiments/crowdpose.yaml', dist_url='tcp://127.0.0.1', gpus='0,1', opts=[], port='23459') 2023-12-15 19:47:19,113 AUTO_RESUME: True CFG_NAME: cid-hrnet32 CUDNN: BENCHMARK: True DETERMINISTIC: False ENABLED: True DATASET: DATASET: crowdpose FILTER_IMAGE: False FLIP: 0.5 FLIP_INDEX: [1, 0, 3, 2, 5, 4, 7, 6, 9, 8, 11, 10, 12, 13] INPUT_SIZE: 512 MAX_INSTANCES: 200 MAX_ROTATION: 30 MAX_SCALE: 1.5 MAX_TRANSLATE: 40 MIN_SCALE: 0.75 NUM_KEYPOINTS: 14 OUTPUT_SIZE: 128 ROOT: ../dataset/crowdpose SCALE_TYPE: short SIGMA: 2.0 TEST: test TRAIN: trainval DDP: True DIST_BACKEND: nccl LOSS: CONTRASTIVE_LOSS_WEIGHT: 1.0 MULTI_HEATMAP_LOSS_WEIGHT: 1.0 SINGLE_HEATMAP_LOSS_WEIGHT: 4.0 MODEL: BACKBONE: PRETRAINED_LAYERS: ['*'] STAGES: BLOCK: ['BASIC', 'BASIC', 'BASIC'] FUSE_METHOD: ['SUM', 'SUM', 'SUM'] NUM_BLOCKS: [[4, 4], [4, 4, 4], [4, 4, 4, 4]] NUM_BRANCHES: [2, 3, 4] NUM_CHANNELS: [[32, 64], [32, 64, 128], [32, 64, 128, 256]] NUM_MODULES: [1, 4, 3] NUM_STAGES: 3 BIAS_PROB: 0.01 DEVICE: cuda GFD: CHANNELS: 32 IN_CHANNELS: 480 OUT_CHANNELS: 14 IIA: IN_CHANNELS: 480 OUT_CHANNELS: 15 INIT_WEIGHTS: True NAME: CID PRETRAINED: model/imagenet/hrnet_w32-36af842e.pth SYNC_BN: False OUTPUT_DIR: runs PIN_MEMORY: True PRINT_FREQ: 20 TEST: CENTER_POOL_KERNEL: 3 FLIP_TEST: True IMAGES_PER_GPU: 1 KEYPOINT_THRESHOLD: 0.01 MAX_PROPOSALS: 30 MODEL_FILE: OKS_SCORE: 0.8 OKS_SIGMAS: [0.79, 0.79, 0.72, 0.72, 0.62, 0.62, 1.07, 1.07, 0.87, 0.87, 0.89, 0.89, 0.79, 0.79] POOL_THRESHOLD1: 300 POOL_THRESHOLD2: 200 TRAIN: BEGIN_EPOCH: 0 CHECKPOINT: END_EPOCH: 300 GAMMA1: 0.99 GAMMA2: 0.0 IMAGES_PER_GPU: 20 LR: 0.001 LR_FACTOR: 0.1 LR_SCHEDULER: step LR_STEP: [200, 260] MOMENTUM: 0.9 NESTEROV: False OPTIMIZER: adam RESUME: False SHUFFLE: True WD: 0.0001 VERBOSE: False WORKERS: 4 2023-12-15 19:47:19,346 Added key: store_based_barrier_key:1 to store for rank: 0 2023-12-15 19:47:19,357 Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2023-12-15 19:47:19,657 => init weights from normal distribution 2023-12-15 19:47:20,119 => loading pretrained model model/imagenet/hrnet_w32-36af842e.pth 2023-12-15 19:47:22,408 Dataset PoseDataset Number of datapoints: 12000 Root Location: ../dataset/crowdpose 2023-12-15 19:47:37,898 Epoch: [0][0/300] Time: 15.362s (15.362s) Speed: 1.3 samples/s Data: 11.070s (11.070s) multiple: 4.522e+00 (4.522e+00) single: 1.813e+01 (1.813e+01) contrast: 1.888e-02 (1.888e-02) ......