search

youngwanLEE / vovnet-detectron2

[CVPR 2020] VoVNet backbone networks for detectron2
Other
374 stars 69 forks source link

Inconsistent evaluation results for same model and dataset #21

Open ppurwar opened 3 years ago

ppurwar commented 3 years ago

Command: python train_net.py --num-gpus 1 --config-file configs/faster_rcnn_V_39_FPN_3x.yaml --resume --eval-only MODEL.WEIGHTS checkpoints/FRCN-V2-39-3x_1/model_final.pth MODEL.ROI_HEADS.SCORE_THRESH_TEST 0.5

The content of config file: BASE: "Base-RCNN-VoVNet-FPN.yaml" MODEL: WEIGHTS: "https://www.dropbox.com/s/q98pypf96rhtd8y/vovnet39_ese_detectron2.pth?dl=1" MASK_ON: False VOVNET: CONV_BODY: "V-39-eSE" ROI_HEADS: NUM_CLASSES: 30 SOLVER: STEPS: (210000, 250000) MAX_ITER: 200000 IMS_PER_BATCH: 8 BASE_LR: 0.0001 CHECKPOINT_PERIOD: 1000 DATASETS: TRAIN: ("train_dataset_leafi",) TEST: ("train_dataset_leafi","val_dataset_leafi") OUTPUT_DIR: "checkpoints/FRCN-V2-39-3x_crops/" DATALOADER: NUM_WORKERS: 4 TEST: EVAL_PERIOD: 1000

The evaluation results (for validation dataset) running the command:

[06/17 10:07:33 d2.evaluation.coco_evaluation]: 'val_dataset_leafi' is not registered by register_coco_instances. Therefore trying to convert it to COCO format ... [06/17 10:07:33 d2.evaluation.evaluator]: Start inference on 66 images [06/17 10:07:35 d2.evaluation.fast_eval_api]: Evaluate annotation type bbox [06/17 10:07:35 d2.evaluation.fast_eval_api]: COCOeval_opt.evaluate() finished in 0.01 seconds. [06/17 10:07:35 d2.evaluation.fast_eval_api]: Accumulating evaluation results... [06/17 10:07:35 d2.evaluation.fast_eval_api]: COCOeval_opt.accumulate() finished in 0.04 seconds. Average Precision (AP) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.086 Average Precision (AP) @[ IoU=0.50 area= all maxDets=100 ] = 0.209 Average Precision (AP) @[ IoU=0.75 area= all maxDets=100 ] = 0.052 Average Precision (AP) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.102 Average Precision (AP) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.070 Average Precision (AP) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.137 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 1 ] = 0.093 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 10 ] = 0.093 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.093 Average Recall (AR) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.104 Average Recall (AR) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.085 Average Recall (AR) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.144 [06/17 10:07:35 d2.evaluation.coco_evaluation]: Evaluation results for bbox: AP AP50 AP75 APs APm APl AR1 AR10 AR100 ARs ARm ARl
8.572 20.874 5.198 10.182 6.964 13.746 9.268 9.326 9.326 10.449 8.500 14.444
[06/17 10:11:20 d2.evaluation.coco_evaluation]: 'val_dataset_leafi' is not registered by register_coco_instances. Therefore trying to convert it to COCO format ... WARNING [06/17 10:11:20 d2.data.datasets.coco]: Using previously cached COCO format annotations at 'checkpoints/FRCN-V2-39-3x_crops/inference/val_dataset_leafi_coco_format.json'. You need to clear the cache file if your dataset has been modified. [06/17 10:11:20 d2.evaluation.evaluator]: Start inference on 66 images Average Precision (AP) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.094 Average Precision (AP) @[ IoU=0.50 area= all maxDets=100 ] = 0.242 Average Precision (AP) @[ IoU=0.75 area= all maxDets=100 ] = 0.043 Average Precision (AP) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.110 Average Precision (AP) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.196 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 1 ] = 0.116 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 10 ] = 0.116 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.116 Average Recall (AR) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.004 Average Recall (AR) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.142 Average Recall (AR) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.233 [06/17 10:11:22 d2.evaluation.coco_evaluation]: Evaluation results for bbox: AP AP50 AP75 APs APm APl AR1 AR10 AR100 ARs ARm ARl
9.403 24.244 4.297 0.042 10.984 19.574 11.642 11.642 11.642 0.353 14.227 23.333

This seems to be issue similar to https://github.com/facebookresearch/detectron2/issues/739.