Low mAP due to wrong num_gt

[Jinbang129]

I run the tensorflow 2.3 Object Detection API on my own dataset, but the mAP is very low. Since I have dense objects on the images, I changed the (max_number_of_boxes) here to 600. I used the evaluation protocol of pascal_voc_detection_metrics.

After running the _model_maintf2.py with _checkpointdir for evaluation on 30 images, we can find the printed num_gt, precision and recall from here are weird as showed below.

The max_number_of_boxes is 600 as shown in _eval_inputreader (in Configuration file) below for each images. Thus, I guess the num_gt=18000 is actuallly the sum for 30 images. And when I change the max_number_of_boxes, the num_gt will change, correspondingly!

In additon, the result for the same dataset in tensorflow 1.13 seems correct. num-gt=6474, which is right.

How can I get the right num_gt? Please do me a favor. Many thanks!

[Jinbang129]

Configuration file: model { faster_rcnn { num_classes: 1 image_resizer { fixed_shape_resizer { width: 1024 height: 1024 } } feature_extractor { type: 'faster_rcnn_resnet101_keras' batch_norm_trainable: true } first_stage_anchor_generator { grid_anchor_generator { height: 32 width: 32 scales: [0.25, 0.5, 1.0, 2.0] aspect_ratios: [0.5, 1.0, 2.0] height_stride: 16 width_stride: 16 } } first_stage_box_predictor_conv_hyperparams { op: CONV regularizer { l2_regularizer { weight: 0.0 } } initializer { truncated_normal_initializer { stddev: 0.01 } } } first_stage_nms_score_threshold: 0.0 first_stage_nms_iou_threshold: 0.7 first_stage_max_proposals: 600 first_stage_localization_loss_weight: 2.0 first_stage_objectness_loss_weight: 1.0 initial_crop_size: 14 maxpool_kernel_size: 2 maxpool_stride: 2 second_stage_box_predictor { mask_rcnn_box_predictor { use_dropout: false dropout_keep_probability: 1.0 fc_hyperparams { op: FC regularizer { l2_regularizer { weight: 0.0 } } initializer { variance_scaling_initializer { factor: 1.0 uniform: true mode: FAN_AVG } } } share_box_across_classes: true } } second_stage_post_processing { batch_non_max_suppression { score_threshold: 0.0 iou_threshold: 0.6 max_detections_per_class: 400 max_total_detections: 12000 } score_converter: SOFTMAX } second_stage_localization_loss_weight: 2.0 second_stage_classification_loss_weight: 1.0 use_static_shapes: true use_matmul_crop_and_resize: true clip_anchors_to_image: true use_static_balanced_label_sampler: true use_matmul_gather_in_matcher: true } }

train_config: { batch_size: 1 sync_replicas: true startup_delay_steps: 0 replicas_to_aggregate: 8 num_steps: 100000 optimizer { momentum_optimizer: { learning_rate: { cosine_decay_learning_rate { learning_rate_base: .04 total_steps: 100000 warmup_learning_rate: .013333 warmup_steps: 2000 } } momentum_optimizer_value: 0.9 } use_moving_average: false } fine_tune_checkpoint_version: V2 fine_tune_checkpoint: "D:/tf2/workspace/pretrained models/faster_rcnn_resnet101_v1_1024x1024_coco17_tpu-8/checkpoint/ckpt-0" fine_tune_checkpoint_type: "detection" data_augmentation_options { random_horizontal_flip { } }

data_augmentation_options { random_adjust_hue { } }

data_augmentation_options { random_adjust_contrast { } }

data_augmentation_options { random_adjust_saturation { } }

data_augmentation_options { random_square_crop_by_scale { scale_min: 0.6 scale_max: 1.3 } } max_number_of_boxes: 300 unpad_groundtruth_tensors: false use_bfloat16: false # works only on TPUs } train_input_reader: { label_map_path: "D:/tf2/workspace/images/labelmap.pbtxt" tf_record_input_reader { input_path: "D:/tf2/workspace/images/train.record" }

max_number_of_boxes: 2000

}

eval_config: { metrics_set: "pascal_voc_detection_metrics" use_moving_averages: false batch_size: 1 num_visualizations: 30 min_score_threshold: 0.4 max_num_boxes_to_visualize: 300 visualize_groundtruth_boxes: True groundtruth_box_visualization_color: "Blue" ignore_groundtruth: false visualization_export_dir: "D:/tf2/workspace/images/eval_result/frcnn/" skip_scores: False skip_labels: True keep_image_id_for_visualization_export: True }

eval_input_reader: { label_map_path: "D:/tf2/workspace/images/labelmap.pbtxt" shuffle: false num_epochs: 1 max_number_of_boxes: 600 tf_record_input_reader { input_path: "D:/tf2/workspace/images/test.record" } }

[Jinbang129]

I reckon this problem is because the groundtruth number is too big.

[SaketChaturvedi]

Hi @Jinbang129,

I am also experiencing similar problems with mAP. My dataset has very dense objects; I got a low mAP score of 38% when I performed training using the FasterRCNN model on my dataset. However, the model's prediction on evaluation dataset sample images looks better than the evaluation dataset mAP score. The model is catching almost every object in the images correctly. I suspect the evaluation metrics or parameters in the config file might be required to change to fit my problem. I am using coco_evaluation_metrics for evaluation. How did you solve your low mAP problem? Can you suggest what changes can be made in the parameter values for the dense object dataset to get a better mAP score?

Thank you so much for your help!