TensorFlow object detection API evaluate training performance on both training data and validation data

[Mazharul-Hossain]

System information

• What is the top-level directory of the model you are using: object detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04.3 LTS
• TensorFlow installed from (source or binary): from source
• TensorFlow version (use command below): 'v1.12.2-0-g6b634657d8' 1.12.2
• Bazel version (if compiling from source): 0.17.2
• CUDA/cuDNN version: cuda 10.0 / cuDnn 7.3
• GPU model and memory: GeForce RTX 2080 Ti Rev. A (11 GB)
• Exact command to reproduce:

  python model_main.py --alsologtostderr \
  --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
  --model_dir=${TRAIN_DIR} \
  --num_train_steps=24000 \
  --sample_1_of_n_eval_on_train_examples=25 \
  --num_eval_steps=100 \
  --sample_1_of_n_eval_examples=1 \
  --eval_training_data=True

Describe the problem

To identify overfitting, I need both performances on training data and validation data. I am using my data. And my mask_rcnn_resnet101_atrous_coco NN is not performing well on the validation dataset. So, I wanted to know the performance of NN on training data and validation data during a training session.

I found similar problems on StackOverflow but no solution.

1. how to check both training/eval performances in TensorFlow object_detection
2. How to calculate evaluation metrics on training data in TensorFlow's Object Detection API? And I also post a question TensorFlow object detection API evaluate training performance

I used --eval_training_data=True as a parameter but the run performed worse than run without it. However, --eval_training_data=True suppose to run evaluation on the training dataset and should have a better precision (mAP) result than the validation dataset. I could not find any option to run the calculation on both datasets at the same time and report them separately during a training session.

Source code / logs

Evaluation result with --eval_training_data=True included:

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.165 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.281 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.167 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.051 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.109 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.202 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.164 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.202 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.202 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.057 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.141 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.236

Evaluation result without --eval_training_data :

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.168 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.283 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.173 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.049 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.108 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.208 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.170 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.208 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.208 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.056 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.139 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.248

How to run the script to compute accuracy (mAP) both on the validation set and the training set (randomly some percentage) and report them separately during a training session?

[marcoruizrueda]

Did you find any script for that?

[tispratik]

Facing same issue :(

[Mazharul-Hossain]

I asked the same question on Stackoverflow I got an answer there. I have tried it once, however, I can not ensure it works perfectly.

[Krishna2709]

@tispratik @Mazharul-Hossain Can anyone of you share your code of evaluating the TensorFlow object detection model on Validation set? Thank you :-)

[rabiaathar]

@tispratik @Mazharul-Hossain Can anyone of you share your code of evaluating the TensorFlow object detection model on Validation set? Thank you :-)

@Krishna2709 did you find anything?

[kabyanil]

any update on this?

[Petros626]

@Mazharul-Hossain why in my case with model_main.py the test frames are evaluated I don't get it. I use the following commands:

set CONFIG_FILE=C:\Users\petros.katsoulakos\models2\models-master\research\object_detection\training\ssd_mobilenet_v2_quantized_300x300_coco_custom_aspect_ratios.config set OUTPUT_DIR=C:\Users\petros.katsoulakos\models2\models-master\research\object_detection\tensorboard_outputs\after_training\eval_train_data set CHECKPOINT_PATH= C:\Users\petros.katsoulakos\models2\models-master\research\object_detection\training\model.ckpt-200000

python model_main.py --pipeline_config_path=%CONFIG_FILE% --model_dir=%OUTPUT_DIR% --eval_training_data=True --checkpoint_dir=%CHECKPOINT_PATH% --run_once=True

Must I change the eval_input_reader in the config file to train.record instead of test.record? Furthermore the command doesn't recognize, that I already have model checkpoints and start from beginning every time to train....