Accuracy reproducibility issue for ssd_mobiledet_dsp_coco from TF1 modelzoo

[debapriyamaji]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [ ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [x] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [x] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/...

2. Describe the bug

I am trying to reproduce the accuracy of ssd_mobiledet_dsp_coco from TF modelzoo. Reported mAP for float model model on coco val2017 is 28.4 as mentioned in pipeleine.config. However, I am getting much higher accuracy of 36.3.

3. Steps to reproduce

In order to reproduce the issue, run the following command:

python object_detection/model_main.py \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
    --checkpoint_dir=${CHECKPOINT_DIR}\
    --run_once

}

The above script recports an mAP of 36.3.

Any other way of evaluation is also fine. Intention is to reproduce the mAP number mentioned in the paper and documentation.

4. Expected behavior

Expected mAP for this model is 28.4 instead of 36.3 as reported by the above scripts.

5. Additional context

Here is a detailed output accuracy log:

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• Mobile device name if the issue happens on a mobile device:
• TensorFlow installed from (source or binary): Binary
• TensorFlow version (use command below): 1.15
• Python version: 3.6.13
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: 10.0
• GPU model and memory: GTX1080TI, 11GB

[kumariko]

@debapriyamaji We see that you are using older version of tensorflow 1.15 which is not actively supported, We recommend that you upgrade to 2.6.0 which is latest stable version of TF and let us know if the issue still persists in newer versions .Thank you!

[debapriyamaji]

@kumariko This model is part of TF1 modelzoo and excluded from TF2 modelzoo, Hence, I had to use TF1.15 to run the evaluation.

I tried running the evaluation in TF2.6 . As expected, I am getting the following error:

ValueError: ssd_mobiledet_dsp is not supported. See model_builder.py for features extractors compatible with different versions of Tensorflow_

Let me know what you think.. Thanks.

[debapriyamaji]

@ymodak @kumariko Any update on this?