SSD mobilenet checkpoints are broken

[ShawnDing1994]

• What is the top-level directory of the model you are using: research/
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): NO
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.6 (pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.6.0-cp27-none-linux_x86_64.whl)
• Bazel version (if compiling from source):
• CUDA/cuDNN version: CUDA9.0 cuDNN7.0.5
• GPU model and memory: 1080Ti, 11GB
• Exact command to reproduce: python object_detection/train.py \ --logtostderr \ --pipeline_config_path=/home/dingxiaohan/rda/ssd_mobilenet_v2_coco.config \ --train_dir=p2_debug

python object_detection/eval.py \ --logtostderr \ --pipeline_config_path=/home/dingxiaohan/rda/ssd_mobilenet_v2_coco.config \ --checkpoint_dir=p2_debug \ --eval_dir=p2_debug_eval

Describe the problem

I am trying to train the pre-trained SSD and faster rcnn models on COCO. But when the training of SSD-mobilenet-v1 began, the loss was initially above 300 and decreased drastically. When the loss became stable (around 5) after a few minutes, the visualization of the detections made no sense, as the image is filled up with big boxes of all classes. After ten hours of training, the predictions started to make some sense. I think this suggests that the checkpoints do not work on COCO. I am sure that the checkpoints are loaded by the codes, as the parameters do not look like randomly initialized (I ran codes like [print(np.sum(sess.run(SOME_CONV_KERNEL_TENSORS)))] and got values with large magnitude (100 ~ 3000)). SSD-mobilenet-v2 behaved similarly, where the initial loss was around 280. The SSD-mobilenet models were downloaded from the model zoo (http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2017_11_17.tar.gz, http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz). Faster-rcnn-resnet101 works fine for me. Only the path-related lines in the config files were modified. Switching py27 to py36 made no difference.

[tensorflowbutler]

Hi There, We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing. If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.