Detection - ShapeMask TensorRT models cannot predict DetectionMasks

nss-ys commented 4 years ago

Hi community,

I am trying to optimize a retrained ShapeMask model for inference with export_tensorrt_model.py (a nice addition to tpu/detection toolkit!). The transformed model returns e.g. num_detections, detection_scores,... as expected:

> ret = sess.run(["NumDetections:0", "DetectionScores:0", "DetectionBoxes:0"], feed_dict={'Placeholder:0': image})
> ret[0]
array([3], dtype=int32)
> ret[2].shape
(1, 100, 4)

But it throws an error when it calculates detection_masks:

> ret = sess.run(["DetectionMasks:0"], feed_dict={'Placeholder:0': image})

...
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: Input to reshape is a tensor with 10240000 values, but the requested shape has 102400
         [[{{node fine_mask/dense/Tensordot/Reshape}}]]
         [[DetectionMasks/_53]]
  (1) Invalid argument: Input to reshape is a tensor with 10240000 values, but the requested shape has 102400
         [[{{node fine_mask/dense/Tensordot/Reshape}}]]
0 successful operations.
0 derived errors ignored.

I believe conversion to TensorRT causes this, since the savedmodel from which I made TRT optimization returns predicted DetectionMasks without problem.

> # SavedModel before TRT optimization
> ret = sess.run(["NumDetections:0", "DetectionScores:0", "DetectionBoxes:0", "DetectionMasks:0"], feed_dict={'Placeholder:0': image})
> ret[3].shape
(1, 100, 256, 256)

Any ideas on how this happens?

Here is the procedure I followed to convert the model:

Pull NVIDIA-TensorFlow docker image (nvcr.io/nvidia/tensorflow:20.03-tf1-py3) and create an container.


$ docker container run --gpus all -itdv /blah:/blah nvcr.io/nvidia/tensorflow:20.03-tf1-py3 --name TRT2003_TF1
$ docker container exec -it TRT2003_TF1 bash
$ nvidia-smi
Wed Jul 29 08:59:19 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN V             Off  | 00000000:17:00.0 Off |                  N/A |
| 32%   48C    P0    47W / 250W |    227MiB / 12066MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN V             Off  | 00000000:65:00.0 Off |                  N/A |
| 61%   82C    P2   150W / 250W |  11932MiB / 12066MiB |     86%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| +-----------------------------------------------------------------------------+


2. Convert the ShapeMask checkpoint to savedmodel with `export_saved_model.py`. (My model is trained on 256,256)

$ python3 ./export_saved_model.py \ --export_dir ./theModel \ --checkpoint_path /the/model.ckpt \ --model shapemask \ --input_type raw_image_tensor \ --config_file /the/shapemask_config.yaml \ --input_image_size 256,256


3. Optimize the SavedModel with `export_tensorrt_model.py`

$ python3 export_tensorrt_model.py \ --saved_model_dir ./theModel \ --output_dir ./theModel/RT


4. Run prediction

$ python3

import tensorflow.compat.v1 as tf, numpy as np from tensorflow.python.compiler.tensorrt import trt_convert as trt

nongreedy = tf.ConfigProto() nongreedy.gpu_options.allow_growth = True sess = tf.Session(config=nongreedy) model = tf.saved_model.load(sess, [tf.saved_model.SERVING], './theModel/RT') image = np.random.uniform(size=(1,256,256,3)) ret = sess.run(["DetectionMasks:0"], feed_dict={'Placeholder:0': image}) # error

vaghelaviral commented 4 years ago

ValueError: Could not load all requested variables from checkpoint. Please make sure your model_fn does not expect variables that were not saved in the checkpoint.

Encountered error with mode infer while restoring checkpoint from: /home/deploy/Viral/potato_shapemask/model_output/ctl_step_100000.ckpt-21. Full Traceback:

Existing variables not in the checkpoint: coarse_mask/class-0-bn/beta, coarse_mask/class-0-bn/gamma, coarse_mask/class-0-bn/moving_mean, coarse_mask/class-0-bn/moving_variance, coarse_mask/class-0/bias, coarse_mask/class-0/kernel, coarse_mask/class-1-bn/beta, coarse_mask/class-1-bn/gamma, coarse_mask/class-1-bn/moving_mean, coarse_mask/class-1-bn/moving_variance, coarse_mask/class-1/bias, coarse_mask/class-1/kernel, coarse_mask/class-2-bn/beta, coarse_mask/class-2-bn/gamma, coarse_mask/class-2-bn/moving_mean, coarse_mask/class-2-bn/moving_variance, coarse_mask/class-2/bias, coarse_mask/class-2/kernel, coarse_mask/class-3-bn/beta, coarse_mask/class-3-bn/gamma, coarse_mask/class-3-bn/moving_mean, coarse_mask/class-3-bn/moving_variance, coarse_mask/class-3/bias, coarse_mask/class-3/kernel, coarse_mask/class-predict/bias, coarse_mask/class-predict/kernel, coarse_mask/dense/bias, coarse_mask/dense/kernel, fine_mask/class-0-bn/beta, fine_mask/class-0-bn/gamma, fine_mask/class-0-bn/moving_mean, fine_mask/class-0-bn/moving_variance, fine_mask/class-0/bias, fine_mask/class-0/kernel, fine_mask/class-1-bn/beta, fine_mask/class-1-bn/gamma, fine_mask/class-1-bn/moving_mean, fine_mask/class-1-bn/moving_variance, fine_mask/class-1/bias, fine_mask/class-1/kernel, fine_mask/class-2-bn/beta, fine_mask/class-2-bn/gamma, fine_mask/class-2-bn/moving_mean, fine_mask/class-2-bn/moving_variance, fine_mask/class-2/bias, fine_mask/class-2/kernel, fine_mask/class-3-bn/beta, fine_mask/class-3-bn/gamma, fine_mask/class-3-bn/moving_mean, fine_mask/class-3-bn/moving_variance, fine_mask/class-3/bias, fine_mask/class-3/kernel, fine_mask/class-predict/bias, fine_mask/class-predict/kernel, fine_mask/conv2d_transpose/bias, fine_mask/conv2d_transpose/kernel, fine_mask/dense/bias, fine_mask/dense/kernel, global_step, prior_mask/dense_1/bias, prior_mask/dense_1/kernel, retinanet/box_net/box-0-3/beta, retinanet/box_net/box-0-3/gamma, retinanet/box_net/box-0-3/moving_mean, retinanet/box_net/box-0-3/moving_variance, retinanet/box_net/box-0-4/beta, retinanet/box_net/box-0-4/gamma, retinanet/box_net/box-0-4/moving_mean, retinanet/box_net/box-0-4/moving_variance, retinanet/box_net/box-0-5/beta, retinanet/box_net/box-0-5/gamma, retinanet/box_net/box-0-5/moving_mean, retinanet/box_net/box-0-5/moving_variance, retinanet/box_net/box-0-6/beta, retinanet/box_net/box-0-6/gamma, retinanet/box_net/box-0-6/moving_mean, retinanet/box_net/box-0-6/moving_variance, retinanet/box_net/box-0-7/beta, retinanet/box_net/box-0-7/gamma, retinanet/box_net/box-0-7/moving_mean, retinanet/box_net/box-0-7/moving_variance, retinanet/box_net/box-0/bias, retinanet/box_net/box-0/kernel, retinanet/box_net/box-1-3/beta, retinanet/box_net/box-1-3/gamma, retinanet/box_net/box-1-3/moving_mean, retinanet/box_net/box-1-3/moving_variance, retinanet/box_net/box-1-4/beta, retinanet/box_net/box-1-4/gamma, retinanet/box_net/box-1-4/moving_mean, retinanet/box_net/box-1-4/moving_variance, retinanet/box_net/box-1-5/beta, retinanet/box_net/box-1-5/gamma, retinanet/box_net/box-1-5/moving_mean, retinanet/box_net/box-1-5/moving_variance, retinanet/box_net/box-1-6/beta, retinanet/box_net/box-1-6/gamma, retinanet/box_net/box-1-6/moving_mean, retinanet/box_net/box-1-6/moving_variance, retinanet/box_net/box-1-7/beta, retinanet/box_net/box-1-7/gamma, retinanet/box_net/box-1-7/moving_mean, retinanet/box_net/box-1-7/moving_variance, retinanet/box_net/box-1/bias, retinanet/box_net/box-1/kernel, retinanet/box_net/box-2-3/beta, retinanet/box_net/box-2-3/gamma, retinanet/box_net/box-2-3/moving_mean, retinanet/box_net/box-2-3/moving_variance, retinanet/box_net/box-2-4/beta, retinanet/box_net/box-2-4/gamma, retinanet/box_net/box-2-4/moving_mean, retinanet/box_net/box-2-4/moving_variance, retinanet/box_net/box-2-5/beta, retinanet/box_net/box-2-5/gamma, retinanet/box_net/box-2-5/moving_mean, retinanet/box_net/box-2-5/moving_variance, retinanet/box_net/box-2-6/beta, retinanet/box_net/box-2-6/gamma, retinanet/box_net/box-2-6/moving_mean, retinanet/box_net/box-2-6/moving_variance, retinanet/box_net/box-2-7/beta, retinanet/box_net/box-2-7/gamma, retinanet/box_net/box-2-7/moving_mean, retinanet/box_net/box-2-7/moving_variance, retinanet/box_net/box-2/bias, retinanet/box_net/box-2/kernel, retinanet/box_net/box-3-3/beta, retinanet/box_net/box-3-3/gamma, retinanet/box_net/box-3-3/moving_mean, retinanet/box_net/box-3-3/moving_variance, retinanet/box_net/box-3-4/beta, retinanet/box_net/box-3-4/gamma, retinanet/box_net/box-3-4/moving_mean, retinanet/box_net/box-3-4/moving_variance, retinanet/box_net/box-3-5/beta, retinanet/box_net/box-3-5/gamma, retinanet/box_net/box-3-5/moving_mean, retinanet/box_net/box-3-5/moving_variance, retinanet/box_net/box-3-6/beta, retinanet/box_net/box-3-6/gamma, retinanet/box_net/box-3-6/moving_mean, retinanet/box_net/box-3-6/moving_variance, retinanet/box_net/box-3-7/beta, retinanet/box_net/box-3-7/gamma, retinanet/box_net/box-3-7/moving_mean, retinanet/box_net/box-3-7/moving_variance, retinanet/box_net/box-3/bias, retinanet/box_net/box-3/kernel, retinanet/box_net/box-predict/bias, retinanet/box_net/box-predict/kernel, retinanet/class_net/class-0-3/beta, retinanet/class_net/class-0-3/gamma, retinanet/class_net/class-0-3/moving_mean, retinanet/class_net/class-0-3/moving_variance, retinanet/class_net/class-0-4/beta, retinanet/class_net/class-0-4/gamma, retinanet/class_net/class-0-4/moving_mean, retinanet/class_net/class-0-4/moving_variance, retinanet/class_net/class-0-5/beta, retinanet/class_net/class-0-5/gamma, retinanet/class_net/class-0-5/moving_mean, retinanet/class_net/class-0-5/moving_variance, retinanet/class_net/class-0-6/beta, retinanet/class_net/class-0-6/gamma, retinanet/class_net/class-0-6/moving_mean, retinanet/class_net/class-0-6/moving_variance, retinanet/class_net/class-0-7/beta, retinanet/class_net/class-0-7/gamma, retinanet/class_net/class-0-7/moving_mean, retinanet/class_net/class-0-7/moving_variance, retinanet/class_net/class-0/bias, retinanet/class_net/class-0/kernel, retinanet/class_net/class-1-3/beta, retinanet/class_net/class-1-3/gamma, retinanet/class_net/class-1-3/moving_mean, retinanet/class_net/class-1-3/moving_variance, retinanet/class_net/class-1-4/beta, retinanet/class_net/class-1-4/gamma, retinanet/class_net/class-1-4/moving_mean, retinanet/class_net/class-1-4/moving_variance, retinanet/class_net/class-1-5/beta, retinanet/class_net/class-1-5/gamma, retinanet/class_net/class-1-5/moving_mean, retinanet/class_net/class-1-5/moving_variance, retinanet/class_net/class-1-6/beta, retinanet/class_net/class-1-6/gamma, retinanet/class_net/class-1-6/moving_mean, retinanet/class_net/class-1-6/moving_variance, retinanet/class_net/class-1-7/beta, retinanet/class_net/class-1-7/gamma, retinanet/class_net/class-1-7/moving_mean, retinanet/class_net/class-1-7/moving_variance, retinanet/class_net/class-1/bias, retinanet/class_net/class-1/kernel, retinanet/class_net/class-2-3/beta, retinanet/class_net/class-2-3/gamma, retinanet/class_net/class-2-3/moving_mean, retinanet/class_net/class-2-3/moving_variance, retinanet/class_net/class-2-4/beta, retinanet/class_net/class-2-4/gamma, retinanet/class_net/class-2-4/moving_mean, retinanet/class_net/class-2-4/moving_variance, retinanet/class_net/class-2-5/beta, retinanet/class_net/class-2-5/gamma, retinanet/class_net/class-2-5/moving_mean, retinanet/class_net/class-2-5/moving_variance, retinanet/class_net/class-2-6/beta, retinanet/class_net/class-2-6/gamma, retinanet/class_net/class-2-6/moving_mean, retinanet/class_net/class-2-6/moving_variance, retinanet/class_net/class-2-7/beta, retinanet/class_net/class-2-7/gamma, retinanet/class_net/class-2-7/moving_mean, retinanet/class_net/class-2-7/moving_variance, retinanet/class_net/class-2/bias, retinanet/class_net/class-2/kernel, retinanet/class_net/class-3-3/beta, retinanet/class_net/class-3-3/gamma, retinanet/class_net/class-3-3/moving_mean, retinanet/class_net/class-3-3/moving_variance, retinanet/class_net/class-3-4/beta, retinanet/class_net/class-3-4/gamma, retinanet/class_net/class-3-4/moving_mean, retinanet/class_net/class-3-4/moving_variance, retinanet/class_net/class-3-5/beta, retinanet/class_net/class-3-5/gamma, retinanet/class_net/class-3-5/moving_mean, retinanet/class_net/class-3-5/moving_variance, retinanet/class_net/class-3-6/beta, retinanet/class_net/class-3-6/gamma, retinanet/class_net/class-3-6/moving_mean, retinanet/class_net/class-3-6/moving_variance, retinanet/class_net/class-3-7/beta, retinanet/class_net/class-3-7/gamma, retinanet/class_net/class-3-7/moving_mean, retinanet/class_net/class-3-7/moving_variance, retinanet/class_net/class-3/bias, retinanet/class_net/class-3/kernel, retinanet/class_net/class-predict/bias, retinanet/class_net/class-predict/kernel

Variables names when this checkpoint was written which don't exist now: SGD/box_net/box-0-3/beta/momentum, SGD/box_net/box-0-3/gamma/momentum, SGD/box_net/box-0-4/beta/momentum, SGD/box_net/box-0-4/gamma/momentum, SGD/box_net/box-0-5/beta/momentum, SGD/box_net/box-0-5/gamma/momentum, SGD/box_net/box-0-6/beta/momentum, SGD/box_net/box-0-6/gamma/momentum, SGD/box_net/box-0-7/beta/momentum, SGD/box_net/box-0-7/gamma/momentum, SGD/box_net/box-0/bias/momentum, SGD/box_net/box-0/kernel/momentum, SGD/box_net/box-1-3/beta/momentum, SGD/box_net/box-1-3/gamma/momentum, SGD/box_net/box-1-4/beta/momentum, SGD/box_net/box-1-4/gamma/momentum, SGD/box_net/box-1-5/beta/momentum, SGD/box_net/box-1-5/gamma/momentum, SGD/box_net/box-1-6/beta/momentum, SGD/box_net/box-1-6/gamma/momentum, SGD/box_net/box-1-7/beta/momentum, SGD/box_net/box-1-7/gamma/momentum, SGD/box_net/box-1/bias/momentum, SGD/box_net/box-1/kernel/momentum, SGD/box_net/box-2-3/beta/momentum, SGD/box_net/box-2-3/gamma/momentum, SGD/box_net/box-2-4/beta/momentum, SGD/box_net/box-2-4/gamma/momentum, SGD/box_net/box-2-5/beta/momentum, SGD/box_net/box-2-5/gamma/momentum, SGD/box_net/box-2-6/beta/momentum, SGD/box_net/box-2-6/gamma/momentum, SGD/box_net/box-2-7/beta/momentum, SGD/box_net/box-2-7/gamma/momentum, SGD/box_net/box-2/bias/momentum, SGD/box_net/box-2/kernel/momentum, SGD/box_net/box-3-3/beta/momentum, SGD/box_net/box-3-3/gamma/momentum, SGD/box_net/box-3-4/beta/momentum, SGD/box_net/box-3-4/gamma/momentum, SGD/box_net/box-3-5/beta/momentum, SGD/box_net/box-3-5/gamma/momentum, SGD/box_net/box-3-6/beta/momentum, SGD/box_net/box-3-6/gamma/momentum, SGD/box_net/box-3-7/beta/momentum, SGD/box_net/box-3-7/gamma/momentum, SGD/box_net/box-3/bias/momentum, SGD/box_net/box-3/kernel/momentum, SGD/box_net/box-predict/bias/momentum, SGD/box_net/box-predict/kernel/momentum, SGD/class_net/class-0-3/beta/momentum, SGD/class_net/class-0-3/gamma/momentum, SGD/class_net/class-0-4/beta/momentum, SGD/class_net/class-0-4/gamma/momentum, SGD/class_net/class-0-5/beta/momentum, SGD/class_net/class-0-5/gamma/momentum, SGD/class_net/class-0-6/beta/momentum, SGD/class_net/class-0-6/gamma/momentum, SGD/class_net/class-0-7/beta/momentum, SGD/class_net/class-0-7/gamma/momentum, SGD/class_net/class-0/bias/momentum, SGD/class_net/class-0/kernel/momentum, SGD/class_net/class-1-3/beta/momentum, SGD/class_net/class-1-3/gamma/momentum, SGD/class_net/class-1-4/beta/momentum, SGD/class_net/class-1-4/gamma/momentum, SGD/class_net/class-1-5/beta/momentum, SGD/class_net/class-1-5/gamma/momentum, SGD/class_net/class-1-6/beta/momentum, SGD/class_net/class-1-6/gamma/momentum, SGD/class_net/class-1-7/beta/momentum, SGD/class_net/class-1-7/gamma/momentum, SGD/class_net/class-1/bias/momentum, SGD/class_net/class-1/kernel/momentum, SGD/class_net/class-2-3/beta/momentum, SGD/class_net/class-2-3/gamma/momentum, SGD/class_net/class-2-4/beta/momentum, SGD/class_net/class-2-4/gamma/momentum, SGD/class_net/class-2-5/beta/momentum, SGD/class_net/class-2-5/gamma/momentum, SGD/class_net/class-2-6/beta/momentum, SGD/class_net/class-2-6/gamma/momentum, SGD/class_net/class-2-7/beta/momentum, SGD/class_net/class-2-7/gamma/momentum, SGD/class_net/class-2/bias/momentum, SGD/class_net/class-2/kernel/momentum, SGD/class_net/class-3-3/beta/momentum, SGD/class_net/class-3-3/gamma/momentum, SGD/class_net/class-3-4/beta/momentum, SGD/class_net/class-3-4/gamma/momentum, SGD/class_net/class-3-5/beta/momentum, SGD/class_net/class-3-5/gamma/momentum, SGD/class_net/class-3-6/beta/momentum, SGD/class_net/class-3-6/gamma/momentum, SGD/class_net/class-3-7/beta/momentum, SGD/class_net/class-3-7/gamma/momentum, SGD/class_net/class-3/bias/momentum, SGD/class_net/class-3/kernel/momentum, SGD/class_net/class-predict/bias/momentum, SGD/class_net/class-predict/kernel/momentum, SGD/coarse_mask/coarse-mask-class-0-bn/beta/momentum, SGD/coarse_mask/coarse-mask-class-0-bn/gamma/momentum, SGD/coarse_mask/coarse-mask-class-0/bias/momentum, SGD/coarse_mask/coarse-mask-class-0/kernel/momentum, SGD/coarse_mask/coarse-mask-class-1-bn/beta/momentum, SGD/coarse_mask/coarse-mask-class-1-bn/gamma/momentum, SGD/coarse_mask/coarse-mask-class-1/bias/momentum, SGD/coarse_mask/coarse-mask-class-1/kernel/momentum, SGD/coarse_mask/coarse-mask-class-2-bn/beta/momentum, SGD/coarse_mask/coarse-mask-class-2-bn/gamma/momentum, SGD/coarse_mask/coarse-mask-class-2/bias/momentum, SGD/coarse_mask/coarse-mask-class-2/kernel/momentum, SGD/coarse_mask/coarse-mask-class-3-bn/beta/momentum, SGD/coarse_mask/coarse-mask-class-3-bn/gamma/momentum, SGD/coarse_mask/coarse-mask-class-3/bias/momentum, SGD/coarse_mask/coarse-mask-class-3/kernel/momentum, SGD/coarse_mask/coarse-mask-class-predict/bias/momentum, SGD/coarse_mask/coarse-mask-class-predict/kernel/momentum, SGD/coarse_mask/coarse-mask-fc/bias/momentum, SGD/coarse_mask/coarse-mask-fc/kernel/momentum, SGD/decay, SGD/fine_mask/fine-mask-class-0-bn/beta/momentum, SGD/fine_mask/fine-mask-class-0-bn/gamma/momentum, SGD/fine_mask/fine-mask-class-0/bias/momentum, SGD/fine_mask/fine-mask-class-0/kernel/momentum, SGD/fine_mask/fine-mask-class-1-bn/beta/momentum, SGD/fine_mask/fine-mask-class-1-bn/gamma/momentum, SGD/fine_mask/fine-mask-class-1/bias/momentum, SGD/fine_mask/fine-mask-class-1/kernel/momentum, SGD/fine_mask/fine-mask-class-2-bn/beta/momentum, SGD/fine_mask/fine-mask-class-2-bn/gamma/momentum, SGD/fine_mask/fine-mask-class-2/bias/momentum, SGD/fine_mask/fine-mask-class-2/kernel/momentum, SGD/fine_mask/fine-mask-class-3-bn/beta/momentum, SGD/fine_mask/fine-mask-class-3-bn/gamma/momentum, SGD/fine_mask/fine-mask-class-3/bias/momentum, SGD/fine_mask/fine-mask-class-3/kernel/momentum, SGD/fine_mask/fine-mask-class-predict/bias/momentum, SGD/fine_mask/fine-mask-class-predict/kernel/momentum, SGD/fine_mask/fine-mask-conv2d-tran/bias/momentum, SGD/fine_mask/fine-mask-conv2d-tran/kernel/momentum, SGD/fine_mask/fine-mask-fc/bias/momentum, SGD/fine_mask/fine-mask-fc/kernel/momentum, SGD/fpn/l3/bias/momentum, SGD/fpn/l3/kernel/momentum, SGD/fpn/l4/bias/momentum, SGD/fpn/l4/kernel/momentum, SGD/fpn/l5/bias/momentum, SGD/fpn/l5/kernel/momentum, SGD/fpn/p3-bn/beta/momentum, SGD/fpn/p3-bn/gamma/momentum, SGD/fpn/p4-bn/beta/momentum, SGD/fpn/p4-bn/gamma/momentum, SGD/fpn/p5-bn/beta/momentum, SGD/fpn/p5-bn/gamma/momentum, SGD/fpn/p6-bn/beta/momentum, SGD/fpn/p6-bn/gamma/momentum, SGD/fpn/p6/bias/momentum, SGD/fpn/p6/kernel/momentum, SGD/fpn/p7-bn/beta/momentum, SGD/fpn/p7-bn/gamma/momentum, SGD/fpn/p7/bias/momentum, SGD/fpn/p7/kernel/momentum, SGD/fpn/post_hoc_d3/bias/momentum, SGD/fpn/post_hoc_d3/kernel/momentum, SGD/fpn/post_hoc_d4/bias/momentum, SGD/fpn/post_hoc_d4/kernel/momentum, SGD/fpn/post_hoc_d5/bias/momentum, SGD/fpn/post_hoc_d5/kernel/momentum, SGD/momentum, SGD/prior_mask/dense/bias/momentum, SGD/prior_mask/dense/kernel/momentum, SGD/prior_mask/shape-prior-fc/bias/momentum, SGD/prior_mask/shape-prior-fc/kernel/momentum, SGD/resnet50/batch_normalization/beta/momentum, SGD/resnet50/batch_normalization/gamma/momentum, SGD/resnet50/batch_normalization_1/beta/momentum, SGD/resnet50/batch_normalization_1/gamma/momentum, SGD/resnet50/batch_normalization_10/beta/momentum, SGD/resnet50/batch_normalization_10/gamma/momentum, SGD/resnet50/batch_normalization_11/beta/momentum, SGD/resnet50/batch_normalization_11/gamma/momentum, SGD/resnet50/batch_normalization_12/beta/momentum, SGD/resnet50/batch_normalization_12/gamma/momentum, SGD/resnet50/batch_normalization_13/beta/momentum, SGD/resnet50/batch_normalization_13/gamma/momentum, SGD/resnet50/batch_normalization_14/beta/momentum, SGD/resnet50/batch_normalization_14/gamma/momentum, SGD/resnet50/batch_normalization_15/beta/momentum, SGD/resnet50/batch_normalization_15/gamma/momentum, SGD/resnet50/batch_normalization_16/beta/momentum, SGD/resnet50/batch_normalization_16/gamma/momentum, SGD/resnet50/batch_normalization_17/beta/momentum, SGD/resnet50/batch_normalization_17/gamma/momentum, SGD/resnet50/batch_normalization_18/beta/momentum, SGD/resnet50/batch_normalization_18/gamma/momentum, SGD/resnet50/batch_normalization_19/beta/momentum, SGD/resnet50/batch_normalization_19/gamma/momentum, SGD/resnet50/batch_normalization_2/beta/momentum, SGD/resnet50/batch_normalization_2/gamma/momentum, SGD/resnet50/batch_normalization_20/beta/momentum, SGD/resnet50/batch_normalization_20/gamma/momentum, SGD/resnet50/batch_normalization_21/beta/momentum, SGD/resnet50/batch_normalization_21/gamma/momentum, SGD/resnet50/batch_normalization_22/beta/momentum, SGD/resnet50/batch_normalization_22/gamma/momentum, SGD/resnet50/batch_normalization_23/beta/momentum, SGD/resnet50/batch_normalization_23/gamma/momentum, SGD/resnet50/batch_normalization_24/beta/momentum, SGD/resnet50/batch_normalization_24/gamma/momentum, SGD/resnet50/batch_normalization_25/beta/momentum, SGD/resnet50/batch_normalization_25/gamma/momentum, SGD/resnet50/batch_normalization_26/beta/momentum, SGD/resnet50/batch_normalization_26/gamma/momentum, SGD/resnet50/batch_normalization_27/beta/momentum, SGD/resnet50/batch_normalization_27/gamma/momentum, SGD/resnet50/batch_normalization_28/beta/momentum, SGD/resnet50/batch_normalization_28/gamma/momentum, SGD/resnet50/batch_normalization_29/beta/momentum, SGD/resnet50/batch_normalization_29/gamma/momentum, SGD/resnet50/batch_normalization_3/beta/momentum, SGD/resnet50/batch_normalization_3/gamma/momentum, SGD/resnet50/batch_normalization_30/beta/momentum, SGD/resnet50/batch_normalization_30/gamma/momentum, SGD/resnet50/batch_normalization_31/beta/momentum, SGD/resnet50/batch_normalization_31/gamma/momentum, SGD/resnet50/batch_normalization_32/beta/momentum, SGD/resnet50/batch_normalization_32/gamma/momentum, SGD/resnet50/batch_normalization_33/beta/momentum, SGD/resnet50/batch_normalization_33/gamma/momentum, SGD/resnet50/batch_normalization_34/beta/momentum, SGD/resnet50/batch_normalization_34/gamma/momentum, SGD/resnet50/batch_normalization_35/beta/momentum, SGD/resnet50/batch_normalization_35/gamma/momentum, SGD/resnet50/batch_normalization_36/beta/momentum, SGD/resnet50/batch_normalization_36/gamma/momentum, SGD/resnet50/batch_normalization_37/beta/momentum, SGD/resnet50/batch_normalization_37/gamma/momentum, SGD/resnet50/batch_normalization_38/beta/momentum, SGD/resnet50/batch_normalization_38/gamma/momentum, SGD/resnet50/batch_normalization_39/beta/momentum, SGD/resnet50/batch_normalization_39/gamma/momentum, SGD/resnet50/batch_normalization_4/beta/momentum, SGD/resnet50/batch_normalization_4/gamma/momentum, SGD/resnet50/batch_normalization_40/beta/momentum, SGD/resnet50/batch_normalization_40/gamma/momentum, SGD/resnet50/batch_normalization_41/beta/momentum, SGD/resnet50/batch_normalization_41/gamma/momentum, SGD/resnet50/batch_normalization_42/beta/momentum, SGD/resnet50/batch_normalization_42/gamma/momentum, SGD/resnet50/batch_normalization_43/beta/momentum, SGD/resnet50/batch_normalization_43/gamma/momentum, SGD/resnet50/batch_normalization_44/beta/momentum, SGD/resnet50/batch_normalization_44/gamma/momentum, SGD/resnet50/batch_normalization_45/beta/momentum, SGD/resnet50/batch_normalization_45/gamma/momentum, SGD/resnet50/batch_normalization_46/beta/momentum, SGD/resnet50/batch_normalization_46/gamma/momentum, SGD/resnet50/batch_normalization_47/beta/momentum, SGD/resnet50/batch_normalization_47/gamma/momentum, SGD/resnet50/batch_normalization_48/beta/momentum, SGD/resnet50/batch_normalization_48/gamma/momentum, SGD/resnet50/batch_normalization_49/beta/momentum, SGD/resnet50/batch_normalization_49/gamma/momentum, SGD/resnet50/batch_normalization_5/beta/momentum, SGD/resnet50/batch_normalization_5/gamma/momentum, SGD/resnet50/batch_normalization_50/beta/momentum, SGD/resnet50/batch_normalization_50/gamma/momentum, SGD/resnet50/batch_normalization_51/beta/momentum, SGD/resnet50/batch_normalization_51/gamma/momentum, SGD/resnet50/batch_normalization_52/beta/momentum, SGD/resnet50/batch_normalization_52/gamma/momentum, SGD/resnet50/batch_normalization_6/beta/momentum, SGD/resnet50/batch_normalization_6/gamma/momentum, SGD/resnet50/batch_normalization_7/beta/momentum, SGD/resnet50/batch_normalization_7/gamma/momentum, SGD/resnet50/batch_normalization_8/beta/momentum, SGD/resnet50/batch_normalization_8/gamma/momentum, SGD/resnet50/batch_normalization_9/beta/momentum, SGD/resnet50/batch_normalization_9/gamma/momentum, SGD/resnet50/conv2d_11/kernel/momentum, SGD/resnet50/conv2d_12/kernel/momentum, SGD/resnet50/conv2d_13/kernel/momentum, SGD/resnet50/conv2d_14/kernel/momentum, SGD/resnet50/conv2d_15/kernel/momentum, SGD/resnet50/conv2d_16/kernel/momentum, SGD/resnet50/conv2d_17/kernel/momentum, SGD/resnet50/conv2d_18/kernel/momentum, SGD/resnet50/conv2d_19/kernel/momentum, SGD/resnet50/conv2d_20/kernel/momentum, SGD/resnet50/conv2d_21/kernel/momentum, SGD/resnet50/conv2d_22/kernel/momentum, SGD/resnet50/conv2d_23/kernel/momentum, SGD/resnet50/conv2d_24/kernel/momentum, SGD/resnet50/conv2d_25/kernel/momentum, SGD/resnet50/conv2d_26/kernel/momentum, SGD/resnet50/conv2d_27/kernel/momentum, SGD/resnet50/conv2d_28/kernel/momentum, SGD/resnet50/conv2d_29/kernel/momentum, SGD/resnet50/conv2d_30/kernel/momentum, SGD/resnet50/conv2d_31/kernel/momentum, SGD/resnet50/conv2d_32/kernel/momentum, SGD/resnet50/conv2d_33/kernel/momentum, SGD/resnet50/conv2d_34/kernel/momentum, SGD/resnet50/conv2d_35/kernel/momentum, SGD/resnet50/conv2d_36/kernel/momentum, SGD/resnet50/conv2d_37/kernel/momentum, SGD/resnet50/conv2d_38/kernel/momentum, SGD/resnet50/conv2d_39/kernel/momentum, SGD/resnet50/conv2d_40/kernel/momentum, SGD/resnet50/conv2d_41/kernel/momentum, SGD/resnet50/conv2d_42/kernel/momentum, SGD/resnet50/conv2d_43/kernel/momentum, SGD/resnet50/conv2d_44/kernel/momentum, SGD/resnet50/conv2d_45/kernel/momentum, SGD/resnet50/conv2d_46/kernel/momentum, SGD/resnet50/conv2d_47/kernel/momentum, SGD/resnet50/conv2d_48/kernel/momentum, SGD/resnet50/conv2d_49/kernel/momentum, SGD/resnet50/conv2d_50/kernel/momentum, SGD/resnet50/conv2d_51/kernel/momentum, SGD/resnet50/conv2d_52/kernel/momentum, box_net/box-0-3/beta, box_net/box-0-3/gamma, box_net/box-0-3/moving_mean, box_net/box-0-3/moving_variance, box_net/box-0-4/beta, box_net/box-0-4/gamma, box_net/box-0-4/moving_mean, box_net/box-0-4/moving_variance, box_net/box-0-5/beta, box_net/box-0-5/gamma, box_net/box-0-5/moving_mean, box_net/box-0-5/moving_variance, box_net/box-0-6/beta, box_net/box-0-6/gamma, box_net/box-0-6/moving_mean, box_net/box-0-6/moving_variance, box_net/box-0-7/beta, box_net/box-0-7/gamma, box_net/box-0-7/moving_mean, box_net/box-0-7/moving_variance, box_net/box-0/bias, box_net/box-0/kernel, box_net/box-1-3/beta, box_net/box-1-3/gamma, box_net/box-1-3/moving_mean, box_net/box-1-3/moving_variance, box_net/box-1-4/beta, box_net/box-1-4/gamma, box_net/box-1-4/moving_mean, box_net/box-1-4/moving_variance, box_net/box-1-5/beta, box_net/box-1-5/gamma, box_net/box-1-5/moving_mean, box_net/box-1-5/moving_variance, box_net/box-1-6/beta, box_net/box-1-6/gamma, box_net/box-1-6/moving_mean, box_net/box-1-6/moving_variance, box_net/box-1-7/beta, box_net/box-1-7/gamma, box_net/box-1-7/moving_mean, box_net/box-1-7/moving_variance, box_net/box-1/bias, box_net/box-1/kernel, box_net/box-2-3/beta, box_net/box-2-3/gamma, box_net/box-2-3/moving_mean, box_net/box-2-3/moving_variance, box_net/box-2-4/beta, box_net/box-2-4/gamma, box_net/box-2-4/moving_mean, box_net/box-2-4/moving_variance, box_net/box-2-5/beta, box_net/box-2-5/gamma, box_net/box-2-5/moving_mean, box_net/box-2-5/moving_variance, box_net/box-2-6/beta, box_net/box-2-6/gamma, box_net/box-2-6/moving_mean, box_net/box-2-6/moving_variance, box_net/box-2-7/beta, box_net/box-2-7/gamma, box_net/box-2-7/moving_mean, box_net/box-2-7/moving_variance, box_net/box-2/bias, box_net/box-2/kernel, box_net/box-3-3/beta, box_net/box-3-3/gamma, box_net/box-3-3/moving_mean, box_net/box-3-3/moving_variance, box_net/box-3-4/beta, box_net/box-3-4/gamma, box_net/box-3-4/moving_mean, box_net/box-3-4/moving_variance, box_net/box-3-5/beta, box_net/box-3-5/gamma, box_net/box-3-5/moving_mean, box_net/box-3-5/moving_variance, box_net/box-3-6/beta, box_net/box-3-6/gamma, box_net/box-3-6/moving_mean, box_net/box-3-6/moving_variance, box_net/box-3-7/beta, box_net/box-3-7/gamma, box_net/box-3-7/moving_mean, box_net/box-3-7/moving_variance, box_net/box-3/bias, box_net/box-3/kernel, box_net/box-predict/bias, box_net/box-predict/kernel, class_net/class-0-3/beta, class_net/class-0-3/gamma, class_net/class-0-3/moving_mean, class_net/class-0-3/moving_variance, class_net/class-0-4/beta, class_net/class-0-4/gamma, class_net/class-0-4/moving_mean, class_net/class-0-4/moving_variance, class_net/class-0-5/beta, class_net/class-0-5/gamma, class_net/class-0-5/moving_mean, class_net/class-0-5/moving_variance, class_net/class-0-6/beta, class_net/class-0-6/gamma, class_net/class-0-6/moving_mean, class_net/class-0-6/moving_variance, class_net/class-0-7/beta, class_net/class-0-7/gamma, class_net/class-0-7/moving_mean, class_net/class-0-7/moving_variance, class_net/class-0/bias, class_net/class-0/kernel, class_net/class-1-3/beta, class_net/class-1-3/gamma, class_net/class-1-3/moving_mean, class_net/class-1-3/moving_variance, class_net/class-1-4/beta, class_net/class-1-4/gamma, class_net/class-1-4/moving_mean, class_net/class-1-4/moving_variance, class_net/class-1-5/beta, class_net/class-1-5/gamma, class_net/class-1-5/moving_mean, class_net/class-1-5/moving_variance, class_net/class-1-6/beta, class_net/class-1-6/gamma, class_net/class-1-6/moving_mean, class_net/class-1-6/moving_variance, class_net/class-1-7/beta, class_net/class-1-7/gamma, class_net/class-1-7/moving_mean, class_net/class-1-7/moving_variance, class_net/class-1/bias, class_net/class-1/kernel, class_net/class-2-3/beta, class_net/class-2-3/gamma, class_net/class-2-3/moving_mean, class_net/class-2-3/moving_variance, class_net/class-2-4/beta, class_net/class-2-4/gamma, class_net/class-2-4/moving_mean, class_net/class-2-4/moving_variance, class_net/class-2-5/beta, class_net/class-2-5/gamma, class_net/class-2-5/moving_mean, class_net/class-2-5/moving_variance, class_net/class-2-6/beta, class_net/class-2-6/gamma, class_net/class-2-6/moving_mean, class_net/class-2-6/moving_variance, class_net/class-2-7/beta, class_net/class-2-7/gamma, class_net/class-2-7/moving_mean, class_net/class-2-7/moving_variance, class_net/class-2/bias, class_net/class-2/kernel, class_net/class-3-3/beta, class_net/class-3-3/gamma, class_net/class-3-3/moving_mean, class_net/class-3-3/moving_variance, class_net/class-3-4/beta, class_net/class-3-4/gamma, class_net/class-3-4/moving_mean, class_net/class-3-4/moving_variance, class_net/class-3-5/beta, class_net/class-3-5/gamma, class_net/class-3-5/moving_mean, class_net/class-3-5/moving_variance, class_net/class-3-6/beta, class_net/class-3-6/gamma, class_net/class-3-6/moving_mean, class_net/class-3-6/moving_variance, class_net/class-3-7/beta, class_net/class-3-7/gamma, class_net/class-3-7/moving_mean, class_net/class-3-7/moving_variance, class_net/class-3/bias, class_net/class-3/kernel, class_net/class-predict/bias, class_net/class-predict/kernel, coarse_mask/coarse-mask-class-0-bn/beta, coarse_mask/coarse-mask-class-0-bn/gamma, coarse_mask/coarse-mask-class-0-bn/moving_mean, coarse_mask/coarse-mask-class-0-bn/moving_variance, coarse_mask/coarse-mask-class-0/bias, coarse_mask/coarse-mask-class-0/kernel, coarse_mask/coarse-mask-class-1-bn/beta, coarse_mask/coarse-mask-class-1-bn/gamma, coarse_mask/coarse-mask-class-1-bn/moving_mean, coarse_mask/coarse-mask-class-1-bn/moving_variance, coarse_mask/coarse-mask-class-1/bias, coarse_mask/coarse-mask-class-1/kernel, coarse_mask/coarse-mask-class-2-bn/beta, coarse_mask/coarse-mask-class-2-bn/gamma, coarse_mask/coarse-mask-class-2-bn/moving_mean, coarse_mask/coarse-mask-class-2-bn/moving_variance, coarse_mask/coarse-mask-class-2/bias, coarse_mask/coarse-mask-class-2/kernel, coarse_mask/coarse-mask-class-3-bn/beta, coarse_mask/coarse-mask-class-3-bn/gamma, coarse_mask/coarse-mask-class-3-bn/moving_mean, coarse_mask/coarse-mask-class-3-bn/moving_variance, coarse_mask/coarse-mask-class-3/bias, coarse_mask/coarse-mask-class-3/kernel, coarse_mask/coarse-mask-class-predict/bias, coarse_mask/coarse-mask-class-predict/kernel, coarse_mask/coarse-mask-fc/bias, coarse_mask/coarse-mask-fc/kernel, fine_mask/fine-mask-class-0-bn/beta, fine_mask/fine-mask-class-0-bn/gamma, fine_mask/fine-mask-class-0-bn/moving_mean, fine_mask/fine-mask-class-0-bn/moving_variance, fine_mask/fine-mask-class-0/bias, fine_mask/fine-mask-class-0/kernel, fine_mask/fine-mask-class-1-bn/beta, fine_mask/fine-mask-class-1-bn/gamma, fine_mask/fine-mask-class-1-bn/moving_mean, fine_mask/fine-mask-class-1-bn/moving_variance, fine_mask/fine-mask-class-1/bias, fine_mask/fine-mask-class-1/kernel, fine_mask/fine-mask-class-2-bn/beta, fine_mask/fine-mask-class-2-bn/gamma, fine_mask/fine-mask-class-2-bn/moving_mean, fine_mask/fine-mask-class-2-bn/moving_variance, fine_mask/fine-mask-class-2/bias, fine_mask/fine-mask-class-2/kernel, fine_mask/fine-mask-class-3-bn/beta, fine_mask/fine-mask-class-3-bn/gamma, fine_mask/fine-mask-class-3-bn/moving_mean, fine_mask/fine-mask-class-3-bn/moving_variance, fine_mask/fine-mask-class-3/bias, fine_mask/fine-mask-class-3/kernel, fine_mask/fine-mask-class-predict/bias, fine_mask/fine-mask-class-predict/kernel, fine_mask/fine-mask-conv2d-tran/bias, fine_mask/fine-mask-conv2d-tran/kernel, fine_mask/fine-mask-fc/bias, fine_mask/fine-mask-fc/kernel, iter, prior_mask/shape-prior-fc/bias, prior_mask/shape-prior-fc/kernel, save_counter

(303 variable name(s) did match)

Could not find some variables in the checkpoint (see names above). Saver was attempting to load an object-based checkpoint (saved using tf.train.Checkpoint or tf.keras.Model.save_weights) using variable names. If the checkpoint was written with eager execution enabled, it's possible that variable names have changed (for example missing a '_1' suffix). It's also possible that there are new variables which did not exist when the checkpoint was written. You can construct a Saver(var_list=...) with only the variables which previously existed, and if variable names have changed you may need to make this a dictionary with the old names as keys. If you're using an Estimator, you'll need to return a tf.train.Saver inside a tf.train.Scaffold from your model_fn.

vaghelaviral commented 4 years ago

Not the same issue but getting Above issue while exporting shapemask checkpoints Any help would be appreciated

tensorflow / tpu

Detection - ShapeMask TensorRT models cannot predict DetectionMasks #815