search

tensorflow / models

Models and examples built with TensorFlow
Other
77.03k stars 45.77k forks source link

deeplab/export_model.py fails for deeplabv3_mnv2_dm05_pascal_trainaug #10614

Open alexander-sony opened 2 years ago

alexander-sony commented 2 years ago

Prerequisites

Please answer the following questions for yourself before submitting an issue.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/deeplab/export_model.py

2. Describe the bug

export_model.py fails exporting one of the models in model zoo, http://download.tensorflow.org/models/deeplabv3_mnv2_dm05_pascal_trainaug_2018_10_01.tar.gz

3. Steps to reproduce

see colab, https://colab.research.google.com/drive/1eUukOqChFFLPZa7TwUOcXZguvJINTMqy?usp=sharing

4. Expected behavior

frozen_inference_graph.pb

5. Additional context

WARNING:tensorflow:From /content/models/research/deeplab/core/conv2d_ws.py:40: The name tf.layers.Layer is deprecated. Please use tf.compat.v1.layers.Layer instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From deeplab/export_model.py:201: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From deeplab/export_model.py:117: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W0429 13:33:00.107792 139722982799232 module_wrapper.py:139] From deeplab/export_model.py:117: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From deeplab/export_model.py:117: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

W0429 13:33:00.107982 139722982799232 module_wrapper.py:139] From deeplab/export_model.py:117: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

WARNING:tensorflow:From deeplab/export_model.py:118: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

W0429 13:33:00.108135 139722982799232 module_wrapper.py:139] From deeplab/export_model.py:118: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:Prepare to export model to: /content/exported/deeplabv3_mnv2_dm05_pascal_trainaug/frozen_inference_graph.pb
I0429 13:33:00.108238 139722982799232 export_model.py:118] Prepare to export model to: /content/exported/deeplabv3_mnv2_dm05_pascal_trainaug/frozen_inference_graph.pb
WARNING:tensorflow:From deeplab/export_model.py:91: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0429 13:33:00.108820 139722982799232 module_wrapper.py:139] From deeplab/export_model.py:91: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

INFO:tensorflow:Exported model performs single-scale inference.
I0429 13:33:00.157119 139722982799232 export_model.py:130] Exported model performs single-scale inference.
WARNING:tensorflow:From /content/models/research/deeplab/model.py:320: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

W0429 13:33:00.157407 139722982799232 module_wrapper.py:139] From /content/models/research/deeplab/model.py:320: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:From /content/models/research/deeplab/core/feature_extractor.py:490: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0429 13:33:00.157798 139722982799232 deprecation.py:323] From /content/models/research/deeplab/core/feature_extractor.py:490: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
W0429 13:33:00.163173 139722982799232 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From /content/models/research/deeplab/core/utils.py:41: The name tf.image.resize_bilinear is deprecated. Please use tf.compat.v1.image.resize_bilinear instead.

W0429 13:33:01.552271 139722982799232 module_wrapper.py:139] From /content/models/research/deeplab/core/utils.py:41: The name tf.image.resize_bilinear is deprecated. Please use tf.compat.v1.image.resize_bilinear instead.

WARNING:tensorflow:From /content/models/research/deeplab/model.py:894: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0429 13:33:01.607824 139722982799232 module_wrapper.py:139] From /content/models/research/deeplab/model.py:894: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From deeplab/export_model.py:162: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

W0429 13:33:01.735388 139722982799232 module_wrapper.py:139] From deeplab/export_model.py:162: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

INFO:tensorflow:Skipping quant after MobilenetV2/Conv/add_fold
I0429 13:33:02.865628 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/Conv/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv/depthwise/add_fold
I0429 13:33:02.865986 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_1/expand/add_fold
I0429 13:33:02.866302 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_1/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_1/depthwise/add_fold
I0429 13:33:02.866502 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_1/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_2/expand/add_fold
I0429 13:33:02.866819 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_2/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_2/depthwise/add_fold
I0429 13:33:02.867004 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_2/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_3/expand/add_fold
I0429 13:33:02.867296 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_3/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_3/depthwise/add_fold
I0429 13:33:02.867472 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_3/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_4/expand/add_fold
I0429 13:33:02.867732 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_4/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_4/depthwise/add_fold
I0429 13:33:02.867913 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_4/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_5/expand/add_fold
I0429 13:33:02.868172 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_5/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_5/depthwise/add_fold
I0429 13:33:02.868381 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_5/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_6/expand/add_fold
I0429 13:33:02.868636 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_6/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_6/depthwise/add_fold
I0429 13:33:02.868814 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_6/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_7/expand/add_fold
I0429 13:33:02.869067 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_7/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_7/depthwise/add_fold
I0429 13:33:02.869250 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_7/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_8/expand/add_fold
I0429 13:33:02.869482 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_8/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_8/depthwise/add_fold
I0429 13:33:02.869660 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_8/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_9/expand/add_fold
I0429 13:33:02.869906 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_9/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_9/depthwise/add_fold
I0429 13:33:02.870081 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_9/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_10/expand/add_fold
I0429 13:33:02.870318 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_10/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_10/depthwise/add_fold
I0429 13:33:02.870484 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_10/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_11/expand/add_fold
I0429 13:33:02.870728 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_11/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_11/depthwise/add_fold
I0429 13:33:02.870896 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_11/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_12/expand/add_fold
I0429 13:33:02.871158 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_12/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_12/depthwise/add_fold
I0429 13:33:02.871338 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_12/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_13/expand/add_fold
I0429 13:33:02.871595 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_13/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_13/depthwise/add_fold
I0429 13:33:02.871773 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_13/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_14/expand/add_fold
I0429 13:33:02.872102 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_14/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_14/depthwise/add_fold
I0429 13:33:02.872298 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_14/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_15/expand/add_fold
I0429 13:33:02.872565 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_15/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_15/depthwise/add_fold
I0429 13:33:02.872778 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_15/depthwise/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_16/expand/add_fold
I0429 13:33:02.873050 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_16/expand/add_fold
INFO:tensorflow:Skipping quant after MobilenetV2/expanded_conv_16/depthwise/add_fold
I0429 13:33:02.873232 139722982799232 quantize.py:299] Skipping quant after MobilenetV2/expanded_conv_16/depthwise/add_fold
INFO:tensorflow:Skipping quant after image_pooling/add_fold
I0429 13:33:02.873485 139722982799232 quantize.py:299] Skipping quant after image_pooling/add_fold
INFO:tensorflow:Skipping quant after aspp0/add_fold
I0429 13:33:02.873652 139722982799232 quantize.py:299] Skipping quant after aspp0/add_fold
INFO:tensorflow:Skipping quant after concat_projection/add_fold
I0429 13:33:02.873811 139722982799232 quantize.py:299] Skipping quant after concat_projection/add_fold
WARNING:tensorflow:From deeplab/export_model.py:178: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0429 13:33:02.875690 139722982799232 module_wrapper.py:139] From deeplab/export_model.py:178: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From deeplab/export_model.py:178: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Please use tf.global_variables instead.
W0429 13:33:02.875893 139722982799232 deprecation.py:323] From deeplab/export_model.py:178: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Please use tf.global_variables instead.
WARNING:tensorflow:From deeplab/export_model.py:181: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W0429 13:33:03.247953 139722982799232 module_wrapper.py:139] From deeplab/export_model.py:181: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

WARNING:tensorflow:From deeplab/export_model.py:182: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

W0429 13:33:03.248302 139722982799232 module_wrapper.py:139] From deeplab/export_model.py:182: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0429 13:33:03.372743 139722982799232 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2022-04-29 13:33:03.926309: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2022-04-29 13:33:03.939609: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2022-04-29 13:33:03.939659: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (92dcdf2110ba): /proc/driver/nvidia/version does not exist
2022-04-29 13:33:03.944503: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2022-04-29 13:33:03.944738: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x27e99c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-04-29 13:33:03.944769: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
INFO:tensorflow:Restoring parameters from /content/deeplabv3_mnv2_dm05_pascal_trainaug/model.ckpt
I0429 13:33:03.946846 139722982799232 saver.py:1284] Restoring parameters from /content/deeplabv3_mnv2_dm05_pascal_trainaug/model.ckpt
2022-04-29 13:33:04.626014: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key MobilenetV2/Conv/act_quant/max not found in checkpoint
Traceback (most recent call last):
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key MobilenetV2/Conv/act_quant/max not found in checkpoint
     [[{{node save/RestoreV2}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 1290, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key MobilenetV2/Conv/act_quant/max not found in checkpoint
     [[node save/RestoreV2 (defined at /tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'save/RestoreV2':
  File "deeplab/export_model.py", line 201, in <module>
    tf.app.run()
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "deeplab/export_model.py", line 178, in main
    saver = tf.train.Saver(tf.all_variables())
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 828, in __init__
    self.build()
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 840, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 878, in _build
    build_restore=build_restore)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps
    restore_sequentially)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 575, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
    name=name)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 1300, in restore
    names_to_keys = object_graph_key_mapping(save_path)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 1618, in object_graph_key_mapping
    object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/pywrap_tensorflow_internal.py", line 915, in get_tensor
    return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "deeplab/export_model.py", line 201, in <module>
    tf.app.run()
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "deeplab/export_model.py", line 192, in main
    initializer_nodes=None)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/tools/freeze_graph.py", line 151, in freeze_graph_with_def_protos
    saver.restore(sess, input_checkpoint)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 1306, in restore
    err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key MobilenetV2/Conv/act_quant/max not found in checkpoint
     [[node save/RestoreV2 (defined at /tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'save/RestoreV2':
  File "deeplab/export_model.py", line 201, in <module>
    tf.app.run()
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "deeplab/export_model.py", line 178, in main
    saver = tf.train.Saver(tf.all_variables())
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 828, in __init__
    self.build()
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 840, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 878, in _build
    build_restore=build_restore)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps
    restore_sequentially)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 575, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
    name=name)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

6. System information

google colab notebook TF 1.5.2

saberkun commented 2 years ago

Could you use the deeplap v2 TF2 version? I am afraid that the deeplab research team may not answer TF1 codebase questions.

google-ml-butler[bot] commented 2 years ago

Are you satisfied with the resolution of your issue? Yes No

alexander-sony commented 2 years ago

Could you use the deeplap v2 TF2 version? I am afraid that the deeplab research team may not answer TF1 codebase questions.

Not really since it's not been tested with TFLite. I want to use semantic segmentation on a DSP and need a (~300x300) quantized TFLite model.

alexander-sony commented 2 years ago

I still do not understand why deeplab/export_model.py fails when used in a colab notebook. However, have instead successfully used deeplab/local_test.py using TF1.15 running from docker. local_test.sh is running all the essential scripts; test, training evaluation and export.

chunduriv commented 2 years ago

Reopened the issue as per above comments.