search

KhronosGroup / NNEF-Tools

The NNEF Tools repository contains tools to generate and consume NNEF documents
https://www.khronos.org/nnef
222 stars 57 forks source link

Converting Mask_RCNN to NNEF #103

Closed as-icuro closed 4 years ago

as-icuro commented 4 years ago

Hello,

We have been trying to convert Mask_RCNN to NNEF and are running into the following errors. We have tried two different approaches:

1. Converting matterport's Mask_RCNN (Keras -> TensorFlow -> NNEF)

We used the inspect_model.ipynb to load the weights which was trained on the COCO dataset. Once the weights were loaded, we used the method described in this Medium article to convert the Keras model to a TensorFlow frozen graph. After obtaining the frozen graph, we tried using nnef_tools/convert.py to convert the model to NNEF, but then the following error was generated:

/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Info: Input shape: input_image:0: [-1, -1, -1, 3]
Info: Input shape: input_image_meta:0: [-1, 93]
Info: Input shape: input_anchors:0: [-1, -1, 4]
Warning: Incomplete input shape is auto-fixed: input_image:0. [-1, -1, -1, 3] -> [1, 1, 1, 3]. Use --input-shape if other shape is desired.
Warning: Incomplete input shape is auto-fixed: input_image_meta:0. [-1, 93] -> [1, 93]. Use --input-shape if other shape is desired.
Warning: Incomplete input shape is auto-fixed: input_anchors:0. [-1, -1, 4] -> [1, 1, 4]. Use --input-shape if other shape is desired.
Traceback (most recent call last):
  File "./nnef_tools/convert.py", line 679, in <module>
    convert_using_argv(sys.argv)
  File "./nnef_tools/convert.py", line 667, in convert_using_argv
    conversion_info=args.conversion_info)
  File "./nnef_tools/convert.py", line 480, in convert
    custom_converters=custom_converters))
  File "./nnef_tools/convert.py", line 377, in convert_using_premade_objects
    source_graph = reader(*in_filename)
  File "/home/boat/NNEF-Tools/nnef_tools/io/tensorflow/tf_pb_io.py", line 447, in __call__
    tf_pb_to_tf_py.evaluate_and_convert(g, source_shapes=self._input_shape)
  File "/home/boat/NNEF-Tools/nnef_tools/conversion/tensorflow/tf_pb_to_tf_py.py", line 85, in evaluate_and_convert
    tf_pb_shape_inference._DefaultPropagators[op.name](op, const_value_by_tensor)
  File "/home/boat/NNEF-Tools/nnef_tools/io/tensorflow/tf_pb/tf_pb_shape_inference.py", line 190, in propagate_resize
    size = const_value_by_tensor[size].tolist()  # type: typing.List[int]
KeyError: T(fpn_p5upsampled/mul:0)

2. Converting the model from TensorFlow Model Zoo (TensorFlow -> NNEF)

In this approach, we tried using the model mask_rcnn_resnet101_atrous_coco from the TensorFlow Detection Model Zoo and then tried converting the "frozen_inference_graph.pb" to NNEF using nnef_tools/convert.py. The following error was generated:

/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/boat/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Info: Input shape: image_tensor:0: [-1, -1, -1, 3]
Info: Input shape: FirstStageFeatureExtractor/Assert/Assert/data_0:0: []
Info: Input shape: GridAnchorGenerator/assert_equal/Const:0: [0]
Info: Input shape: GridAnchorGenerator/assert_equal/Assert/Assert/data_0:0: []
Info: Input shape: GridAnchorGenerator/assert_equal/Assert/Assert/data_1:0: []
Info: Input shape: GridAnchorGenerator/assert_equal/Assert/Assert/data_2:0: []
Info: Input shape: GridAnchorGenerator/assert_equal/Assert/Assert/data_4:0: []
Info: Input shape: BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Assert/Assert/data_0:0: []
Info: Input shape: BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/SortByField/Assert/Assert/data_0:0: []
Info: Input shape: BatchMultiClassNonMaxSuppression_1/map/while/MultiClassNonMaxSuppression/Assert/Assert/data_0:0: []
Info: Input shape: BatchMultiClassNonMaxSuppression_1/map/while/MultiClassNonMaxSuppression/SortByField/Assert/Assert/data_0:0: []
Warning: Incomplete input shape is auto-fixed: image_tensor:0. [-1, -1, -1, 3] -> [1, 1, 1, 3]. Use --input-shape if other shape is desired.
Error: No shape propagator for TensorArrayV3

What could be causing these errors and how do we resolve them? Any help regarding this issue is much appreciated. Thanks.

Software Versions:

gyenesvi commented 4 years ago

Unfortunately, the error messages are misleading here (we will try to improve those), but in general, this network contains many operations that are not supported by NNEF. Namely, the network has dynamic control flow operations (like 'loop' and 'switch') and operations with dynamic output shapes (like bounding-box 'non-max suppression' and 'where') for which NNEF is not prepared yet.

as-icuro commented 4 years ago

That makes a lot of sense. Thank you for the quick reply.