PINTO0309
commented
3 years ago Fixed a bug in NonZero, and it now works properly until NonMaxSuppresion is processed. This is just a quick fix, so I haven't committed it yet.
### NonZero
elif layer.attrib['type'] == 'NonZero':
output_type = tf.int64
if not data is None and 'output_type' in data.attrib:
output_type = cast_type_ov_tf[data.attrib['output_type']]
try:
# type_spec.dtype
if tf_layers_dict[get_tf_edges_from(tf_edges, layer_id, 0)].type_spec.dtype != tf.bool:
input_type = tf_layers_dict[get_tf_edges_from(tf_edges, layer_id, 0)].type_spec.dtype
mask = tf.math.not_equal(
tf_layers_dict[get_tf_edges_from(tf_edges, layer_id, 0)],
tf.constant([0], dtype=input_type)
)
tf_layers_dict[layer_id] = tf.expand_dims(
tf.boolean_mask(tf_layers_dict[get_tf_edges_from(tf_edges, layer_id, 0)], mask),
axis=0
)
else:
temp_op = tf.cast(tf_layers_dict[get_tf_edges_from(tf_edges, layer_id, 0)], output_type)
mask = tf.math.not_equal(
temp_op,
tf.constant([0], dtype=output_type)
)
tf_layers_dict[layer_id] = tf.expand_dims(
tf.boolean_mask(temp_op, mask),
axis=0
)
except:
# dtype
if tf_layers_dict[get_tf_edges_from(tf_edges, layer_id, 0)].dtype != tf.bool:
input_type = tf_layers_dict[get_tf_edges_from(tf_edges, layer_id, 0)].dtype
mask = tf.math.not_equal(
tf_layers_dict[get_tf_edges_from(tf_edges, layer_id, 0)],
tf.constant([0], dtype=input_type)
)
tf_layers_dict[layer_id] = tf.expand_dims(
tf.boolean_mask(tf_layers_dict[get_tf_edges_from(tf_edges, layer_id, 0)], mask),
axis=0
)
else:
temp_op = tf.cast(tf_layers_dict[get_tf_edges_from(tf_edges, layer_id, 0)], output_type)
mask = tf.math.not_equal(
temp_op,
tf.constant([0], dtype=output_type)
)
tf_layers_dict[layer_id] = tf.expand_dims(
tf.boolean_mask(temp_op, mask),
axis=0
)There is a big problem here. The result of NonZero should exactly be a tensor of indefinite size, but OpenVINO appears to produce a fixed size output. The behavior specified in the specification does not match the specification of the actually generated model.
True = 1
False = 0
For example, NonZero( [True, False, False, True] ) should be [1, 2], but it produces output of size [1, 4]. In the first place, the number of True's is not constant, so the size can be all combinations of 0 or 1 or 2 or 3 or 4. The figure below suggests that NonZero( [19248 of True/False] ) results in [1, 19248]. However, the number of non-zero values should not be constant, so there is a problem with the specification. The NonZero result has an almost 100% probability of not returning 19248 values. It is inevitable that you will be confused. This is because the structure of the model you see does not match the internal workings of the framework.
https://docs.openvinotoolkit.org/latest/openvino_docs_ops_condition_NonZero_3.html
As a result, TensorFlow is unable to predict the size of the tensor to be passed to NMS, resulting in the error shown in the figure below. scores.shape[0] is the scores parameter of NMS. NMS requires score, boxes, and class probabilities with matching dimensions, but because the number of dimensions is undefined, errors cannot be avoided in the subsequent transformation process.
ERROR: list index out of range
ERROR: model_path : openvino/yolact_detect_pytorchnms11/550x550/FP32/yolact_detect_pytorchnms11.xml
ERROR: weights_path: openvino/yolact_detect_pytorchnms11/550x550/FP32/yolact_detect_pytorchnms11.bin
ERROR: layer_id : 707
ERROR: The trace log is below.
Traceback (most recent call last):
File "openvino2tensorflow.py", line 2098, in convert
class_count = scores.shape[0]
File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/tensor_shape.py", line 896, in __getitem__
return self._dims[key].value
IndexError: list index out of rangeDo you understand what I'm saying? This is the reason why I told you in my email that I don't want you to use NonZero a lot if possible. The post-processing implemented in PyTorch uses a lot of NonZero to mass produce processes with indeterminate tensor sizes, which is why I have not actively implemented NMS. This is because TensorFlow's default processing cannot handle it.
Therefore, you really need to consider whether you need to do the following,
keep = (conf_scores > self.conf_thresh)and
id = (conf_scores > self.conf_thresh)By the way, I don't think it's necessary.
taisuke-tomida
I introduced NMS to yolact's ONNX model in my own unique way.
However, after converting to OpenVINO, I am unable to convert to tflite.
os etc
windows10 cuda10.2
Version
python=3.6.8 tensorflow=2.5.0 onnx=1.9.0 openvino2tensorflow=1.15.3 pytorch=1.8.1+cu102
How to add an NMS
out = []
with timer.env('Detect'): batch_size = 1 num_priors = 19248
return out
Error in openvino2tensorflow
command
result
thanks