RuntimeError: Select TensorFlow op(s), included in the given model, is(are) not supported by this interpreter.

[neps-smartfox]

Hi Everyone,

I'm trying to use ArmNN in my Rock PI N10. I was able to run the examples from the model zoo. Now I'm trying to load my own model but is currently having error.

I reproduced the error using the simple example from TensorFlow page. First I generated the TFLite model as follows.

import tensorflow as tf
print("TensorFlow version:", tf.__version__)

mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10)
])

predictions = model(x_train[:1]).numpy()
predictions

tf.nn.softmax(predictions).numpy()

loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
loss_fn(y_train[:1], predictions).numpy()
model.compile(optimizer='adam',
              loss=loss_fn,
              metrics=['accuracy'])
model.fit(x_train, y_train, epochs=1, verbose=0)
model.evaluate(x_test,  y_test, verbose=0)

input_spec = tf.TensorSpec([None, 28, 28], tf.int8)
# model._set_inputs(input_spec, training=False) # for tf < 2.2
model._saved_model_inputs_spec = None # for tf > 2.2
model._set_save_spec(input_spec) # for tf > 2.2

# Convert model from TF to TFLite
saved_model_path = "./"
tf.saved_model.save(model, saved_model_path)
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path)
converter.target_spec.supported_ops = [
 tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
 tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
tflite_model = converter.convert()
lite_model_name = "beginner.tflite"

with open(lite_model_name, 'wb') as f:
   f.write(tflite_model)

Then I tried to load the model using the TFLite delegate.

import tflite_runtime.interpreter as tflite

delegate = tflite.load_delegate(library="/home/rock/armnn_dl/libarmnnDelegate.so", options={"backends":"GpuAcc","logging-severity":"info"})
interpreter = tflite.Interpreter("beginner.tflite", experimental_delegates=[delegate])

However, I'm getting the following error.

Info: ArmNN v30.0.0
Info: Initialization time: 9.35 ms.
INFO: TfLiteArmnnDelegate: Created TfLite ArmNN delegate.
Traceback (most recent call last):
  File "simple_armnn.py", line 5, in <module>
    interpreter = tflite.Interpreter("beginner.tflite", experimental_delegates=[delegate])
  File "/home/rock/my_tflite/venv/lib/python3.7/site-packages/tflite_runtime/interpreter.py", line 496, in __init__
    delegate._get_native_delegate_pointer())  # pylint: disable=protected-access
RuntimeError: Select TensorFlow op(s), included in the given model, is(are) not supported by this interpreter. Make sure you apply/link the Flex delegate before inference. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-lite-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/guide/ops_selectNode number 0 (FlexCast) failed to prepare.
Info: Shutdown time: 0.76 ms.

Could you please advice about this error?

Does ArmNN have restrictions about the models it can run (e.g. tool version, quantization technique, etc)?

Thank you very much in advance.

[keidav01]

Hi @neps-smartfox, thank you for using ArmNN.

This appears to be more of an issue with how you are converting the model, rather than something with the delegate.

If the ArmNN Delegate does not support an op, it will fallback to Tensorflow Lite. This delegate method tends not to restrict you when ArmNN does not support something because of this fallback. If Tensorflow does not support it though, an issue would arise.

The issue appears to be that you are allowing the converter to select Tensorflow Operators as well. What happens if you do not enable this during conversion?

If this is a requirement, maybe you should play with the interpreters experimental flag experimental_op_resolver_type. By default, the builtin_op_resolver is used, so this could be where the issue is, since the builtin_ops do not contain select_tf_ops. You might be able to extend this resolver to look at tensorflow operations too. https://www.tensorflow.org/api_docs/python/tf/lite/Interpreter#args

Let me know how you get on. Though, I would recommend logging a ticket against Tensorflow as they will probably be able to provide you with a solution straight away.

Cheers, Keith

[neps-smartfox]

Hi Keith,

Thank you very much for you answer.

I am particulary interested to use the ArmNN delegate as much as possible. If it is a TFLite issue, then probably I need to ask in Tensorflow discussions.

As you've mentioned, if the ArmNN delegate doesn't support an op, it will fallback to Tensorflow Lite. Will there be a warning or an error message to give a clue on which op is not supported? There is a list that I found, but I'm not sure whether is it accurate or there are other existing documentation for this.

Then aside from the ops, are there also other requirements for Armnn(e.g. version from which the model is implemented/coverted from, or quantization technique used)?

Best Regards

[keidav01]

Hey @neps-smartfox,

No that list would not be relevant as that is for the ArmNN Serializer, rather than the ArmNN TF Lite Delegate. Here is the list: https://arm-software.github.io/armnn/latest/delegate.xhtml

I would convert on Tensorflow 2.5 as that is what we test on. I don't believe quantization technique is relevant.

As you have logging set to "Info" when setting the delegate options, you should will receive info from the delegate when it does not support an operation, yes.

However, it looks like you may need to link Flex first before it gets to this point.

Have you tried disabling tf_supported_ops during conversion yet?

[neps-smartfox]

Hi Keith,

I tried using tensorflow 2.5.0 instead of tflite_runtime 2.5.0. So my code now looks like the following.

import tensorflow as tf;

delegate = tf.lite.experimental.load_delegate(library="/home/rock/armnn_dl/libarmnnDelegate.so", options={"backends":"CpuAcc","logging-severity":"info"})
interpreter = tf.lite.Interpreter("beginner.tflite", experimental_delegates=[delegate])

This time I am now getting a new error.

Info: ArmNN v30.0.0
Info: Initialization time: 39.87 ms.
INFO: TfLiteArmnnDelegate: Created TfLite ArmNN delegate.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
INFO: Created TensorFlow Lite delegate for select TF ops.
INFO: TfLiteFlexDelegate delegate: 1 nodes delegated out of 4 nodes with 1 partitions.

Warning: The backend makes use of a deprecated interface to read constant tensors. If you are a backend developer please find more information in our doxygen documentation on github https://github.com/ARM-software/armnn under the keyword 'ConstTensorsAsInputs'.
Warning: The backend makes use of a deprecated interface to read constant tensors. If you are a backend developer please find more information in our doxygen documentation on github https://github.com/ARM-software/armnn under the keyword 'ConstTensorsAsInputs'.
Info: ArmnnSubgraph creation
Info: Parse nodes to ArmNN time: 4.31 ms
Warning: The backend makes use of a deprecated interface to read constant tensors. If you are a backend developer please find more information in our doxygen documentation on github https://github.com/ARM-software/armnn under the keyword 'ConstTensorsAsInputs'.
Warning: The backend makes use of a deprecated interface to read constant tensors. If you are a backend developer please find more information in our doxygen documentation on github https://github.com/ARM-software/armnn under the keyword 'ConstTensorsAsInputs'.
Info: Optimize ArmnnSubgraph time: 1.68 ms
Info: Load ArmnnSubgraph time: 761.35 ms
Info: Overall ArmnnSubgraph creation time: 771.12 ms

Exception ignored in: <function Delegate.__del__ at 0x7f5f740620>
Traceback (most recent call last):
  File "/home/rock/tf_250/venv/lib/python3.7/site-packages/tensorflow/lite/python/interpreter.py", line 125, in __del__
TypeError: item 1 in _argtypes_ has no from_param method

I am getting the same result even if I disabled SELECT_TF_OPS.

Seems like I was able to get past the Flex delegate issue. Do you have any advice regarding this error?

Thanks and Best Regards

[matthewsloyanARM]

Hi @neps-smartfox,

Thank you for getting in touch, just wondering if this is still an issue for you? Also, in our 22.11 we updated the TensorFlow version to 2.10, would you be able to try this out and see if it fixes your issues? You can checkout the TensorFlow version using this script in Arm NN armnn/scripts/get_tensorflow.sh.

Kind regards,

Matthew

[MikeJKelly]

Closed due to inactivity, if this is still an issue for you can you please reopen the issue or create a new one. Best regards, Mike.