[Feature] IsFinite op cannot be converted to Inference Engine IR (for TF 2.4.1)

puitk-olp commented 2 years ago

System information (version)

OpenVINO=> 2022.1
Operating System / Platform => Linux Ubuntu 2020.04 LTS
Compiler => Model Optimizer
Problem classification => Model Conversion
Framework => TensorFlow 2.4.1
Model name: custom model created with Keras (with MultiHeadAttention layers)

Detailed description

I work with custom Keras model with MultiHeadattention layers and I have been trying to convert it to IR with Model Optimizer. I get the Error in convertion that some operations coannot be converted (IsFinite). This operation is a part of ReduceLogSumExp op, and that one is part of Softmax op.

Softmax is listed as supported op on https://docs.openvino.ai/2022.1/openvino_docs_MO_DG_prepare_model_Supported_Frameworks_Layers.html however ReduceLogSumExp and IsFinite are not.

Softmax layer is widely used in different types of NN models and it's suprising to me, that convertion cannot be done (for TF2)

I checked Tensorflow source code, and it really uses functions: reduce_logsumexp and is_finite to calculate Softmax in all versions of TF2 (2.4.1 and above).

My questions are: 1) May the problem come from inproper convertion from h5 to saved_model format or this convertion changed (somehow) architecture of the model? Maybe I do something wrong. 2) Are there any ways to overcome the problem of unsupported IsFinite operation in OpenVINO (best with pure Python implementations) - in previous versions of OV it was possible to offload some operations and/or subgraph (with --tensorflow_operation_patterns or --tensorflow_subgraph_patterns options of MO) now not? 3) Is it possible to add support of IsFinite op (for TF2) in future releases of OpenVINO?

Steps to reproduce

Convertion from h5 to saved_model format:

(openvino_env) $ python
>>> import tensorflow as tf
>>> m = tf.keras.models.load_model("6-4_SA_CMH_H3.h5")
>>> m.save('6-4_SA_CMH_H3')

Model's compilation with Model Optimizer:

(openvino_env) $ mo --saved_model_dir=./ --framework=tf --log_level=ERROR --input=input0 --input_shape [1,25,513,6] --disable_nhwc_to_nchw
[ WARNING ]  Use of deprecated cli option --disable_nhwc_to_nchw detected. Option use in the following releases will be fatal.
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      None
        - Path for generated IR:        /home/pablo/AudioProcessing/ContentEnhancement/VVM/Python_RefOLS/LocFiles/6-4_SA_CMH_H3/.
        - IR output name:       saved_model
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         input0
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         [1,25,513,6]
        - Source layout:        Not specified
        - Target layout:        Not specified
        - Layout:       Not specified
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - User transformations:         Not specified
        - Reverse input channels:       False
        - Enable IR generation for fixed input shape:   False
        - Use the transformations config file:  None
Advanced parameters:
        - Force the usage of legacy Frontend of Model Optimizer for model conversion into IR:   False
        - Force the usage of new Frontend of Model Optimizer for model conversion into IR:      False
TensorFlow specific parameters:
        - Input model in text protobuf format:  False
        - Path to model dump for TensorBoard:   None
        - List of shared libraries with TensorFlow custom layers implementation:        None
        - Update the configuration file with input/output node names:   None
        - Use configuration file used to generate the model with Object Detection API:  None
        - Use the config file:  None
OpenVINO runtime found in:      /home/pablo/openvino_env/lib/python3.8/site-packages/openvino
OpenVINO runtime version:       2022.1.0-7019-cdb9bec7210-releases/2022/1
Model Optimizer version:        2022.1.0-7019-cdb9bec7210-releases/2022/1
[ WARNING ]  Please set `version` attribute for node StatefulPartitionedCall/model/att00/softmax/ReduceLogSumExp/IsFinite with type=<UNKNOWN>
[ WARNING ]  Changing Const node '459' data type from int64 to <class 'numpy.float32'> for Mul operation
[ ERROR ]  List of operations that cannot be converted to Inference Engine IR:
[ ERROR ]      IsFinite (1)
[ ERROR ]          StatefulPartitionedCall/model/att00/softmax/ReduceLogSumExp/IsFinite
[ ERROR ]  Part of the nodes was not converted to IR. Stopped.

I also tried to load model with custom objects configuration:

>>>  m = tf.keras.models.load_model("6-4_SA_CMH_H3.h5", custom_objects={'MultiHeadAttention': tf.keras.layers.MultiHeadAttention})

but Model Optimizer produced the same errors during convertion.

Issue submission checklist

[ ] I report the issue, it's not a question
[x] I checked the problem with documentation, FAQ, open issues, Stack Overflow, etc and have not found solution
[x] There is reproducer code and related data files: images, videos, models, etc.

andrei-kochin commented 2 years ago

Hello @puitk-olp,

Thank you for reaching OpenVINO! As you've already noticed IsFinite is not supported yet by ModelOptimizer but it is in our backlog.

Meanwhile you can try to prepare extension on your own to make it working. Pleas try following: https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Customize_Model_Optimizer.html#model-optimizer-extensions

Thank you!

andrei-kochin commented 2 years ago

@puitk-olp By the way as you see in the output --disable_nhwc_to_nchw is optional now and may not be specified which will reduce your cmd line a bit.

Also input_shape can be left unspecified as well so you cmd might be even like: mo --saved_model_dir=./ --log_level=ERROR --input=input0

rkazants commented 2 years ago

@puitk-olp, I can suggest you a quick workaround to replace IsFinite operation to IsNonEqual with two inputs (data and Constant with floating-point infinite value) using front transformation. Of course, this transformation will work if you don't have nan values in the data. Example of the front transformation can be found here openvino\tools\mo\openvino\tools\mo\front\Log1p.py.

puitk-olp commented 2 years ago

Hello @puitk-olp,

Thank you for reaching OpenVINO! As you've already noticed IsFinite is not supported yet by ModelOptimizer but it is in our backlog.

Meanwhile you can try to prepare extension on your own to make it working. Pleas try following: https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Customize_Model_Optimizer.html#model-optimizer-extensions

Thank you!

Thank you (@andrei-kochin), for your all suggestions. I'll try to prepare my extension.

I'd like to ask one more thing, if the lack of support for the IsFinite operation also implies the lack of support for Softmax (IsFinite is used by the TF function in calculation of Softmax) for models defined in Keras (then converted to saved_model format, before converting to IR)?

BR

puitk-olp commented 2 years ago

@puitk-olp By the way as you see in the output --disable_nhwc_to_nchw is optional now and may not be specified which will reduce your cmd line a bit.

Also input_shape can be left unspecified as well so you cmd might be even like: mo --saved_model_dir=./ --log_level=ERROR --input=input0

When omitted --disable_nhwc_to_nchw, I get the dimension mismatch for some layer during convertion to IR, so it's necessary in my case.

puitk-olp commented 2 years ago

@puitk-olp, I can suggest you a quick workaround to replace IsFinite operation to IsNonEqual with two inputs (data and Constant with floating-point infinite value) using front transformation. Of course, this transformation will work if you don't have nan values in the data. Example of the front transformation can be found here openvino\tools\mo\openvino\tools\mo\front\Log1p.py.

@rkazants, in my case op IsFinite is not directly used by me, but by layer Softmax (TF uses IsFinite in some function which calculates Softmax), so it will be difficult to replace it. My model is originally defined in Keras, and I'm not sure what is introduced in it while saving H5 model to TF saved_model format (required by MO).

andrei-kochin commented 2 years ago

@puitk-olp starting from 2022.1 --disable_nhwc_to_nchw is optional and should not make any difference to your graph. Does it really make a change to a graph?

Softmax should be supported according to FWK supported layers and should be represented as OpenVINO Softmax.

puitk-olp commented 2 years ago

@andrei-kochin Softmax should be supported according to FWK supported layers and should be represented as OpenVINO Softmax.

I've probably gotten it. There are two Softmax definitions in Keras: keras.activations.softmax and keras.layers.Softmax and their implementations differ:

keras.activations.softmax uses tf.nn.softmax() and then inside: tf.exp() and tf.reduce_sum()
keras.layers.Softmax uses tf.exp() and tf.reduce_logsumexp(), and the last one function uses tensorflow.python.ops.is_finite() - and that one is not supported in OV.

Summarizing, I suspect that in OpenVINO, Softmax is supported as an activation function for other layer, and not as a separated layer. Could it be the case?

rkazants commented 2 years ago

@puitk-olp, you are correct. Very often Keras operations are expressed as the decomposition of several TensorFlow operations and the softmax operation is a case. In your case, you can work around this by using another softmax, this way is simper than implementing transformation for IsFinite.

andrei-kochin commented 2 years ago

Hello @puitk-olp,

Do you have any news for us? have you succeed with workaround suggested?

puitk-olp commented 2 years ago

Hello @andrei-kochin, I asked model's creators (I am only an user of the model) to use another softmax function in it, but I haven't received any feedback from them yet.

avitial commented 2 years ago

Closing this. Feel free to reopen and provide additional information or ask any questions related to this topic.

openvinotoolkit / openvino