ONNX conversion of the Recognition Model

[hardik-ajmani]

Hey there,

Keras-OCR model gives an appreciable accuracy in most of the use cases, kudos for that!

But, we are trying to use pre-trained Keras-OCR on Jetson Nano employing NVIDIA Deepstream. And, the initial steps require the conversion of the model into ONNX representation. Ref - https://github.com/onnx/keras-onnx/tree/master/keras2onnx

Initially, the conversion was successful. But, while converting the model further into TensorRt gave two errors -

1. Unsupported operation of tf.linspace
2. Unsupported datatype of FP32 Both the issues were resolved by making some changes in _meshgrid method of recognition.py as shown below:

   #x_linspace = tf.linspace(-1., 1., width) (removed these lines and used **np**)
   #y_linspace = tf.linspace(-1., 1., height)
   x_linspace = np.linspace(-1., 1., width, dtype='float16')
   y_linspace = np.linspace(-1., 1., height, dtype='float16')

But, now the model doesn't compile correctly on running this command: r = recognition.Recognizer()

and throws these lines of error (complete traceback):

WARNING:tensorflow:From /tensorflow-

1.15.2/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords)
    527                 as_ref=input_arg.is_ref,
--> 528                 preferred_dtype=default_dtype)
    529           except TypeError as err:

15 frames
/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, ctx, accepted_result_types)
   1272           "Tensor conversion requested dtype %s for Tensor with dtype %s: %r" %
-> 1273           (dtype.name, value.dtype.name, value))
   1274     return value

ValueError: Tensor conversion requested dtype float16 for Tensor with dtype float32: <tf.Tensor 'lstm_10/split:0' shape=(128, 128) dtype=float32>

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-5-745942188b24> in <module>()
----> 1 r = recognition.Recognizer()

/content/keras-ocr/keras_ocr/recognition.py in __init__(self, alphabet, weights, build_params)
    323         self.blank_label_idx = len(alphabet)
    324         self.backbone, self.model, self.training_model, self.prediction_model = build_model(
--> 325             alphabet=alphabet, **build_params)
    326         if weights is not None:
    327             weights_dict = PRETRAINED_WEIGHTS[weights]

/content/keras-ocr/keras_ocr/recognition.py in build_model(alphabet, height, width, color, filters, rnn_units, dropout, rnn_steps_to_discard, pool_size, stn)
    263                                       kernel_initializer="he_normal",
    264                                       return_sequences=True,
--> 265                                       name='lstm_10')(x)
    266     rnn_1_back = keras.layers.LSTM(rnn_units[0],
    267                                    kernel_initializer="he_normal",

/tensorflow-1.15.2/python3.6/tensorflow_core/python/keras/layers/recurrent.py in __call__(self, inputs, initial_state, constants, **kwargs)
    621 
    622     if initial_state is None and constants is None:
--> 623       return super(RNN, self).__call__(inputs, **kwargs)
    624 
    625     # If any of `initial_state` or `constants` are specified and are Keras

/tensorflow-1.15.2/python3.6/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    852                     outputs = base_layer_utils.mark_as_return(outputs, acd)
    853                 else:
--> 854                   outputs = call_fn(cast_inputs, *args, **kwargs)
    855 
    856             except errors.OperatorNotAllowedInGraphError as e:

/tensorflow-1.15.2/python3.6/tensorflow_core/python/keras/layers/recurrent.py in call(self, inputs, mask, training, initial_state)
   2547     self.cell.reset_recurrent_dropout_mask()
   2548     return super(LSTM, self).call(
-> 2549         inputs, mask=mask, training=training, initial_state=initial_state)
   2550 
   2551   @property

/tensorflow-1.15.2/python3.6/tensorflow_core/python/keras/layers/recurrent.py in call(self, inputs, mask, training, initial_state, constants)
    754         input_length=input_length,
    755         time_major=self.time_major,
--> 756         zero_output_for_mask=self.zero_output_for_mask)
    757     if self.stateful:
    758       updates = []

/tensorflow-1.15.2/python3.6/tensorflow_core/python/keras/backend.py in rnn(step_function, inputs, initial_states, go_backwards, mask, constants, unroll, input_length, time_major, zero_output_for_mask)
   3929     # the value is discarded.
   3930     output_time_zero, _ = step_function(
-> 3931         input_time_zero, tuple(initial_states) + tuple(constants))
   3932     output_ta = tuple(
   3933         tensor_array_ops.TensorArray(

/tensorflow-1.15.2/python3.6/tensorflow_core/python/keras/layers/recurrent.py in step(inputs, states)
    730       def step(inputs, states):
    731         states = states[0] if len(states) == 1 and is_tf_rnn_cell else states
--> 732         output, new_states = self.cell.call(inputs, states, **kwargs)
    733         if not nest.is_sequence(new_states):
    734           new_states = [new_states]

/tensorflow-1.15.2/python3.6/tensorflow_core/python/keras/layers/recurrent.py in call(self, inputs, states, training)
   2229       k_i, k_f, k_c, k_o = array_ops.split(
   2230           self.kernel, num_or_size_splits=4, axis=1)
-> 2231       x_i = K.dot(inputs_i, k_i)
   2232       x_f = K.dot(inputs_f, k_f)
   2233       x_c = K.dot(inputs_c, k_c)

/tensorflow-1.15.2/python3.6/tensorflow_core/python/keras/backend.py in dot(x, y)
   1695     out = sparse_ops.sparse_tensor_dense_matmul(x, y)
   1696   else:
-> 1697     out = math_ops.matmul(x, y)
   1698   return out
   1699 

/tensorflow-1.15.2/python3.6/tensorflow_core/python/util/dispatch.py in wrapper(*args, **kwargs)
    178     """Call target, and fall back on dispatchers if there is a TypeError."""
    179     try:
--> 180       return target(*args, **kwargs)
    181     except (TypeError, ValueError):
    182       # Note: convert_to_eager_tensor currently raises a ValueError, not a

/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/math_ops.py in matmul(a, b, transpose_a, transpose_b, adjoint_a, adjoint_b, a_is_sparse, b_is_sparse, name)
   2752     else:
   2753       return gen_math_ops.mat_mul(
-> 2754           a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
   2755 
   2756 

/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/gen_math_ops.py in mat_mul(a, b, transpose_a, transpose_b, name)
   6134   _, _, _op = _op_def_lib._apply_op_helper(
   6135         "MatMul", a=a, b=b, transpose_a=transpose_a, transpose_b=transpose_b,
-> 6136                   name=name)
   6137   _result = _op.outputs[:]
   6138   _inputs_flat = _op.inputs

/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords)
    562                   "%s type %s of argument '%s'." %
    563                   (prefix, dtypes.as_dtype(attrs[input_arg.type_attr]).name,
--> 564                    inferred_from[input_arg.type_attr]))
    565 
    566           types = [values.dtype]

TypeError: Input 'b' of 'MatMul' Op has type float32 that does not match type float16 of argument 'a'.

Request your help/guidance on this or any other alternative for converting the model for Jetson Nano. Thank you :)

[csmcallister]

@hardik-ajmani I'm in nearly the same boat as you, trying to convert the recognizer to ONNX using both keras-onnx and tf2onnx.

My system info:

• Python 3.7.4
• Windows 10 (build 1809)
• keras_ocr==0.6.3
• tensorflow-gpu==2.1.0
• onnxruntime==1.3.0
• keras2onnc==1.70
• onnx==1.7.0
• tf2onnx==1.6.2

However, I was able to make your suggested changes to _meshgrid() in the recognition model:

#x_linspace = tf.linspace(-1., 1., width) (removed these lines and used **np**)
#y_linspace = tf.linspace(-1., 1., height)
x_linspace = np.linspace(-1., 1., width, dtype='float16')
y_linspace = np.linspace(-1., 1., height, dtype='float16')

And then successfully instantiate the model:

import keras_ocr
recognizer = keras_ocr.recognition.Recognizer()

However, when trying to convert to ONNX with this code:

import keras2onnx
import keras_ocr
import onnxruntime as rt

recognizer = keras_ocr.recognition.Recognizer()
onnx_model = keras2onnx.convert_keras(recognizer.model, recognizer.model.name)
content = onnx_model.SerializeToString()
sess = rt.InferenceSession(content)

I get a dtype error when trying to start an onnx runtime session:

Traceback (most recent call last):
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 468, in _apply_op_helper
    preferred_dtype=default_dtype)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1290, in convert_to_tensor
    (dtype.name, value.dtype.name, value))
ValueError: Tensor conversion requested dtype float32 for Tensor with dtype float16: <tf.Tensor 'lambda_1/Reshape_5:0' shape=(None, 3, None) dtype=float16>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "recon_convert.py", line 15, in <module>
    main()
  File "recon_convert.py", line 7, in main
    recognizer = keras_ocr.recognition.Recognizer()
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\keras_ocr\recognition.py", line 325, in __init__
    alphabet=alphabet, **build_params)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\keras_ocr\recognition.py", line 254, in build_model
    output_shape=stn_input_output_shape)([x, localization_net(x)])
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 773, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\tensorflow_core\python\keras\layers\core.py", line 846, in call
    result = self.function(inputs, **kwargs)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\keras_ocr\recognition.py", line 95, in _transform
    transformed_grid = tf.matmul(locnet_y, indices_grid)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\tensorflow_core\python\util\dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\tensorflow_core\python\ops\math_ops.py", line 2760, in matmul
    a, b, adj_x=adjoint_a, adj_y=adjoint_b, name=name)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\tensorflow_core\python\ops\gen_math_ops.py", line 1526, in batch_mat_mul_v2
    "BatchMatMulV2", x=x, y=y, adj_x=adj_x, adj_y=adj_y, name=name)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 504, in _apply_op_helper
    inferred_from[input_arg.type_attr]))
TypeError: Input 'y' of 'BatchMatMulV2' Op has type float16 that does not match type float32 of argument 'x'.

If I go back into recognition.py and set the dtypes to float32:

x_linspace = np.linspace(-1., 1., width, dtype='float32')
y_linspace = np.linspace(-1., 1., height, dtype='float32')

I'm still able to instantiate the recognizer and convert with keras2onnx. However, I get an invalid graph error:

Traceback (most recent call last):
  File "recon_convert.py", line 15, in <module>
    main()
  File "recon_convert.py", line 11, in main
    sess = rt.InferenceSession(content)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\onnxruntime\capi\session.py", line 158, in __init__
    self._load_model(providers)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\onnxruntime\capi\session.py", line 177, in _load_model
    self._sess.load_model(providers)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Error in Node:lambda_1/AddN_add : Node (lambda_1/AddN_add) has input size 4 not in range [min=2, max=2].

I tried using the onnx model checker to debug, but it naturally crashes Python:

import onnx
onnx.checker.check_model(onnx_model)

But retrying using the onnx shape inference method gets past the error:

from onnx import shape_inference
...
inferred_model = shape_inference.infer_shapes(onnx_model)
content = inferred_model.SerializeToString()
sess = rt.InferenceSession(content)

But only to raise a new error:

Traceback (most recent call last):
  File "recon_convert.py", line 21, in <module>
    main()
  File "recon_convert.py", line 17, in main
    sess = rt.InferenceSession(content)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\onnxruntime\capi\session.py", line 158, in __init__
    self._load_model(providers)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\onnxruntime\capi\session.py", line 177, in _load_model
    self._sess.load_model(providers)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Node:permute/Identity:0_cropping Output:permute/Identity:0_cropping0 [ShapeInferenceError] Can't merge shape info. Both source and target dimension have values but they differ. Source=31 Target=0 Dimension=2

Incidentally, this is the same error I get when using the tf2onnx python CLI (python -m tf2onnx.convert --saved-model saved_recognizer_model --opset 11 --output recognizer.onnx) to convert the model to an onnx file, which is then loaded into the onnx runtime:

import onnx
import onnxruntime as rt
from onnx import helper, shape_inference
onnx_model_path = 'ocr/recognizer.onnx'
onnx_model = onnx.load(onnx_model_path)
inferred_model = shape_inference.infer_shapes(onnx_model)
content = inferred_model.SerializeToString()
sess = rt.InferenceSession(content)

And it's this same ShapeInferenceError, but for a different node:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\onnxruntime\capi\session.py", line 158, in __init__
    self._load_model(providers)
  File "C:\Users\scottmcallister\anaconda3\envs\xact_model\lib\site-packages\onnxruntime\capi\session.py", line 177, in _load_model
    self._sess.load_model(providers)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Node:StatefulPartitionedCall/model_2/lambda/strided_slice Output:StatefulPartitionedCall/model_2/lambda/strided_slice:0 [ShapeInferenceError] Can't merge shape info. Both source and target dimension have values but they differ. Source=31 Target=0 Dimension=2

Any thoughts or have you made progress since creating this issue?

[hardikajmani]

Hey @csmcallister ! Glad to have you on the same page.

I was using Google Colab till now, I think the aforementioned issue (initially raised by me) would have been due to that. Would try running it locally now.

Not much, only progress from my side is, that the same conversion would have to be done for text-detector model as well. As all the detected text in an image is fed into the recognition model.

Although the detection model converts without any issue, while parsing it throws the following issue

ERROR: builtin_op_importers.cpp:2501 In function importResize:
[8] Assertion failed: scales.is_weights() && "Resize scales must be an initializer!"

Please let me know if you get any updates on the conversion of either of the model. Cheers!

[hardik-ajmani]

@csmcallister greetings!

A quick update: Using tf2onnx I was able to eliminate linspace error, without modifying the code.

I converted the model into .pb file and then, used the following code to convert the model. !python -m tf2onnx.convert --opset 11 --input /content/keras-ocr/keras_ocr/model.pb --inputs input_1:0 --outputs decode/PadV2:0 --output model_tf.onnx

But it throws the following output -

2020-07-27 03:50:57.666890: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2020-07-27 03:50:58,966 - INFO - Using tensorflow=1.15.2, onnx=1.7.0, tf2onnx=1.6.2/8d5253
2020-07-27 03:50:58,967 - INFO - Using opset <onnx, 11>
2020-07-27 03:51:03,337 - ERROR - Tensorflow op [decode/CTCGreedyDecoder: CTCGreedyDecoder] is not supported
2020-07-27 03:51:03,337 - ERROR - Tensorflow op [decode/SparseToDense: SparseToDense] is not supported
2020-07-27 03:51:03,351 - ERROR - Unsupported ops: Counter({'CTCGreedyDecoder': 1, 'SparseToDense': 1})
2020-07-27 03:51:03,408 - INFO - Optimizing ONNX model
2020-07-27 03:51:08,136 - INFO - After optimization: Cast -31 (96->65), Concat -6 (20->14), Const -235 (372->137), Expand -6 (11->5), Identity -109 (114->5), Less -2 (9->7), Reshape +1 (16->17), Shape -9 (22->13), Slice -5 (21->16), Squeeze -9 (21->12), Sub -4 (12->8), Transpose -26 (45->19), Unsqueeze -33 (48->15)
2020-07-27 03:51:08,183 - INFO - 
2020-07-27 03:51:08,183 - INFO - Successfully converted TensorFlow model /content/keras-ocr/keras_ocr/model.pb to ONNX
2020-07-27 03:51:08,225 - INFO - ONNX model is saved at model_tf.onnx

Please let me know if you have got any updates on the same or any suggestions in the workflow?

PS - You may view my notebook here https://colab.research.google.com/drive/1u2rzxD-wSqK2hGlHWPjJDi5TMtC4ysjN?usp=sharing

[sayakpaul]

Glad to have found this thread. If converting to TensorRT compatible graph is the end goal then this article might be useful.

I am looking into converting the models (both the detection and the recognition) to .tflite format.

Furthermore, I think if the issues are stemming from the ONNX conversion step it likely could be an issue of op versions. So, it might be better suited for the ONNX repo.

[Seioch]

Hi,

I'm attempting to do something similar with my project. Have you had any successes on your end?

[tulasiram58827]

Hi guys With the help of @sayakpaul I succesfully converted both detection and recognition models to TFLite Format.

• Detection Model(CRAFT) You can find both ONNX and TFLite models in this Repository
• Recognition Model You can find the TFLite models in this Repository.

[Asad-Raza]

I am able to convert to onnx model using tf2onnx.convert.

import tensorflow as tf import keras_ocr

d=keras_ocr.detection.Detector(weights='clovaai_general') tf.saved_model.save(obj=d.model, export_dir='Data')

!pip install -U tf2onnx

!python -m tf2onnx.convert --saved-model 'Data' --output 'Data/craft_mlt_25k.onnx' --opset 11

[Asad-Raza]

The generated ONNX model is working good with onnxruntime

[as1605]

Hi, has anyone been able to run the ONNX model on Triton? I have loaded the model, but how do we configure the input and output? It apparently takes a 64 dimension input and gives a 97 dimension output

Here's what Triton is showing for the config

{
  "name": "easyocr",
  "versions": ["1"],
  "platform": "onnxruntime_onnx",
  "inputs": [
    {
      "name": "input1",
      "datatype": "FP32",
      "shape": [1, 1, 64, -1]
    }
  ],
  "outputs": [
    {
      "name": "output",
      "datatype": "FP32",
      "shape": [1, -1, 97]
    }
  ]
}