Help to load HRNet model with OpenCV

[Suaro]

Hi,

First congratulate you on your amazing project. I have trained the network without problems and I was able to run the predict.py script on my webcam to check the operation of the model.

However, now I wanted to do a proof of concept to load this model with OpenCV DNN module. I have tried the following:

1. Load the model in SavedModel format directly using cv.dnn.readNetFromTensorflow ("./ exported")

2. Freeze the model and load the protobuf file with cv.dnn.readNetFromTensorflow ("frozen_graph.pb")

3. Convert the model to OpenVino format and load it with cv.dnn.readNetFromModelOptimizer ("./ frozen_graph.xml", "frozen_graph.bin")

All these attempts return the following error when executing net.forward ():

cv2.error: OpenCV (4.3.0-openvino) /home/suaro/libs/opencv/repo/opencv/modules/dnn/src/dnn.cpp:2887: error: (-215: Assertion failed)! layers [0 ] .outputBlobs.empty () in function 'allocateLayers'

I suppose the problem is that you have to apply some kind of modification to the model when optimizing it to be able to load it with OpenCV, but I am not sure where to start.

Could you help me?

Thanks a lot.

[yinguobing]

Hi, @Suaro

I find this issue very useful, according to which:

1. OpenCV do not support loading model from saved_model format directly.

2. Try freezing the graph with the following code:

import tensorflow as tf
from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2

loaded = tf.saved_model.load('my_model')
infer = loaded.signatures['serving_default']

f = tf.function(infer).get_concrete_function(input_1=tf.TensorSpec(shape=[None, 256, 256, 3], dtype=tf.float32))
f2 = convert_variables_to_constants_v2(f)
graph_def = f2.graph.as_graph_def()

# Export frozen graph
with tf.io.gfile.GFile('frozen_graph.pb', 'wb') as f:
   f.write(graph_def.SerializeToString())

3. Then make predictions like this:

import numpy as np
import cv2 as cv

net = cv.dnn.readNet('frozen_graph.pb')
inp = np.random.standard_normal([1, 3, 256, 256]).astype(np.float32)
net.setInput(inp)
out = net.forward()
print(out.shape)

output:

(1, 98, 64, 64)

For reference, I'm using TensorFlow 2.3.1 and OpenCV 4.5.0.

[Suaro]

Hi again,

Thank you very much for your prompt reply. The problem was apparently related to the OpenCV version in the case of .pb models. In version 4.5.0 it works correctly for me, but not in 4.3.0.

However, I have gotten it to work on version 4.3.0-openvino by converting the Tensorflow model to an OpenVino model.

Regards!

[yinguobing]

Glad to know :smile:

[tracysw]

import tensorflow as tf from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2

loaded = tf.saved_model.load('my_model') infer = loaded.signatures['serving_default']

f = tf.function(infer).get_concrete_function(input_1=tf.TensorSpec(shape=[None, 256, 256, 3], dtype=tf.float32)) f2 = convert_variables_to_constants_v2(f) graph_def = f2.graph.as_graph_def()

Export frozen graph

with tf.io.gfile.GFile('frozen_graph.pb', 'wb') as f: f.write(graph_def.SerializeToString())

这一步，转换模型。 报错，找了很久没有找到哪里的问题。能帮忙看下吗 TypeError: in user code:

/home/tensorflow/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1707 __call__  *
    return self._call_impl(args, kwargs)
/home/tensorflow/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1723 _call_impl  **
    raise structured_err
/home/tensorflow/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1716 _call_impl
    return self._call_with_structured_signature(args, kwargs,
/home/tensorflow/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1794 _call_with_structured_signature
    self._structured_signature_check_missing_args(args, kwargs)
/home/tensorflow/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1813 _structured_signature_check_missing_args
    raise TypeError("{} missing required arguments: {}".format(

TypeError: signature_wrapper(*, input_tensor) missing required arguments: input_tensor

[ghost]

import tensorflow as tf from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2

loaded = tf.saved_model.load('my_model') infer = loaded.signatures['serving_default']

f = tf.function(infer).get_concrete_function(input_1=tf.TensorSpec(shape=[None, 256, 256, 3], dtype=tf.float32)) f2 = convert_variables_to_constants_v2(f) graph_def = f2.graph.as_graph_def()

Export frozen graph

with tf.io.gfile.GFile('frozen_graph.pb', 'wb') as f: f.write(graph_def.SerializeToString())

这一步，转换模型。 报错，找了很久没有找到哪里的问题。能帮忙看下吗 TypeError: in user code:

/home/tensorflow/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1707 __call__  *
    return self._call_impl(args, kwargs)
/home/tensorflow/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1723 _call_impl  **
    raise structured_err
/home/tensorflow/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1716 _call_impl
    return self._call_with_structured_signature(args, kwargs,
/home/tensorflow/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1794 _call_with_structured_signature
    self._structured_signature_check_missing_args(args, kwargs)
/home/tensorflow/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1813 _structured_signature_check_missing_args
    raise TypeError("{} missing required arguments: {}".format(

TypeError: signature_wrapper(*, input_tensor) missing required arguments: input_tensor

和您一样的问题，请问现在解决了嘛？