Is there anyone used the model in inference mode by C++?

[lunasdejavu]

Since I have trained the model and validate it many times, now I need to read the model and detect the object in images I want in C++.

First, I found that it could read Mask_RCNN model by opencv, but it needed weights in pb and model in pbtxt, so I write something like

from keras.models import load_model
import tensorflow as tf
import os 
import os.path as osp
from keras import backend as K
from mrcnn import utils
import mrcnn.model as modellib
from mrcnn.config import Config
class ShapesConfig(Config):
    """Configuration for training on the toy shapes dataset.
    Derives from the base Config class and overrides values specific
    to the toy shapes dataset.
    """
    # Give the configuration a recognizable name
    NAME = "Newton Ring"

    # Train on 1 GPU and 8 images per GPU. We can put multiple images on each
    # GPU because the images are small. Batch size is 8 (GPUs * images/GPU).
    GPU_COUNT = 1
    IMAGES_PER_GPU = 1
    BACKBONE = "resnet50"
    # Number of classes (including background)
    NUM_CLASSES = 1 + 1  # background + 1 shapes

    # Use small images for faster training. Set the limits of the small side
    # the large side, and that determines the image shape.
    IMAGE_MIN_DIM = 1024
    IMAGE_MAX_DIM = 1024
    qq = 4
    # Use smaller anchors because our image and objects are small
    RPN_ANCHOR_SCALES = (8 * qq, 16 * qq, 32 * qq, 64 * qq, 128 * qq) #(32, 64, 128, 256, 512)  # anchor side in pixels

    # Reduce training ROIs per image because the images are small and have
    # few objects. Aim to allow ROI sampling to pick 33% positive ROIs.
    # TRAIN_ROIS_PER_IMAGE = 512

    # Use a small epoch since the data is simple
    STEPS_PER_EPOCH = 841

    # use small validation steps since the epoch is small
    VALIDATION_STEPS = 105

    # iter_num 
    iter_num = 0
    #basic config
    LEARNING_RATE= 0.001
    # IMAGE_RESIZE_MODE = "crop"
    # MEAN_PIXEL = np.array([128, 128, 128])
    MAX_GT_INSTANCES = 3
    # MINI_MASK_SHAPE = (56, 80)
    DA_NUMBER_PER_IMAGE=20
    DA_TYPEC= False
    REMOVE_DARKRAINBOW=False
    # IMAGE_CHANNEL_COUNT = 1
    # MEAN_PIXEL = 1

config = ShapesConfig()

input_path = 'input path'
weight_file = 'weight.h5'
weight_file_path = osp.join(input_path,weight_file)
weight_file_path ='C:/Mask_RCNN/logs/newton ring20181226T1139/mask_rcnn_newton ring_0001.h5'
output_graph_name = 'model.pb'

def h5_to_pb(h5_model,output_dir,model_name,out_prefix = "output_",log_tensorboard = True):
    if osp.exists(output_dir) == False:
        os.mkdir(output_dir)
    out_nodes = []
    for i in range(len(h5_model.outputs)):
        out_nodes.append(out_prefix + str(i + 1))
        tf.identity(h5_model.output[i],out_prefix + str(i + 1))
    sess = K.get_session()
    from tensorflow.python.framework import graph_util,graph_io
    init_graph = sess.graph.as_graph_def()
    main_graph = graph_util.convert_variables_to_constants(sess,init_graph,out_nodes)
    graph_io.write_graph(main_graph,output_dir,name = model_name,as_text = False)
    if log_tensorboard:
        from tensorflow.python.tools import import_pb_to_tensorboard
        import_pb_to_tensorboard.import_to_tensorboard(osp.join(output_dir,model_name),output_dir)

output_dir = osp.join(os.getcwd(),"trans_model")
output_dir = "D:/"

ROOT_DIR = os.path.abspath("C:/Mask_RCNN/")
MODEL_DIR = os.path.join(ROOT_DIR, "logs")
model = modellib.MaskRCNN(mode="inference", config=config,
                          model_dir=MODEL_DIR)
# model.keras_model.summary()
# print(model.keras_model.inputs)
# print(model.keras_model.outputs)

model.load_weights(weight_file_path, by_name=True)
h5_model = model.keras_model
print(len(h5_model.outputs))
h5_to_pb(h5_model,output_dir = output_dir,model_name = output_graph_name)
print('model saved')

then I turned the pb file to pbtxt

import tensorflow as tf
from tensorflow.python.platform import gfile
from google.protobuf import text_format

def convert_pb_to_pbtxt(filename):
  with gfile.FastGFile(filename,'rb') as f:
    graph_def = tf.GraphDef()

    graph_def.ParseFromString(f.read())

    tf.import_graph_def(graph_def, name='')

    tf.train.write_graph(graph_def, './', 'protobuf.pbtxt', as_text=True)
  return

def convert_pbtxt_to_pb(filename):
  """Returns a `tf.GraphDef` proto representing the data in the given pbtxt file.
  Args:
    filename: The name of a file containing a GraphDef pbtxt (text-formatted
      `tf.GraphDef` protocol buffer data).
  """
  with tf.gfile.FastGFile(filename, 'r') as f:
    graph_def = tf.GraphDef()

    file_content = f.read()

    # Merges the human-readable string in `file_content` into `graph_def`.
    text_format.Merge(file_content, graph_def)
    tf.train.write_graph( graph_def , './' , 'protobuf.pb' , as_text = False )
  return
input_path = "D:/model.pb"
convert_pb_to_pbtxt(input_path)

then there was an error like [libprotobuf ERROR C:\build\3_4_winpack-build-win64-vc14\opencv\3rdparty\protobu f\src\google\protobuf\text_format.cc:288] Error parsing text-format opencv_tenso rflow.GraphDef: 41208:5: Unknown enumeration value of "DT_RESOURCE" for field "t ype". OpenCV(3.4.4) Error: Unspecified error (FAILED: ReadProtoFromTextFile(param_file , param). Failed to parse GraphDef file: D:/model_1.pbtxt) in cv::dnn::ReadTFNet ParamsFromTextFileOrDie, file C:\build\3_4_winpack-build-win64-vc14\opencv\modul es\dnn\src\tensorflow\tf_io.cpp, line 54

then I found the models are probably different, so is there anyone can give me some guides to do inference mode in C++?

[oriolorra]

Hi,

You could use Boost.Python libraries, as I have done. These libraries enable you to do call python methods from C++.

But, you must have a Python Runtime running in the C++ exe, afterward you could load python files with methods inside (obviously jeje) and call them. Also, you are able to pass through boost::python::objects between C++<->Python.

There is a lot of info about it on the Internet if you search a little bit.

Regards,

[lunasdejavu]

@oriolorra thanks a lot! I am trying to figure out how to do that now. But since this project needs keras_model() and python is a dynamic language, how do we link tensorflow and keras and compile it after all? I am not quite sure about what Python Runtime is for.

[lunasdejavu]

I found the opencv forum administor told me that their api can't work on this project. I think I might try the tensorflow C++ version first.

[lunasdejavu]

I found this project is really helpful https://github.com/Neargye/hello_tf_c_api easy to establish the environment to load the model by tensorflow c++ api.

[CasonTsai]

yes,i did

[buaacarzp]

how can I genetate a pbtxtx file to inference in the opencv

[121649982]

hello,have you sloved this problem?can you tell me how to translate the model to .pb?

[CasonTsai]

this is my code implementation https://github.com/CasonTsai/MaskRcnn_tensorflow_cpp_inference

[HKun1997]

@CasonTsai Hello, have you try to use the .pb and .pbtxt to detect in Opencv4(C++)?

[CasonTsai]

@CasonTsai Hello, have you try to use the .pb and .pbtxt to detect in Opencv4(C++)?

sorry,i don't inference it by c++ with Opencv4. I just transformed the keras to tensorflow model and then used tensorflow c++ api to inferenced it

[HKun1997]

@CasonTsai Hello, have you try to use the .pb and .pbtxt to detect in Opencv4(C++)?

sorry,i don't inference it by c++ with Opencv4. I just transformed the keras to tensorflow model and then used tensorflow c++ api to inferenced it

Ok, thx a lot!