How to convert .ckpt model to SavedModel .pb format for hosting with Tensorflow Model Serving?

[igorvishnevskiy]

Models trained by this pipeline perform great. But how to host them using Tesorflow Model Serving? Checkpoint needs to be converted into SavedModel (.pb) format.

What I've done so far is:

1. I've modified following method to provide model with an output tensor. The line of code I added is: logits = tf.nn.softmax(logits, name="softmax_tensor")

def _get_image_info( features, mode ):
    """Calculates the logits and sequence length"""

    image = features['image']
    width = features['width']

    conv_features,sequence_length = model.convnet_layers(image,
                                                         width,
                                                         mode)

    logits = model.rnn_layers(conv_features, sequence_length,
                              charset.num_classes())

    logits = tf.nn.softmax(logits, name="softmax_tensor")

    return logits, sequence_length 

2. Then following MNIST eample, I have added a simple serving input receiver function:

def serving_input_receiver_fn():
    """
    This is used to define inputs to serve the model.
    :return: ServingInputReciever
    """
    reciever_tensors = {
        # The size of input image is flexible.
        'image': tf.placeholder(tf.float32, [None, None, None, 1]),
        'width': tf.placeholder(tf.int32, [None, 1]),
        'length': tf.placeholder(tf.int64, [None, 1]),
        'text': tf.placeholder(tf.string, [None,]),
    }

    # Convert give inputs to adjust to the model.
    features = {
        # Resize given images.
        'image': tf.image.resize_images(reciever_tensors['image'], [28, 28]),
        'width': tf.shape(reciever_tensors['image'])[1],
    }
    return tf.estimator.export.ServingInputReceiver(receiver_tensors=reciever_tensors,
                                                    features=features)

3. Next in train.py, I have added: classifier.export_saved_model(saved_dir, serving_input_receiver_fn=model_fn.serving_input_receiver_fn)

4. After that, when I tried to train, I received following error: TypeError: Expected labels (first argument) to be a SparseTensor

5.To fix that, in model.py, I modified following method, where I converted "sequence_labels" from dense to sparse tensor.

def ctc_loss_layer( rnn_logits, sequence_labels, sequence_length,
                    reduce_mean=True ):
    """Build CTC Loss layer for training"""
    labels_sparse = dense_to_sparse(sequence_labels, sparse_val=0)
    losses = tf.nn.ctc_loss( labels_sparse,
                             rnn_logits, 
                             sequence_length,
                             time_major=True, 
                             ignore_longer_outputs_than_inputs=True )
    if (reduce_mean):
        loss = tf.reduce_mean( losses )
    else:
        loss = tf.reduce_sum( losses )

    return loss

def dense_to_sparse(dense_tensor, sparse_val=0):
    with tf.name_scope("dense_to_sparse"):
        sparse_inds = tf.where(tf.not_equal(dense_tensor, sparse_val),
                               name="sparse_inds")
        sparse_vals = tf.gather_nd(dense_tensor, sparse_inds,
                                   name="sparse_vals")
        dense_shape = tf.shape(dense_tensor, name="dense_shape",
                               out_type=tf.int64)
        return tf.SparseTensor(sparse_inds, sparse_vals, dense_shape)

6. Now I am facing the next exception: ValueError: Tried to convert 'x' to a tensor and failed. Error: None values not supported.

If anyone tried to convert the model by this pipeline to SavedModel for hosting with Tensorflow Model Serving, all help is welcome. Thank you. This pipeline is generating very good accuracy. We need to add handling for SavedModel conversion so we could host it using Tensorflow Model Serving. So far I've been unsuccessful, but going in the right direction. I think collaboratively we can do it faster. Thank you for your help.

[igorvishnevskiy]

I also tried to tackle this problem from the different side. Sort of the way it is done for .h5 models generated by Keras. With Keras, the conversion to SavedModel format is straight forward, but not the same for .ckpt models. So I converted model into SavedModel format this way too, I got .pb file and variables are present too. My gorilla way of making it. I didn't know the names of input and output nodes, therefore just iterated and added all, which contained output data in them. Issue I face with this way of converting is described at the bottom on this reply msg.

    def tf_save_model_raw_all_inputs_outputs_as_tensors(self, session):
        ignore_list = []
        run = True
        while run:
            try:
                tf.compat.v1.reset_default_graph()
                with tf.Session(graph=tf.Graph()) as sess:
                    # Restore the graph
                    saver = tf.train.import_meta_graph(MODEL_PATH + '.meta')
                    saver.restore(sess, MODEL_PATH)

                    output_node_names = [n.name for n in tf.get_default_graph().as_graph_def().node
                                         if str(n.name) not in ignore_list]

                    tensor_inputs = {node.name: tf.saved_model.utils.build_tensor_info(sess.graph.get_tensor_by_name(node.name + ":0"))
                                     for node in tf.get_default_graph().as_graph_def().node if node.name in output_node_names}

                    tensor_outputs = {node: tf.saved_model.utils.build_tensor_info(sess.graph.get_tensor_by_name(node + ":0"))
                                     for node in output_node_names}

                    export_path = os.path.join(tf.compat.as_bytes(self.output_path),
                                               tf.compat.as_bytes(MODEL_VERSION))
                    builder = tf.compat.v1.saved_model.builder.SavedModelBuilder(export_path)

                    prediction_signature = (
                        tf.saved_model.signature_def_utils.build_signature_def(
                            inputs=tensor_inputs,
                            outputs=tensor_outputs,
                            method_name=tf.saved_model.PREDICT_METHOD_NAME
                        )
                    )

                    builder.add_meta_graph_and_variables(sess,
                                                         [tf.saved_model.tag_constants.SERVING],
                                                         signature_def_map={self.PREDICT_KEY: prediction_signature})
                    builder.save()
                    break
            except Exception as e:
                node_to_ignore = str(e).split("The operation, '")[1].replace("', exists but only has 0 outputs.\"", "")
                ignore_list.append(node_to_ignore)
                continue

Tensorflow Model Server loads model fine. But the problem I am facing here is when I send input to tensorflow model serving as the gRPC request: grpc_request.inputs["images"].CopyFrom(tf.compat.v1.make_tensor_proto(images, dtype=None, shape=images.shape)), I get the following error:

ERROR:root:Exception: <_Rendezvous of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "input size does not match signature: 1!=0 len({images}) != len({}). Sent extra: {images}. Missing but required: {}."
        debug_error_string = "{"created":"@1605898068.137292989","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1095,"grpc_message":"input size does not match signature: 1!=0 len({images}) != len({}). Sent extra: {images}. Missing but required: {}.","grpc_status":3}"
>

[weinman]

@igorvishnevskiy thanks for your comments and all your hard work so far! I'm definitely eager to follow the thread your progress, and contribute wherever I might be able to help.

Full disclosure: I've never tried to use the SavedModel format or TensorFlow serving.

Your error in item 4 suggests that for some reason or another the data coming into the model through this mechanism was dense, rather than sparse, as expected; and your subsequent change in item 5 seems reasonable to me.

Is there a stack trace or some clue you might give us as to the source or location of the error you've reported? I can't quite see the connection to what's come before.

[igorvishnevskiy]

@weinman Hello Jerod. Apology for the delay. I was simply busy working on the complex project as work. However I finally got back to it and made conversion to SavedModel happen and also added a module for the GRPC inference to TF Serving as well. Works great. I would love to contribute my code to your platform. I tried many and yours produces great accuracy and trains fast too. I did it for a single input inference. We could see how we could also handle bulk inference in the future too for better performance, for Tensorflow Model Serving in particular. If you don't mind I will create a PR. Thank you.

[igorvishnevskiy]

All that I was trying to do above was all wrong by the way. Estimator() class already has a method: export_saved_model(). However it requires the "serving_input_receiver_fn" to be passed to it and that is what I figure out how to do right. With correct Serving Input Receiver function, conversion works nicely and I get good responses from the inference against the model that is hosted by the Tensorflow Model Serving.

[weinman]

@igorvishnevskiy I'm thrilled you got it figured out. This seems like a nice enhancement to the tool, I'd be glad to see a PR so it can be shared with others!

(I have a backlog in PRs but will hope to get to it once things slow down here as well.)

Please make sure you're OK with having your contributions appear under the GPL3 license associated with the project.

Thanks for the update!

[igorvishnevskiy]

@weinman I just created a pull request. Happy to contribute. Your platform is very nice. Thank you. https://github.com/weinman/cnn_lstm_ctc_ocr/pull/70