Can't get custom output name from tfserving

[xyangk]

I want the outputs from tfserving is: {"outputs":{"scores":[0.936071157,0.0527787767]}} but I got : { "outputs": [ 0.936071038, 0.0527787916 ] }

Environment: python 3.6 tensorflow 2.3.0 tfserving 2.3.0

save model code:

class MyModel(tf.keras.Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.pmodel = model.model
        self.out = keras.layers.Lambda(lambda x:x[:,1])

    @tf.function(input_signature=[[tf.TensorSpec(shape=[None,1], dtype=tf.string, name='input_q'), tf.TensorSpec(shape=[None,1], dtype=tf.string, name='input_a')]])
    def call(self, inputs):
        output = self.pmodel(inputs)
        out = self.out(output)
        return {"scores": out} 

NM = MyModel()

# test
# print(NM([qi, ai]))
# {'scores': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([0.93607116, 0.05277878], dtype=float32)>}

# save model
tf.saved_model.save(NM, 'saved_model/202011032114')

By the way, model.model is tf.keras.Model :

SignatureDefs:

Start tfserving

What I got :(

But!, If model output two keys like {"scores": out, "give_up":out}, it works well.

class MyModel(tf.keras.Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.pmodel = model.model
        self.out = keras.layers.Lambda(lambda x:x[:,1])

    @tf.function(input_signature=[[tf.TensorSpec(shape=[None,1], dtype=tf.string, name='input_q'), tf.TensorSpec(shape=[None,1], dtype=tf.string, name='input_a')]])
    def call(self, inputs):
        output = self.pmodel(inputs)
        out = self.out(output)
        return {"scores": out, "give_up":out}

NM = MyModel()

# test
# print(NM([qi, ai]))
# {'scores': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([0.93607116, 0.05277878], dtype=float32)>, 'give_up': <tf.Tensor: shape=(2,), dtype=float32, numpy=array([0.93607116, 0.05277878], dtype=float32)>}

# save model
tf.saved_model.save(NM, 'saved_model/202011032126')

And the result:

I don't know why and didn't find any solution, Is there any kind person to help me ? Many thanks !

[guanxinq]

Sorry, I did not quite get your question. Could you clarify it?

[xyangk]

Sorry, I did not quite get your question. Could you clarify it?

I want my model output a uniform format like : {"outputs":{"scores":[0.936071157, 0.0527787767]}}, key scores is needed, I saved my model to output this format, see first SignatureDefs picture. I'm pretty sure that I saved it properly, I tested saved model use tf.saved_model.load, but when I using tfserving serving this model, I got {"outputs": [ 0.936071038, 0.0527787916 ] }, lost key scores. I don't know wether this problem came from tfserving or tensorflow, when I saved two keys scores and give_up, tfserving return {"outputs":{"scores":[0.936071157, 0.0527787767], "give_up":[0.936071157, 0.0527787767]}}, now I have key scores but with a redudant key give_up, this is my temporary solution. In my opinion, one key will lose, two keys will keep, did I describe the problem clearly?

[guanxinq]

Thanks for your clarification. It is intended. "If the output of the model contains only one named tensor, we omit the name and outputs key maps to a list of scalar or list values." More details: https://www.tensorflow.org/tfx/serving/api_rest#specifying_input_tensors_in_column_format.

[xyangk]

Thanks for your clarification. It is intended. "If the output of the model contains only one named tensor, we omit the name and outputs key maps to a list of scalar or list values." More details: https://www.tensorflow.org/tfx/serving/api_rest#specifying_input_tensors_in_column_format.

Thanks for your reply, It helps. Sorry I missed this detail before. So my temporary solution actually is final solution.:sweat_smile:

[guanxinq]

Feel free to create a PR or file a feature request. Close this bug now.