Support for ExampleListWithContext requests formatted as JSON in Predict REST API

azagniotov commented 4 years ago

Please go to Stack Overflow for help and support:

https://stackoverflow.com/questions/tagged/tensorflow-serving

If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
The form below must be filled out.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

Feature Request

System information

tensorflow-serving-api==2.0.0
tensorflow==2.0.0
tensorflow-ranking==0.2.0

Disclaimer

I am not sure if the following is a bug, therefore I am listing this under a feature request

Describe the problem the feature is intended to solve

I am trying to POST an ELWC (ExampleListWithContext) to the predict REST API.

Based on what I see at https://www.tensorflow.org/tfx/serving/api_rest , the request body format for the classify and regress APIs supports the ELWC structure formatted as JSON. Yet, the predict API does not support the same JSON format for ELWC.

In order for me to make a successful request with the ELWC to predict API, I need to first serialize & base64 encode the ELWC proto and then make a request to predict API using a cURL request that looks as follows:

import base64
from tensorflow_serving.apis import input_pb2
from google.protobuf import text_format

EXAMPLE_LIST_WITH_CONTEXT_PROTO = text_format.Parse(
      """
     examples {
          ....
     }
    context {
         .....
    }
    """, input_pb2.ExampleListWithContext())

serialized_elwc_proto = EXAMPLE_LIST_WITH_CONTEXT_PROTO.SerializeToString()
serialized_elwc_proto_bytes = base64.b64encode(serialized_elwc_proto)
serialized_elwc_proto_base64 = serialized_elwc_proto_bytes.decode('utf-8')

the cURL (the serialized_elwc_proto_base64 holds the CqABCp0BCiQK...):

curl -H "Content-Type: application/json" -X POST \
http://192.168.99.100:8501/v1/models/my_model/versions/1589680164:predict \
-d '{"instances": [{"b64": "CqABCp0BCiQK.... TRUNCATED"}]}'

We can be a little more descriptive by specifying the signature_name and the receiver_name (whatever was defined when creating serving_input_receiver_fn):

curl -H "Content-Type: application/json" -X POST \
http://192.168.99.100:8501/v1/models/my_model/versions/1589680164:predict \
-d '{"signature_name": "predict", "instances": [{"input_ranking_data": {"b64": "CqABCp0BCiQK.... TRUNCATED"}}]}'

For context, full discussion: https://github.com/tensorflow/ranking/issues/189

Question

Why does predict REST API does not support the same request body format for ELWC, just like classify & regress do? Is this by design or an overlook (i.e.: bug). If this is by design, what is the rationale behind this decision?

Describe the solution

Ability to make request to predict REST API with a request body as a JSON object formatted as follows:

{
  // Optional: serving signature to use.
  // If unspecifed default serving signature is used.
  "signature_name": <string>,

  // Optional: Common context shared by all examples.
  // Features that appear here MUST NOT appear in examples (below).
  "context": {
    "<feature_name3>": <value>|<list>
    "<feature_name4>": <value>|<list>
  },

  // List of Example objects
  "examples": [
    {
      // Example 1
      "<feature_name1>": <value>|<list>,
      "<feature_name2>": <value>|<list>,
      ...
    },
    {
      // Example 2
      "<feature_name1>": <value>|<list>,
      "<feature_name2>": <value>|<list>,
      ...
    }
    ...
  ]
}

Additional context

https://github.com/tensorflow/ranking/issues/189

netfs commented 4 years ago

Predict API uses TensorProto as its input/output type. Protocol buffers (like ELWC you are referring to here) are passed in serialized form as DT_STRING typed tensors.

Further, the REST Predict API mimics the gRPC API. This API takes PredictRequest protocol buffer as its input, defined here: https://github.com/tensorflow/serving/blob/3a0bdbe5c216e4eeab8110cad8d959d21a2e376a/tensorflow_serving/apis/predict.proto#L12-L21

As you see, this request takes a TensorProto as its input.

So Predict API does not/cannot treat ExampleListWithContext protobuf differently like Classify/Regress methods do, and you need to serialize and pass such protos as DT_STRING tensors. Predict API is designed to be agnostic to specific protocol buffers (by design), to allow the freedom to pass any arbitrary protobufs.

I am going to close this feature request because it is at odds with the Predict API design.

azagniotov commented 4 years ago

Hi @netfs ,

Thank you for your input and confirming that Predict API is designed to be agnostic to specific protocol by design.

tensorflow / serving