Serving predictions from TF Serving via Docker

[aburkard]

I'm having trouble figuring out how to get predictions from a ranking model running on the TensorFlow Serving Docker container.

I was able to successfully export by model to a SavedModel as described in https://github.com/tensorflow/ranking/issues/53 and other issues.

I'm also able to use the saved_model_cli to view the signature and run inference:

$ saved_model_cli show --dir . --tag_set serve --signature_def predict
The given SavedModel SignatureDef contains the following input(s):
  inputs['examples'] tensor_info:
      dtype: DT_STRING
      shape: (-1)
      name: input_example_tensor:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['output'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, -1)
      name: groupwise_dnn_v2/accumulate_scores/div_no_nan:0
Method name is: tensorflow/serving/predict

$ saved_model_cli run  --dir .  --tag_set serve --signature_def predict --input_examples 'examples=[{"column_1":[436772, 1234, 3942], "column_2":[1681, 1681, 1681]}]'

Result for output key output:
[[0.1218676 0.8255983 0.9231393]]

But I can't for the life of me figure out what the payload should be to my docker container.

$ curl -d '{"inputs": { "inputs": [] } }' -X POST http://localhost:8501/v1/models/my_model:predict
{
    "outputs": []
}

$ curl -d '{"inputs": { "inputs": [{"column_1":[436772, 87602, 223161], "column_2":[1681, 1681, 1681]}] } }' -X POST http://localhost:8501/v1/models/my_model:predict
{ "error": "JSON Value: {\n    \"column_1\": [\n        436772,\n        87602,\n        223161\n    ],\n    \"column_2\": [\n        1681,\n        1681,\n        1681\n    ]\n} not formatted correctly for base64 data" }

I've tried many variations of nesting the keys inputs, instances, and examples, as well as base64 encoding the json input. Has anyone been able to successfully do this?

[aburkard]

I've solved this by using the regress endpoint instead:

curl -d '{"signature_name": "regression", "examples": [{"column_1": [436772], "column_2": [1681]}, {"column_1": 1100, "column_2": 2000}]}' \
    -X POST http://localhost:8501/v1/models/my_model:regress

On a side note, I also noticed the existence of tfr.data.build_ranking_serving_input_receiver_fn, which seems to be the correct way to do things as discussed in other issues. However, that seems to make things even more challenging as I'd have to serialize everything the protobufs first.

[xuanhuiwang]

Thanks for your questions and figure out a solution on your end. tfr.data.build_ranking_serving_input_receiver_fn is the listwise serving and it accepts ExampleListWithContext, but not tf.Example. I think your curl command line above send a tf.Example, right?