When using ELWC - OP_REQUIRES failed at example_parsing_ops.cc:91 : Invalid argument: Could not parse example input

[azagniotov]

Hello Team,

I trained a TF ranking model (basing my training on the following example: https://github.com/tensorflow/ranking/blob/master/tensorflow_ranking/examples/tf_ranking_tfrecord.py) and saved it using estimator.export_saved_model('my_model', serving_input_receiver_fn), the model was trained successfully & saved without any warnings/errors.

I deployed the model to a local TensorFlow ModelServer and made an call to it over HTTP using cURL as described on https://www.tensorflow.org/tfx/serving/api_rest#request_format. Unfortunately I see the following error after making the request:

W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at example_parsing_ops.cc:91 : Invalid argument: Could not parse example input, value: '

ctx_f0

{ "error": "Could not parse example input, value: \'\n\035\n\021ctx_f0\022\010\022\006\n\004\000\000\340@\'\n\t [[{{node ParseExample/ParseExample}}]]" }

I understand that this is a problem that may be related to serialization where my input was not properly serialized, but saving the model by generating serving_input_receiver_fn & using it produced no errors/warnings, so I am not sure where to start looking to resolve this.

I am providing some details below, please let me know if you need more information.

Details

TF framework module versions

• tensorflow-serving-api==2.0.0
• tensorflow==2.0.0
• tensorflow-ranking==0.2.0

Some training parameters and functions

• _CONTEXT_FEATURES = {'ctx_f0'}
• _DOCUMENT_FEATURES = {'f0', 'f1', 'f2'}
• _DATA_FORMAT = tfr.data.ELWC
• _PADDING_LABEL = -1

def example_feature_columns():
    spec = {}
    for f in _DOCUMENT_FEATURES:
        spec[f] = tf.feature_column.numeric_column(f, shape=(1,), default_value=_PADDING_LABEL, dtype=tf.float32)
    return spec

def context_feature_columns():
    spec = {}
    for f in _CONTEXT_FEATURES:
        spec[f] = tf.feature_column.numeric_column(f, shape=(1,), default_value=_PADDING_LABEL, dtype=tf.float32)
    return spec

Creating the serving_input_receiver_fn

context_feature_spec = tf.feature_column.make_parse_example_spec(context_feature_columns().values())
example_feature_spec = tf.feature_column.make_parse_example_spec(example_feature_columns().values())

serving_input_receiver_fn = tfr.data.build_ranking_serving_input_receiver_fn(
        data_format=_DATA_FORMAT,
        list_size=20,
        default_batch_size=None,
        receiver_name="input_ranking_data",
        context_feature_spec=context_feature_spec,
        example_feature_spec=example_feature_spec)

When making a REST API to a local TensorFlow ModelServer using the following cURL request

curl -H "Content-Type: application/json" \
-X POST http://192.168.99.100:8501/v1/models/my_model/versions/1587842143:regress \
-d '{"context": {"ctx_f0": 7.2}, "examples":[{"f0":[35.92],"f1":[5.258],"f2":[5.261]},{"f0":[82.337],"f1":[2.06],"f2":[2.068]}]}'

The error is as follows:

W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at example_parsing_ops.cc:91 : Invalid argument: Could not parse example input, value: '

ctx_f0

{ "error": "Could not parse example input, value: \'\n\035\n\021ctx_f0\022\010\022\006\n\004\000\000\340@\'\n\t [[{{node ParseExample/ParseExample}}]]" }

[ramakumar1729]

@azagniotov The serving_input_fn expects a serialized ELWC proto, not the raw Tensor values. Can you try serializing the proto?

[azagniotov]

Hi @ramakumar1729 , thank you.

Just to make sure I got this right: you are saying that when I make the curl request, the request payload should be a serialized proto instead of the raw {"ctx_f0": 7.2} ... ?

[azagniotov]

@ramakumar1729 on a related topic:

I looked into a different way of generating the serving_input_receiver_fn, which allowed me to POST the aforementioned JSON payload to the server and obtain model results without any errors. The following is the code that I used to build the serving_input_receiver_fn:

def all_feature_spec():
    feature_names = ['f0', 'f1', 'f2'] 
    features = {
        name: tf.compat.v1.FixedLenFeature([1], tf.float32) for name in feature_names
    }
    features['ctx_f0'] = tf.compat.v1.FixedLenFeature([1], tf.int64)
    return features

serving_input_receiver_fn = tf.estimator.export.build_parsing_serving_input_receiver_fn(all_feature_spec())

Could you comment (pros/cons/edge cases) on using the API tf.estimator.export.build_parsing_serving_input_receiver_fn(..) VS using the tfr.data.build_ranking_serving_input_receiver_fn(...) to build the serving_input_receiver_fn? It appears that the former does enable me to POST the following JSON with raw numeric values to the REST API without any errors:

{"context": {"ctx_f0": 7.2}, "examples":[{"f0":[35.92],"f1":[5.258],"f2":[5.261]},{"f0":[82.337],"f1":[2.06],"f2":[2.068]}]}

[ramakumar1729]

@azagniotov : TF Serving supports few different kinds of SignatureDefs, for classify, regress and the generic predict apis.

In your cURL request, you are using a regress API call, which only supports tf.Examples as inputs (you are passing features for each example separately). To generate scores for all examples in a list using ELWC format, you need to use the predict api. I guess regress API allows for raw numeric values but I'm not sure predict API does.

Let me know if you have any further questions, happy to help.

[azagniotov]

@ramakumar1729 thank you for your patience.

I have created a serialized proto:

import random
from random import randint

def _float_feature(value):
    return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))

def create_example():
    feature_names = ['f0', 'f1', 'f2']
    features = {name: _float_feature(random.uniform(300, 10)) for name in feature_names}

    return tf.train.Example(features=tf.train.Features(feature=features))

def create_serialized_example_list(num_of_examples):
    example_protos = []
    for x in range(num_of_examples):
        example_proto = create_example()
        print(example_proto)

        example_proto_serialized = example_proto.SerializeToString()
        print(example_proto_serialized)

        example_protos.append(example_proto_serialized)

    return tf.make_tensor_proto(example_protos, dtype=tf.string)

def create_serialized_context():
    feature_names = ['ctx_f0']
    features = {name: _float_feature(random.uniform(300, 10)) for name in feature_names}
    context_example = tf.train.Example(features=tf.train.Features(feature=features))
    context_example_proto = context_example.SerializeToString()

    return tf.make_tensor_proto(context_example_proto, dtype=tf.string)

examples_list = create_serialized_example_list(2)
print(examples_list)

context = create_serialized_context()
print(context)

For visibility & debugging purposes, the print(examples_list) converts the following example proto:

features {
  feature {
    key: "f0"
    value {
      float_list {
        value: 238.68080139160156
      }
    }
  }
  feature {
    key: "f1"
    value {
      float_list {
        value: 297.0102233886719
      }
    }
  }
  feature {
    key: "f2"
    value {
      float_list {
        value: 22.836402893066406
      }
    }
  }
}
...
...
features { ... }

to the serialized proto blob:

dtype: DT_STRING
tensor_shape {
  dim {
    size: 2
  }
}
string_val: "\n0\n\016\n\002f0\022\010\022\006\n\004S\003\201C\n\016\n\002f1\022\010\022\006\n\004z\300CC\n\016\n\002f2\022\010\022\006\n\004\243\307\tC"
string_val: "\n0\n\016\n\002f2\022\010\022\006\n\004\266\271oC\n\016\n\002f0\022\010\022\006\n\004\350E\204B\n\016\n\002f1\022\010\022\006\n\004\214\352\203C"

while the print(context) produces the:

dtype: DT_STRING
tensor_shape {
}
string_val: "\n\024\n\022\n\006ctx_f0\022\010\022\006\n\004\307S\234B"

Now that I have the examples in the list & the context feature protos serialized, can you advise what should the cURL request structure look like, what should be the -d param payload in:

curl -H "Content-Type: application/json" \
-X POST http://192.168.99.100:8501/v1/models/my_model/versions/1587842143:<METHOD> \
-d '?????'

[ramakumar1729]

You can take a look at the resnet client example to create PredictRequests. Based on your receiver_name, the -d param payload should be something like "{'input_ranking_data': <serialized_elwc>}". We usually do this programmatically, similar to the resnet client.

[azagniotov]

@ramakumar1729, thank you.

I am still not fully clear about the structure of serialized_elwc. Based on the debug output generated by my functions above, I have two examples and one context feature as serialized protos. If I read this correctly and by looking at ResNet client example, the payload should be:

{"input_ranking_data": [serialized_example, serialized_example, serialized_context]

i.e.:

{"input_ranking_data": [
"\n0\n\016\n\002f0\022\010\022\006\n\004S\003\201C\n\016\n\002f1\022\010\022\006\n\004z\300CC\n\016\n\002f2\022\010\022\006\n\004\243\307\tC",
"\n0\n\016\n\002f2\022\010\022\006\n\004\266\271oC\n\016\n\002f0\022\010\022\006\n\004\350E\204B\n\016\n\002f1\022\010\022\006\n\004\214\352\203C",
"\n\024\n\022\n\006ctx_f0\022\010\022\006\n\004\307S\234B"]}

Please let me know if I blatantly misunderstood you

EDIT: That would not work as JSON parse won't parse that ^. I will try encoding it as base64 as per ResNet

[azagniotov]

Also @ramakumar1729, another interesting fact: despite what the receiver_name may be set to, regress & classify methods require examples word to be passed as a receiver_name, while predict method name requires instances or inputs:

curl -H "Content-Type: application/json" \
> -X POST http://192.168.99.100:8501/v1/models/tf_ranking_v8_local/versions/1588000040:predict \
> -d '{"input_ranking_data":[]}'
{ "error": "Missing \'inputs\' or \'instances\' key" }

curl -H "Content-Type: application/json" \
> -X POST http://192.168.99.100:8501/v1/models/tf_ranking_v8_local/versions/1588000040:classify \
> -d '{"input_ranking_data":[]}'
{ "error": "JSON Value: {\n    \"input_ranking_data\": []\n} When method is classify, key \'examples\' is expected and was not found" }

curl -H "Content-Type: application/json" \
> -X POST http://192.168.99.100:8501/v1/models/tf_ranking_v8_local/versions/1588000040:regress \
> -d '{"input_ranking_data":[]}'
{ "error": "JSON Value: {\n    \"input_ranking_data\": []\n} When method is classify, key \'examples\' is expected and was not found" }

[azagniotov]

@ramakumar1729, I got the making request to the REST API to work when hitting predict API:

import random
import base64
from random import randint

def _float_feature(value):
    return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))

def _int_feature(value):
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))

def create_example():
    feature_names = ['f0', 'f1', 'f2']
    features = {name: _float_feature(random.uniform(300, 10)) for name in feature_names}
    features['ctx_f0'] = _int_feature(123)
    return tf.train.Example(features=tf.train.Features(feature=features))

def create_serialized_example_list(num_of_examples):
    example_protos = []
    for x in range(num_of_examples):
        example_proto = create_example()
        example_proto_serialized = example_proto.SerializeToString()

        example_protos.append(example_proto_serialized)

    return tf.make_tensor_proto(example_protos, dtype=tf.string)

tensor_proto = create_serialized_example_list(20)
tensor_proto_serialized = tensor_proto.SerializeToString()
tensor_proto_base64_bytes = base64.b64encode(tensor_proto_serialized)
tensor_proto_base64 = tensor_proto_base64_bytes.decode('utf-8')

print(tensor_proto_base64)

and the cURL call is:

the tensor_proto_base64 holds the CAcSBBICCBRCQwpBCg4K..... string

curl -H "Content-Type: application/json" \
-X POST http://192.168.99.100:8501/v1/models/tf_ranking_v9/versions/1588000040:predict \
-d '{"instances": [{"b64": "CAcSBBICCBRCQwpBCg4K..... TRUNCATED"}]}'

The results are as follows:

{
    "predictions": [[0.0222161338, 0.0222163089, 0.02221632, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162679, 0.0222161338]
    ]
}

What's interesting though is that I always get this ^ same result set, no matter what the input is (i.e.: what the feature numeric values are). I can make 3 different cURL requests with a different b64 string as a payload, but the predictions will always be as the aforementioned ^

In fact, I can use completely bogus example feature names (i.e.: apples, oranges & lemons) when constructing the tensor_proto, the API still is able to return the same above predictions. Should not the model complain that the expected features f0, f1 and f2 are not there?

Any thoughts?

[ramakumar1729]

The predict serving API does not work well with cURL request afaik. If you want to make a cURL request, please use the regress serving api:

curl -d '{"signature_name": "regression", "examples": [{"f_0": [val], "f_1": [1681]}, {"f_0": 1100, "f_1": 2000}]}' \
    -X POST http://192.168.99.100:8501/v1/models/tf_ranking_v9/versions/1588000040

[azagniotov]

Hi @ramakumar1729 , Yes, using regress REST API has been working for me already, e.g.:

curl -H "Content-Type: application/json" \
-X POST http://192.168.99.100:8501/v1/models/tf_ranking_v8_local/versions/1587797526:regress \
-d '{"context": {"ctx_f0": 7}, "examples":[{"f1":[335.92166],"f2":[5.2585354E-9],"f3":[5.26169E-4]},{"f1":[82.33758],"f2":[2.068338E-7],"f3":[2.0684237E-7]}]}'

The original reason for the current issue was to understand the gotchas around making inference requests after generating serving_input_receiver_fn using tfr.data.build_ranking_serving_input_receiver_fn, which I clarified with your pointers, thank you.

I do have a few more questions:

1. From what I see it is possible to use predict API when making a request with raw numeric values JSON to the saved_model_cli (https://github.com/tensorflow/ranking/issues/46#issuecomment-523419410). I am curious why the behavior when using saved_model_cli and tensorflow server REST API is not consistent? i.e.: I can submit raw values to the CLI, while I need to serialize the proto and base64 encode it when POSTing to the REST API.

2. Why the regress REST API allows for raw numeric values, while predict does not? Is this by design or an overlook (i.e.: bug). If this is by design, what is the rationale behind this decision? It would be really great is the REST API contract across all REST APIs would be consistent, i.e.: would be possible to POST JSONs with raw numeric values.

3. If I have a listwise trained model, can I use the regress API for inference requests? If I can use regress API for listwise models, what is the input format to issue to the regress sig that is compatible with a listwise trained model?

Thank you

[ramakumar1729]

@azagniotov : TF Serving would have more details on this. For the listwise trained models provided by TF-Ranking, we support regress and predict API.

For regress API, we expect tf.Examples as input. Maybe regress API allows for interchangeably using tf.Examples and raw Tensors, it depends on how the POST request is processed. Note that this would only work for group_size=1 and assuming you do not have a complicated transform_fn that works on multiple examples in an ELWC.

For predict API, we expect ELWCs as input. This is something we have greater control over, and we create a placeholder for the ELWC. Hope this helps clarify things.

[azagniotov]

@ramakumar1729 yes, thank you!

I will close the current issue.

[azagniotov]

Hi @ramakumar1729 ,

Sorry to comment on this closed issue, but I thought I would share my latest (positive & successful) findings regarding making requests to predict serving REST API. Perhaps others may find them useful:

TL;DR

I was able to successfully POST serialized & base64 encoded ELWC proto to the predict REST API and get the expected predictions. These predictions match exactly the predictions that I get if I make a gRPC request using the same ELWC proto to the TensorFlow model server over gRPC.

This gave me the confidence in behavior parity that making inference requests over HTTP vs gRPC produces consistent results for the same ELWC.

Details

Previously in https://github.com/tensorflow/ranking/issues/189#issuecomment-620256362, I was creating a tensor_proto out of the serialized ELWC proto, then serializing the tensor_proto & base64 that, which then I was POSTing to the API.

I found out that I should not create tensor_proto out of serialized ELWC proto string. What needs to be serialized & base64 encoded is the ELWC proto itself. As a result, my cURL looks like as before before, the difference is that the b64 string holds the ELWC proto:

curl -H "Content-Type: application/json" -X POST \
http://192.168.99.100:8501/v1/models/tf_ranking_v10/versions/1589680164:predict \
-d '{"instances": [{"b64": "CqABCp0BCiQK.... TRUNCATED"}]}'

We can be a little more descriptive by specifying the signature_name and the receiver_name (whatever was defined when creating serving_input_receiver_fn):

curl -H "Content-Type: application/json" -X POST \
http://192.168.99.100:8501/v1/models/tf_ranking_v10/versions/1589680164:predict \
-d '{"signature_name": "predict", "instances": [{"input_ranking_data": {"b64": "CqABCp0BCiQK.... TRUNCATED"}}]}'

The predictions from the above cURL requests match the gRPC response prediction after making the gRPC request as follows:

import grpc
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2_grpc
from tensorflow_serving.apis import input_pb2
from google.protobuf import text_format
from google.protobuf.json_format import MessageToDict

EXAMPLE_LIST_WITH_CONTEXT_PROTO = text_format.Parse(
      """
     examples {
          ....
     }
    context {
         .....
    }
    """, input_pb2.ExampleListWithContext())

example_list_with_context_proto = EXAMPLE_LIST_WITH_CONTEXT_PROTO.SerializeToString()
tensor_proto = tf.make_tensor_proto(example_list_with_context_proto, dtype=tf.string, shape=[1])

timeout_in_secs = 3
request = predict_pb2.PredictRequest()
request.inputs['input_ranking_data'].CopyFrom(tensor_proto)
request.model_spec.signature_name = 'predict' 
request.model_spec.name = 'tf_ranking_v10'

channel = grpc.insecure_channel("0.0.0.0:8500")
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)                            

grpc_response = stub.Predict(request, timeout_in_secs)
unpacked_grpc_response = MessageToDict(grpc_response, preserving_proto_field_name = True)

print(unpacked_grpc_response['outputs']['output']['float_val'])

[ramakumar1729]

@azagniotov Thanks for sharing your experience in successfully serving the ranking model with serialized ELWC inputs~ This will be a useful reference for others trying to do the same.

[davidmosca]

Hi @azagniotov , sorry to re-open this closed issue. I have tried to implement the gRPC response prediction from the gRPC request you detailed above, but I get an error message:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.INVALID_ARGUMENT details = "Could not parse example input, value: '

Could you please share the exact content of:

""" examples { .... } context { ..... } """

so that I can attempt to reproduce your exact code? Thanks.

[azagniotov]

@davidmosca sure. give me a day, pls

[azagniotov]

Nah, did it sooner, @davidmosca :

examples {
  features {
    feature {
      key: "feature_name_a"
      value {
        float_list {
          value: 0.46671828627586365
        }
      }
    }
    feature {
      key: "feature_name_b"
      value {
        float_list {
          value: 11.34334447979927063
        }
      }
    }
  }
}
examples {
  features {
    feature {
      key: "feature_name_a"
      value {
        float_list {
          value: 0.34334447979927063
        }
      }
    }
    feature {
      key: "feature_name_b"
      value {
        float_list {
          value: 12.032746315002441
        }
      }
    }
  }
}
context {
  features {
    feature {
      key: "your_context_feature_name"
      value {
        float_list {
          value: 123.456
        }
      }
    }
  }
}

I hope this helps

[davidmosca]

Thanks for that. Unfortunately I still see the same error, which is strange because I just copied/pasted your code and replaced the variables' values where appropriate (my model too is using ELWC data). Also I know the serving API is up and running because I can make regress cURL requests to it. Do you have any idea in your experience of what else could cause this error?

[azagniotov]

@davidmosca I am not sure why this is happening, TBH. But, if you could post here your complete code (parsing the ELWC, making the gRPC request, etc), I think it could help.

[davidmosca]

Here it is:

Details

TF framework module versions

tensorflow-serving-api==2.3.0
tensorflow==2.3.0
tensorflow-ranking==0.3.0
protobuf==3.13.0
grpcio==1.31.0

Some training parameters

_CONTEXT_FEATURES = {'what_tokens'}
_DOCUMENT_FEATURES = {'title_tokens', 'description_tokens', 'relevance'}
_DATA_FORMAT = tfr.data.ELWC
_PADDING_LABEL = -1

gRPC request

import grpc
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2_grpc
from tensorflow_serving.apis import input_pb2
from google.protobuf import text_format
from google.protobuf.json_format import MessageToDict
import tensorflow as tf

EXAMPLE_LIST_WITH_CONTEXT_PROTO = text_format.Parse(
"""
context {
  features {
    feature {
      key: "what_tokens"
      value {
        bytes_list {
          value: ["this", "is", "a", "relevant", "question"]
        }
      }
    }
  }
}
examples {
  features {
    feature {
      key: "title_tokens"
      value {
        bytes_list {
          value: ["this", "is", "a", "relevant", "answer"]
        }
      }
    }
    feature {
      key: "description_tokens"
      value {
        bytes_list {
          value: ["with", "a", "relevant", "description"]
        }
      }
    }
    feature {
        key: "relevance"
        value { float_list { value: 5.0 } }
    }
  }
}
""", input_pb2.ExampleListWithContext())

example_list_with_context_proto = EXAMPLE_LIST_WITH_CONTEXT_PROTO.SerializeToString()
tensor_proto = tf.make_tensor_proto(example_list_with_context_proto, dtype=tf.string, shape=[1])

timeout_in_secs = 5
request = predict_pb2.PredictRequest()
request.inputs['examples'].CopyFrom(tensor_proto)
request.model_spec.signature_name = 'predict'
request.model_spec.name = 'tfr'

channel = grpc.insecure_channel("localhost:8500")
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

grpc_response = stub.Predict(request, timeout_in_secs)
unpacked_grpc_response = MessageToDict(grpc_response, preserving_proto_field_name = True)
print(unpacked_grpc_response['outputs']['output']['float_val'])

The request generates this error:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.INVALID_ARGUMENT details = "Could not parse example input, value: ' ‡
„
8 description_tokens"

with
a relevant description  relevance   @ 1 title_tokens!  this is
a relevant answer6 4 2 what_tokens# ! this is
a relevant question' [[{{node ParseExample/ParseExampleV2}}]]" debug_error_string = "{"created":"@1599211015.577000000","description":"Error received from peer ipv6:[::1]:8500","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"Could not parse example input, value: '\n\u0087\u0001\n\u0084\u0001\n8\n\u0012description_tokens\u0012"\n \n\u0004with\n\u0001a\n\brelevant\n\u000bdescription\n\u0015\n\trelevance\u0012\b\u0012\u0006\n\u0004\u0000\u0000\u00a0@\n1\n\ftitle_tokens\u0012!\n\u001f\n\u0004this\n\u0002is\n\u0001a\n\brelevant\n\u0006answer\u00126\n4\n2\n\u000bwhat_tokens\u0012#\n!\n\u0004this\n\u0002is\n\u0001a\n\brelevant\n\bquestion'\n\t [[{{node ParseExample/ParseExampleV2}}]]","grpc_status":3}"

The underlying TF-Ranking model's structure matches that of the EXAMPLE_LIST_WITH_CONTEXT_PROTO above (I have tried with and without relevance).

TF-Ranking model's predict signature

signature_def['predict']: The given SavedModel SignatureDef contains the following input(s): inputs['examples'] tensor_info: dtype: DT_STRING shape: (-1) name: input_example_tensor:0 The given SavedModel SignatureDef contains the following output(s): outputs['output'] tensor_info: dtype: DT_FLOAT shape: (-1, -1) name: groupwise_dnn_v2/accumulate_scores/div_no_nan:0 Method name is: tensorflow/serving/predict

The Docker dashboard shows that the serving API itself is up and running. Also, a cURL (regress) request to a twin TF Serving API on 8501 (the escaped double quotes are required on Windows):

curl -d "{\"context\": {\"what_tokens\": [\"a\", \"question\"]}, \"examples\": [{\"title_tokens\": \"answer\", \"description_tokens\": \"description\"}]}" -X POST http://localhost:8501/v1/models/tfr:regress

produces this response:

{
    "results": [-10.4137278]
}

[azagniotov]

@davidmosca

TL;DR

I took your code as-is and ran it. I was able successfully (i.e.: I did not observe your aforementioned error and got back a list of predictions) to make a request to the Docker container over gRPC using TF/TFR v2.1.0 and also using v2.3.0 as another attempt, which is the versions you are on.

Question

Are you pinning down the docker container version when pulling it down or are you getting the latest? i.e.: docker pull tensorflow/serving:2.3.0 or docker pull tensorflow/serving ? The latter will pull down the latest tag, which may or may not be what you want. I remember some time ago when I was not pinning down the container version and one day things stopped working for me because I probably pulled down new functionality or changes from the upstream. So, I had to fix the Docker version.

In my Docker start command, I also explicitly start the version that I am on, e.g.:

docker run -t --rm -p 8500:8500 -p 8501:8501 \
-v "/home/azagniotov/tensorflow_ranking_model:/models/tensorflow_ranking_model" \
-e MODEL_NAME="tensorflow_ranking_model" tensorflow/serving:2.1.0 &

MORE DETAILS

tensorflow-serving-api==2.1.0 (or 2.3.0)
tensorflow==2.1.0 (or 2.3.0)
tensorflow-ranking==0.3.0
docker pull tensorflow/serving:2.1.0 (or 2.3.0)

In the following code I am using to generate ranking_serving_input_receiver_fn when saving the model. Not sure if this helps, but I thought to show how I do it:

# Dicts, e.g.: 
# spec['feature_name_a'] = tf.feature_column.numeric_column('feature_name_a', shape=[1], default_value=-1, dtype=tf.float32)
context_feature_spec = tf.feature_column.make_parse_example_spec(context_feature_columns().values())
example_feature_spec = tf.feature_column.make_parse_example_spec(example_feature_columns().values())

# I am using `input_ranking_data` name as the `receiver_name`
serving_input_receiver_fn = tfr.data.build_ranking_serving_input_receiver_fn(
        data_format=tfr.data.ELWC,
        list_size=_LIST_SIZE,
        default_batch_size=None,
        size_feature_name='example_list_size',
        receiver_name='input_ranking_data',
        context_feature_spec=context_feature_spec,
        example_feature_spec=example_feature_spec)

# ranker == tf.estimator.Estimator
ranker.export_saved_model(export_dir_base='tensorflow_ranking_model',
                          serving_input_receiver_fn=serving_input_receiver_fn,
                          assets_extra={'tf_serving_warmup_requests': 'tf_serving_warmup_requests'},
                          as_text=False,
                          checkpoint_path=None,
                          experimental_mode=tf.estimator.ModeKeys.PREDICT)

TF-Ranking model's predict signature

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['predict']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_ranking_data'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: input_ranking_tensor:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['output'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1)
        name: groupwise_dnn_v2/accumulate_scores/div_no_nan:0
  Method name is: tensorflow/serving/predict

[hanwire]

Nah, did it sooner, @davidmosca :

examples {
  features {
    feature {
      key: "feature_name_a"
      value {
        float_list {
          value: 0.46671828627586365
        }
      }
    }
    feature {
      key: "feature_name_b"
      value {
        float_list {
          value: 11.34334447979927063
        }
      }
    }
  }
}
examples {
  features {
    feature {
      key: "feature_name_a"
      value {
        float_list {
          value: 0.34334447979927063
        }
      }
    }
    feature {
      key: "feature_name_b"
      value {
        float_list {
          value: 12.032746315002441
        }
      }
    }
  }
}
context {
  features {
    feature {
      key: "your_context_feature_name"
      value {
        float_list {
          value: 123.456
        }
      }
    }
  }
}

I hope this helps

Hello, @azagniotov, could you please clarify, how to parse such ELWC with tfr.data.build_ranking_dataset, is it ok to use such function?

def create_feature_columns():

  sparse_column = tf.feature_column.categorical_column_with_hash_bucket(
      key="your_context_feature_name", dtype= tf.string, hash_bucket_size=1)
  query_embedding_column = tf.feature_column.embedding_column(
      sparse_column, dimension=1) 
  context_feature_columns = {"your_context_feature_name": query_embedding_column}
   example_feature_columns = {}    
   for fe in ['feature_name_a', 'feature_name_a']:
      example_feature_columns[str(fe)] = tf.feature_column.numeric_column( str(fe), dtype=tf.float32)
  return context_feature_columns, example_feature_columns

def make_dataset(file_pattern,
                 batch_size,
                 randomize_input=False,  
                 num_epochs=None):
  context_feature_columns, example_feature_columns = create_feature_columns()
  context_feature_spec = tf.feature_column.make_parse_example_spec(
      context_feature_columns.values())
  label_column = tf.feature_column.numeric_column(
      'target_name', dtype=tf.float32, default_value= -1)
  example_feature_spec = tf.feature_column.make_parse_example_spec(
      list(example_feature_columns.values()) + [label_column])
  dataset = tfr.data.build_ranking_dataset(
      file_pattern=file_pattern,
      data_format=tfr.data.ELWC,
      batch_size=batch_size,
      context_feature_spec=context_feature_spec,
      example_feature_spec=example_feature_spec,
      reader=tf.data.TFRecordDataset,
      shuffle=randomize_input,
      num_epochs=num_epochs,
      size_feature_name=None)

Thank you in advance, your comments really helpful.

[davidmosca]

OK I found what the problem was. When you define the example feature_columns in the TF-Ranking model, you always need to assign a dummy default value, for instance:

title_column = tf.feature_column.categorical_column_with_vocabulary_file(key="title_tokens", vocabulary_file=FLAGS.vocab_path, default_value=0)

This is because the lists of examples are of varying length, and not setting a default value creates an issue which is not identified as such in the error message. Neither the Colab example nor the TFRecord example contains such default value (or at least the requirement for such value) so that's not helpful. The lack of TF-Ranking documentation is not helpful either.

Interestingly, adding this default value also automatically changes the input tensor alias from examples to input_ranking_data.

@azagniotov, thanks for sharing the gRPC request, that was very helpful in setting up TensorFlow Serving for online inference.

[azagniotov]

@davidmosca No worries. I am glad I could help. I edited my reply in the last few minutes and added some information about pinning down the Docker container version, I am not sure if you saw it.

[davidmosca]

I saw it just after publishing mine :) Thanks again.

[azagniotov]

Hi @utopn , I do not think I understand your question:

Hello, @azagniotov, could you please clarify, how to parse such ELWC with tfr.data.build_ranking_dataset, is it ok to use such function?

In order to build a ranking dataset, you need to provide to the tfr.data.build_ranking_dataset a path (i.e.: file_pattern arg) in the local filesystem where your TF record(s) that contains your ELWCs are located. So, if you have not done so, first you need to generated TF record(s) with ELWCs

[azagniotov]

I am closing this issue, @utopn as your question is out of scope here. Feel free to raise a new issue to address your question specifically