Predict from in-memory dataset

[davidmosca]

Hi, currently the predict_input_fn in the demo loads the file to be scored form disc (using the specified path); however, in a real life scenario (e.g. web search) this is not realistic - the prediction must be made from a dataset already loaded into memory.

How could the demo code be modified to create a dataset on the fly from a context (query) and a set of examples (documents, which can be loaded from disc beforehand), and then use this in-memory dataset to make a prediction? Thanks.

[xuanhuiwang]

Thanks for the question. We don't have a handy example to predict from the in-memory data. Here is my thought:

I think you may want to construct an input function similarly to https://github.com/tensorflow/ranking/blob/master/tensorflow_ranking/examples/tf_ranking_libsvm.py#L253, as an input function for prediction.

Another example is like: https://github.com/tensorflow/ranking/blob/master/tensorflow_ranking/python/model_test.py#L466. You can feed the dict of features directly using lambda.

[davidmosca]

Thanks for the tips. Does the second option (predictions = self._estimator.predict(input_fn=lambda: (features, None))) also require to rewrite the tutorial's estimator? If so, how? And how can the features be converted into the correct format (the example shows a dictionary of arrays of floating numbers but in the Antique dataset the data (answers) are stored as protobuffers).

[xuanhuiwang]

I think features should be a dict of {name: Tensor/SparseTensors} for predict input_fn. Since you mentioned in-memory data, I assumed that the data is already in the dict format. Otherwise, you need converter or parser to prepare the data in the dict format.

I don't think you need to change the estimator.

[davidmosca]

Yes the problem is how to build the converter/parser, in order to convert the data from strings/numbers/arrays to Tensors. The code of the demo cannot be re-used because (1) the data is read from disk and (2) the context and examples are built simultaneously, whereas in a search context the examples (answers) are known in advance (and can/must therefore be pre-loaded and pre-converted) whereas the context (user question) is only known when the user types it, and must then be converted separately on the fly into the appropriate format. The conversion into Tensors must follow the same logic as that of the predict_input_fn function, but starting from in-memory raw data (text, numbers). The question is: how?

[xuanhuiwang]

@davidmosca, have you considered export the model to SavedModel and use the TensorFlow Serving to do the prediction?

The estimator.predict is mainly for offline analysis or debugging purpose, as far as I can tell. For production, you need to export the model and serve it outside of estimator. See https://www.tensorflow.org/tfx/tutorials/serving/rest_simple#serve_your_model_with_tensorflow_serving.