search

awslabs / multi-model-server

Multi Model Server is a tool for serving neural net models for inference
Apache License 2.0
998 stars 230 forks source link

Custom Python Pre-/Postprocessing possible? #813

Closed henningsway closed 5 years ago

henningsway commented 5 years ago

I'm currently looking for a suitable serving-framework for my feedforward neural net. I currently bundle some (custom) python preprocessing in a Sklearn Pipeline and also do some postprocessing in the predicitons.

It may be possible to port these steps to ONNX, but I would accept less speed and a bigger model-size if I could get around it. :)

Does MMS provide a possibility to pre- and postprocess the predictions of a model? Can these custom steps be included in a Model archive?

vdantu commented 5 years ago

MMS can handle custom pre-process and post-process and prediction for a model. You could refer to the following batching example code in the repository. You can replace the "preprocess", "inference" and "postprocess" with your custom code and add that modified file to your model-archive.

MMS only requires an entry-point for the model. In the above example code, that method is called "handle". You can organize your custom processing code within this entry-point and then create a model-archive by giving the "--handler :" .

In this way, if you use different frameworks for your custom processing, you could install all the dependent frameworks in side your MMS container and invoke those frameworks from your entry-point.

Does this explanation help?

henningsway commented 5 years ago

Yes, indeed. The explanation and the example are very helpful. Thanks! :)