search

aws / sagemaker-inference-toolkit

Serve machine learning models within a 🐳 Docker container using 🧠 Amazon SageMaker.
Apache License 2.0
372 stars 82 forks source link

End-to-End example for *NON* multi model deployment #53

Open nvs-abhilash opened 4 years ago

nvs-abhilash commented 4 years ago

What did you find confusing? Please describe. Based on the documentation the complete example multi_model_bring_your_own, it seems that sagemaker-inference-toolkit is only for multi-model requirements. But I have also seen links for sagemaker_pytorch_serving_container which suggests that that is not the case.

There is no clear instruction in the documentation or an end-to-end example link indicating that it can be used for single model hosting scenarios as well.

Describe how documentation can be improved You can provide one more end-to-end example for single model hosting, along with some points in favor of using this python package, instead of designing our own docker containers from scratch.

Additional context

chuyang-deng commented 4 years ago

Hi @nvs-abhilash, Thanks for bringing this to our attention, we are planning to introduce an example of sagemaker-inference-toolkit + multi-model-server.

Meanwhile, you can refer to these instructions to implement your own handler service:

custom handler service: https://github.com/awslabs/multi-model-server/blob/master/docs/custom_service.md#example-custom-service-file

custom inference handler: https://github.com/aws/sagemaker-inference-toolkit/blob/master/README.md#implementation-steps

nvs-abhilash commented 4 years ago

Hi, my confusion is not how to use it with Multi Model Server, I think it is very well documented in the MMS SageMaker Example.

What we don't have is an example for "Bring Your Own Model" using sagemaker-inferece-toolkit.

As we don't have that, it looks like sagemaker-inferece-toolkit can only work with multi-model-server.

Maybe a sister example of scikit_bring_your_own, but using sagemaker-inferece-toolkit.

Let me know if that makes sense.

mb-dev commented 3 years ago

I am looking for the same information, and agree that such an example will be useful.

RyoMazda commented 3 years ago

I've been struggling for the same issue. Any update on this?

DFranch commented 3 years ago

Hi, I've also been looking for examples of applying sagemaker-inference-toolkit for a single model deployment not using PyTorch. The best example which might also benefit others is the following repo from a AWS workshop: https://github.com/awslabs/amazon-sagemaker-mlops-workshop.

RichardSieg commented 2 years ago

This is a particular issue when trying to host the model on the new serverless variant which does not support the multi-model hosting, see excluded features. Trying to host the model with my own flask API now.

Leon-Kani-Dong commented 1 year ago

Hi, I have the same issue now. It has been a few years. Any updates?