search

awslabs / multi-model-server

Multi Model Server is a tool for serving neural net models for inference
Apache License 2.0
998 stars 230 forks source link

Issues with deploying inference model with sklearn Sagemaker container #929

Open sertaco opened 4 years ago

sertaco commented 4 years ago

Hi,

I have an inference model that works fine when I deploy to and Endpoint on Sagemaker. However when I use this model to initiate a multi model endpoint I am receiving this strange Response serialization error.

com.amazonaws.ml.mms.wlm.WorkerLifeCycle - TypeError: Object of type Response is not JSON serializable

More relevant CF error logs:

com.amazonaws.ml.mms.wlm.WorkerLifeCycle - MXNet worker started.
com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Unable to serialize model output.
com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):
com.amazonaws.ml.mms.wlm.WorkerLifeCycle -   File "/miniconda3/lib/python3.7/site-packages/mms/protocol/otf_message_handler.py", line 126, in create_predict_response
com.amazonaws.ml.mms.wlm.WorkerLifeCycle - json_value = json.dumps(val, indent=2).encode("utf-8")
com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/miniconda3/lib/python3.7/json/__init__.py", line 238, in dumps
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Response is not JSON serializable

I think Sagemaker Scikit learn container is supported for Multi model endpoints. However I am seeing MXNet is being kicked here.

I am thinking to create my own container with MME, not being able to solve this. What is your advice?