Closed hubenjm closed 6 years ago
Thanks @hubenjm .
I'm guessing the empty entry_point.py file is a problem here. The Python SDK README has more documentation about what functions are expected to serve an MXNet model. In general, this is model_fn
, input_fn
, predict_fn
, and output_fn
. However, there are default implementations that kick in if you don't specify them. I'm guessing the default isn't suited to your particular use case. You may need to check your CloudWatch logs to get a finer grained understanding of the high-level "ping health check" error.
Would it be helpful to have links to the Python SDK documentation in this repository? Or had you already found that, but just couldn't find the serving information you needed in there? In which case, we'd be open to feedback on how we make that clearer.
The predictor.predict
function is quite flexible. You can set predictor.serializer
to a function which will serialize your data before passing to the endpoint. And then you can use input_fn
within the entry_point file to further manipulate the data before passing to your network for inference. This could include resizing or loading from an image_url (although there may be latency implications for that).
You can pip install from your entry_point script to add libraries that are not installed by default in the container. e.g.
import pip
pip.main(['install', '<my_package_name_here>'])
import <my_package_name_here>
Thanks, and hope this helps.
Thanks for that information. I was essentially trying to follow along with this example: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/mxnet_mnist_byom/mxnet_mnist.ipynb but I wasn't sure if it would work since the provided mnist.py file doesn't include anything about how inference is handled, hence my confusion.
Anyway, looking at the Python SDK readme, it seems that https://github.com/aws/sagemaker-python-sdk/blob/master/README.rst#deploying-endpoints-from-model-data is most relevant to my intended use case. It seems no sample transform_script.py is provided, but I'm guessing it would have to include an input_fn, predict_fn, output_fn and model_fn as you mentioned?
Thanks, @hubenjm . You raise a good point that we should be more explicit about defining the functions and discussing what's going on in the MXNet BYOM example. I've added that to our backlog.
There are two parts to BYOM hosting using MXNet and the Python SDK:
model_fn
, input_fn
, and/or predict_fn
functions. If you don't define them, simple defaults will be used. This functionality is specific to MXNet..deploy()
. That's the section that your link directs to. The section is for MXNet, but the same type of functionality exists for TensorFlow, built-in algorithms, and bring your own container.Just as an update, I figured out that the problem was actually how I packaged the model.tar.gz file. Apparently the script in the example doesn't pack it correctly, or at least I wasn't able to unpack the resulting .tar.gz on my local machine. This code is located here and goes as follows:
import os
import json
os.mkdir('model')
model.save_checkpoint('model/model', 0000)
with open ( 'model/model-shapes.json', "w") as shapes:
json.dump([{"shape": model.data_shapes[0][1], "name": "data"}], shapes)
import tarfile
def flatten(tarinfo):
tarinfo.name = os.path.basename(tarinfo.name)
return tarinfo
tar = tarfile.open("model.tar.gz", "w:gz")
tar.add("model", filter=flatten)
tar.close()
I ended up just manually packing the files myself from the command line.
The other problem I found was that the model files must be named precisely as "model-0000.params", "model-shapes.json", and "model-symbol.json", respectively. Any other names for these files will lead to the ping health check error from what I can tell.
Once this was figured out my original approach of just passing an empty entry_point.py
file when calling MXNetModel worked fine. I obtained an MXNetPredictor object named predictor
by either calling the deploy
function from the MXNetModel object or by calling predictor = MXNetPredictor(endpoint_name = endpoint_name, sagemaker_session = sagemaker_session)
using an existing endpoint name. Either way, then I could perform inference in one of two ways:
Method 1:
endpoint_name = predictor.endpoint # predictor is an MXNetPredictor object
runtime = boto3.Session().client(service_name='runtime.sagemaker',region_name='us-east-1')
j = json.dumps(img.tolist()) # img is a np.ndarray object with shape (1,3,224,224)
response = runtime.invoke_endpoint(EndpointName=endpoint_name, ContentType='application/json', Body=j, Accept='application/json')
result = response['Body'].read()
result = np.array(json.loads(result)).ravel()
Method 2: (much simpler)
result = np.array(predictor.predict(img.tolist())).ravel() # img is np.ndarray object with shape (1,3,224,224)
With regards to Method 1 above, it wasn't obvious to me where to find out that I had to use the 'application/json' content_type when invoking the endpoint. Nor was it obvious where to find how the image should be serialized properly.
@hubenjm
Hi, thanks for using sagemaker and providing your knowledge to the community!
I want to add one more comment to your solution. As @djarpin pointed out, sagemaker python sdk allows user to define some specific functions in the entry point script. Specially to this model loading problem, model_fn() is what you need.
If this model_fn() is not provided and the default model_fn() is invoked, then as you have already found, you need to package the model according to the requirement on file structure and naming.
But if you provide your own model_fn() that defines how your model should be loaded. Then you don't need to worry much about the format. Models will be loaded your way.
Closing for now. Feel free to re-open if you continue to experience problems. Thanks.
I have a model that I've trained in MXNet to classify images, and I already have the model assets saved as model.tar.gz in an s3 bucket.
I just want to be able to deploy this within a sagemaker notebook to a host and then call the predictor.predict function on an input image. However, the above sagemaker_model.deploy call fails and yields following error message:
ValueErrorTraceback (most recent call last)