search

aws / sagemaker-pytorch-inference-toolkit

Toolkit for allowing inference and serving with PyTorch on SageMaker. Dockerfiles used for building SageMaker Pytorch Containers are at https://github.com/aws/deep-learning-containers.
Apache License 2.0
134 stars 72 forks source link

How do I access Custom Attributes from the model during inference? #111

Open bml1g12 opened 2 years ago

bml1g12 commented 2 years ago

What did you find confusing? Please describe. It is clear how to invoke an endpoint with Custom Attributes but it is completely unclear to me how one would access this information from the Pytorch SDK inference code. I can see for the Tensorflow SDK they have documentation for it and imply the second argument of the input handler function will provide access to it but it seems in the Pytorch framework it is just a string indicating the input content MIME type.

It would seem from the source that the context object is not passed to the transformation functions, so not accessible at all to the user.

Describe how documentation can be improved Explanation of how to access provided custom attributes using the Pytorch Inference Toolkit

Additional context I want to achieve what is being done here https://aws.amazon.com/marketplace/pp/prodview-5jlvp43tsn3ny?sr=0-1&ref_=beagle&applicationId=AWSMPContessa - namely the ability to provide a confidence threshold to an inference request. I would like to use a binary format for the body of the payload (image/jpeg) so I cannot provide parameters as part of the body itself like if the body were a JSON.

alar0330 commented 2 years ago

You would have to implement your custom Transformer and set it to be used by the stack's serving Handler Service.