Toolkit for allowing inference and serving with PyTorch on SageMaker. Dockerfiles used for building SageMaker Pytorch Containers are at https://github.com/aws/deep-learning-containers.
Apache License 2.0
131
stars
70
forks
source link
Documentation for inference.py `transform_fn` #142
What did you find confusing? Please describe.
Huggingface have documented how to use the sagemaker pytorch inference API in order to host their models. They make it quite clear that you must supply model_fn and then either transform_fn or (input_fn, predict_fn and output_fn). By using transform_fn you can have fine control of batch size for example, allowing you to handle large requests (in particular I have an issue where my batch transform jobs continuously die because the minimum payload of 1MB is way to large for my model - due to the large intermediate matrices I..e probabilities = batch_szie x num_labels)
What did you find confusing? Please describe. Huggingface have documented how to use the sagemaker pytorch inference API in order to host their models. They make it quite clear that you must supply
model_fnand then eithertransform_fnor (input_fn,predict_fnandoutput_fn). By usingtransform_fnyou can have fine control of batch size for example, allowing you to handle large requests (in particular I have an issue where my batch transform jobs continuously die because the minimum payload of 1MB is way to large for my model - due to the large intermediate matrices I..e probabilities = batch_szie x num_labels)I cannot find any mention of
transform_fnin the documentation - https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.htmlIt is mentioned in passing in one of the examples - https://sagemaker-examples.readthedocs.io/en/latest/frameworks/pytorch/get_started_mnist_deploy.html
Describe how documentation can be improved Document the use of
transform_fnas an alternative toinput_fn,predict_fnandoutput_fnAdditional context [Add any other context or screenshots about the documentation request here.]
This is how I was aware of
transform_fn:https://aws.amazon.com/blogs/machine-learning/run-computer-vision-inference-on-large-videos-with-amazon-sagemaker-asynchronous-endpoints/
The I found this:
(https://huggingface.co/docs/sagemaker/inference)