search

aws / sagemaker-huggingface-inference-toolkit

Apache License 2.0
240 stars 60 forks source link

Bind custom model function #41

Closed dlesomewhere closed 2 years ago

dlesomewhere commented 2 years ago

Issue #40

Description of changes:

This commit changes the way the custom model_fn is bound to the instance of the HuggingFaceHandlerService so the custom model_fn has access to the context available to the handler.

This will allow us to use that context, for example self.device to ensure we properly set the device when creating a custom pipeline.

This PR also includes the fix to a failing test from https://github.com/aws/sagemaker-huggingface-inference-toolkit/pull/38.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

philschmid commented 2 years ago

Hello @dlesomewhere,

thank you for opening the PR. But that's something we cannot add since it would break first of all the backward compatibility with any existing custom inference script. Additionally, it wouldn't be longer compliant with the SageMaker Inference contract. The SageMaker Inference contract is equal across all Inference Toolkits on SageMaker (MXNET, PyTorch, TensorFlow...). You can take a look at the documentation

dlesomewhere commented 2 years ago

Hi @philschmid - thanks for the link to the SageMaker inference contract. I agree with your reasoning! Will close this PR.