Image used: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:2.1.0-transformers4.37.0-gpu-py310-cu118-ubuntu20.04 with custom inference.py.
I am loading some essential data from S3 for post processing on model's output and I included the data loading in the custom model_fn function. The data takes around 3~5 minutes to load. One thing I noticed is that the endpoint creation or update will complete before model_fn returns so the endpoint becomes available for incoming calls before all the data and model is loaded. This resulted in several minutes of additional latency around the period of time when endpoint is created or updated. How can I prevent this from happening?
Image used:
763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:2.1.0-transformers4.37.0-gpu-py310-cu118-ubuntu20.04with custominference.py.I am loading some essential data from S3 for post processing on model's output and I included the data loading in the custom
model_fnfunction. The data takes around 3~5 minutes to load. One thing I noticed is that the endpoint creation or update will complete beforemodel_fnreturns so the endpoint becomes available for incoming calls before all the data and model is loaded. This resulted in several minutes of additional latency around the period of time when endpoint is created or updated. How can I prevent this from happening?