search

triton-inference-server / local_cache

Implementation of a local in-memory cache for Triton Inference Server's TRITONCACHE API
BSD 3-Clause "New" or "Revised" License
4 stars 1 forks source link

Local cache on sagemaker #11

Open andompesta opened 7 months ago

andompesta commented 7 months ago

Experiencing this error when starting a sagemaker endpoint with local-cache: error: creating server: Invalid argument - unable to find 'libtritoncache_local.so' for cache. Searched: /opt/tritonserver/caches

endpoint image: 746233611703.dkr.ecr.eu-central-1.amazonaws.com/sagemaker-tritonserver:23.12-py3 environment variables:

"Environment": {
                "SAGEMAKER_TRITON_DEFAULT_MODEL_NAME": "ensemble",
                "SAGEMAKER_TRITON_THREAD_COUNT": "10",
                "SAGEMAKER_TRITON_LOG_INFO": "true",
                "SAGEMAKER_TRITON_LOG_VERBOSE": "true",

                "SAGEMAKER_TRITON_ALLOW_METRICS": "true",
                "SAGEMAKER_TRITON_PUBLISH_METRICS_TO_CLOUDWATCH": "true",
                "SAGEMAKER_TRITON_CLOUDWATCH_LOG_GROUP": f"/aws/SageMaker/Endpoints/TritonMetrics/{EXPERIMENT}",
                "SAGEMAKER_TRITON_CLOUDWATCH_METRIC_NAMESPACE": f"/aws/SageMaker/Endpoints/TritonMetrics/{EXPERIMENT}",
                "SAGEMAKER_TRITON_METRICS_PORT": "22000",
                "SAGEMAKER_TRITON_ADDITIONAL_ARGS": "--cache-config local,size=1000000",
            },

Any idea ho to add libtritoncache_local to sagemaker image ?

oandreeva-nv commented 4 months ago

Hi @andompesta , apologies for the late response. Could you please confirm that libtritoncache_local.so is located under /opt/tritonserver/caches directory? (cc @rmccorm4 )