search

triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
BSD 3-Clause "New" or "Revised" License
8.08k stars 1.45k forks source link

Support IAM roles for AWS remote repositories #6651

Open markthill opened 9 months ago

markthill commented 9 months ago

Is your feature request related to a problem? Please describe. Ease of using remote repositories

Describe the solution you'd like Currently testing Triton server on AWS EKS. When using remote repositories downloaded from S3 we have determined that we can set environment variables or use the aws config, but IAM roles are not supported. Triton documentation spells this out exactly and therefore adding as a feature request, but allowing for the normal order of the AWS credential process (to include IAM roles) would go a long way to make the process less cluttered. Deployments on EKS supports attaching IAM roles to the containers and thus would be useful in the remote repository process when pulling from S3.

Describe alternatives you've considered The environment variables and aws config both work, but with the current process it would require wiring secrets with aws credential secrets and keys.

Additional context n/a

kthui commented 9 months ago

Hi @markthill, thanks for proposing an enhancement. Can you confirm if what you are asking is allow Triton S3 credentials to work/resolve out of the box on AWS platforms (i.e. EKS, EC2, ...), without needing to set environment variables explicitly by the user?

markthill commented 9 months ago

That is correct. My scenario is with EKS because I'm using the Docker image to run Triton server, but it would hold true with any of the AWS services where running Triton server is possible (EKS, ECS, EC2...) I'm guessing. I have not tested against anything other than EKS, but having an IAM role associated with an EKS container running Triton server will allow me to access my S3 resources given an installed AWS CLI in that container, but Triton will not pick up that IAM role's credentials/permissions and fails to pull from S3 given a model repository starting with s3://. Hope that helps!

kthui commented 9 months ago

Thanks for the update, I have filed a ticket for us to investigate further.

lame-login-name commented 4 days ago

Hi, why was this request just dropped? This is a really important feature request, this app should be using kube workload identity and IAM, not relying on key pairs. WE NEED THIS.

@kthui