Add support for using Kubernetes Service Accounts to authenticate to Minio similar to AWS IRSA

[EStork09]

Is your feature request related to a problem? Please describe. Many clients already use the AWS SDK or MC to interact with S3/Minio resources. It would be useful to allow a IDP and trust relationship for role assumption similar to AWS does for IRSA, right now we can add keycloak or another IDP with a client secret and key but then have to add the logic to get the JWT into the application, it would be nice to be able to take advantage of the k8s projected JWT tokens for the service accounts to simplify the flow like we have today for AWS. Since k8s 1.21 the ServiceAccountIssuerDiscovery has become an option to allow service account JWTs to interact with external services without having to burden the application to perform the oauth2 flow to get a JWT.

Describe the solution you'd like I think in order to do this we would have to add the ability to create IDPs similar to the way they function in AWS IAM and allow the assumption of a minio service account or user? Or rather than using a custom claim to align the policy there needs to be a way to get the info from the sub similar to the logic around the role assumption in AWS. And the requirement for the client secret/client id would have to be dropped?

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. We could use keycloak but need to create the logic to generate the JWT for the service, or we just continue to use minio serviceaccounts/users and have long running access/secret keys.

Additional context Example usage of the K8s OIDC: https://techblog.cisco.com/blog/kubernetes-oidc (References to Vault auth with k8s-oidc) https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-bound-service-account-tokens (General usage and info of k8s-oidc) https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/ (IAM Role assumption with K8s OIDC) https://banzaicloud.com/blog/kubernetes-oidc/ (Vault auth with k8s-oidc)

[harshavardhana]

I think in order to do this we would have to add the ability to create IDPs similar to the way they function in AWS IAM and allow the assumption of a minio service account or user? Or rather than using a custom claim to align the policy there needs to be a way to get the info from the sub similar to the logic around the role assumption in AWS. And the requirement for the client secret/client id would have to be dropped?

The custom claim is one of the ways, there is something called "rolePolicy" which is the recommended way to do things now. This will already work with our current implementation of OIDC with k8s.

What we are thus planning to do is provide a more native mechanism where apps can simply get new STS tokens based on the k8s service account JWT with MinIO installed.

This would allow them to be headless services while they can get temporary tokens only that automatically expire. This would provide a very EC2 IAM-like experience to applications in k8s.

So wait for it - so there is more coming up.

[pjuarezd]

hi @EStork09, I would like to compliment @harshavardhana's here and share a little sneak peek of what he is talking about.

Below is the diagram of what we are intending to do, it's an approach more similar to IMDS IAM Credentials, but closelly integrated with MinIO Operator in the center of the picture.

We don't do commitments yet, and this workflow or some detail could change, but we are seeking the simplest and more effective way to allow applications authentication in MinIO using Service Accounts JWT tokens on Kubernetes.

We are seeking here the MinIO's minimalist approach, AWS IRSA is complicated to setup and requires multiple steps to be operational, on Operator STS (name pending to be oficialized) we are looking to have it ready to go as soon as the Operator is running.

I am going to close this ticket for now, please keep an eye on Operator, we are likelly going to release it as a Beta in some of the new Operator releases sooner than later.

[gclawes]

Is there a way to accomplish this for minio installations running outside of a Kubernetes cluster w/ Operator?

[harshavardhana]

Is there a way to accomplish this for minio installations running outside of a Kubernetes cluster w/ Operator?

@gclawes unfortunately you would re-build everything that k8s provides. You are better off using k8s.

[gclawes]

I'm not suggesting dropping k8s entirely, I'm trying to see if there's a way for applications running in a k8s cluster to use their serviceaccount tokens to authenticate to a minio cluster running outside of k8s.

[EStork09]

I don't 100% know how the implementation for the STS will be done. But in the experience with AWS STS, if you setup the ServiceAccountIssuerDiscovery and properly expose that you should be able to use your clusters serviceaccounts to access external resources. That is how IRSA works with AWS, I assume this will be somewhat similar but I am okay being corrected.

[dekarl]

Just found this issue while trying to use kubernetes service accounts to authenticate against a minio instance running on a dedicated truenas box next to a stateless cluster.

Is minio outside of the cluster going to be a supported use case?

From reading the issue it appears as if all pieces are already in place, just simple configuration for this minimal use case is missing. I'd love to have the same progam run on AWS and on-premises, both "just working" from a developers point of view.

I was expecting to configure a trust to my cluster as an OIDC IDP by setting the issuer, "sub" as identifier and "mycluster:" as prefix, thus using iam roles named "mycluster:system:serviceaccount:namespace:nameofserviceaccount".

But minio wants a clientid and clientsecret instead of letting me configure its audience.

Looks like AWS SDK needs a way to specify a custom STS endpoint via environment variable. (e.g. AWS_STS_DEFAULT_ENDPOINT mentioned in https://github.com/aws/aws-sdk-java/issues/2343 )

Then all pieces of this puzzle would just fall into place. With the PolicyBindingSpec from the operator being a nice KISS implementation of iam roles.

(an option to trust multiple IDPs in the STS would be great, so we can use one IDP for humans and multiple for services)

PS: AWS IRSA is not that hard to setup if you are using the CDK, see https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks.ServiceAccount.html#example it's some lines of setup. Then creating a serviceaccount and bucket is three lines of code. Looking forward to doing a similar setup with cdktf and cdk8s.

[harshavardhana]

Please read about it here https://github.com/minio/operator/blob/8b7654688fafae8459ccc505934f915daa45d785/docs/STS.md#minio-operator-sts

[gclawes]

@harshavardhana that document looks like it's only for minio tennants inside the cluster. @dekarl and I are both wondering about minio installs external to a cluster, which is fairly important for stateless kubernetes clusters. Can the operator be made to provide STS for a minio install not directly managed by it and outside of it's kubernetes cluster?

[harshavardhana]

@harshavardhana that document looks like it's only for minio tennants inside the cluster. @dekarl and I are both wondering about minio installs external to a cluster, which is fairly important for stateless kubernetes clusters. Can the operator be made to provide STS for a minio install not directly managed by it and outside of it's kubernetes cluster?

There are further plans on extending this.

[gclawes]

Thanks, good to hear! Otherwise, the only other thing I can't figure out that applies to both cases is setting the STS endpoint for an app using the AWS S3 SDK, I'm not even sure if that is possible though.

[pjuarezd]

this is an example using the boto3 SDK https://github.com/minio/operator/tree/master/examples/kustomization/sts-example/sample-clients/aws-sdk/python

you would basically pass the endpoint_url parameter to the boto client

https://github.com/minio/operator/blob/29e7431e6e7e1cdb8912b3268aba780f2265bc33/examples/kustomization/sts-example/sample-clients/aws-sdk/python/main.py#L47-L49