Open taer opened 6 months ago
I directly hit the eks-pod-identity pod inside a container agent via
IIUC, you are manually calling the AWS_CONTAINER_CREDENTIALS_FULL_URI
with the token from inside a pod running on your node and use the credentials to make this aws sts get-caller-identity
cli call.
Trying to understand if these credentials ever work from inside a Pod when an application is trying to reach Kafka and fail when using these credentials manual with aws cli? or they never work with Kafka?
Yeah. basically, I was doing what the SDK does manually just to validate.
The credentials given by the pod-identity work perfectly w/ Kafka until they reach their expiration. When the SDK goes back to the pod-identity-agent near the expiration, the agent refreshes the tokens for it. The issue though is it refreshes it with a new sessionName. And the MSK IAM auth scheme hates that. It considers the session-name change to be a change in principal, and screams
failed authentication due to: Cannot change principals during re-authentication from IAM.arn:aws:sts::BLAH:assumed-role/prd-use1-default-bookingApiPod-20240529174141238800000002/eks-k8s-wl-prd-engine-boo-064f1ed1-2349-4774-b895-9a69ccc3eeb1: IAM.arn:aws:sts::BLAH:assumed-role/prd-use1-default-bookingApiPod-20240529174141238800000002/eks-k8s-wl-prd-engine-boo-875fdfcd-19cd-4ac3-8544-7077f94a6e39
important parts in there is the change here
arn:aws:sts......roleName/eks-k8s-wl-prd-engine-boo-064f1ed1-2349-4774-b895-9a69ccc3eeb1
arn:aws:sts......roleName/eks-k8s-wl-prd-engine-boo-875fdfcd-19cd-4ac3-8544-7077f94a6e39
That sessionName of pod-name-uuid is generated by the eks-pod-identity-agent. Ideally, we could configure the agent to just use the pod-name and not include the uuid, and we'd be golden.
fyi code is here now - https://github.com/aws/eks-pod-identity-agent - can we please move this to an issue there? 🙏🏾
I've opened feature request asking for support of Custom Session Tags and Custom Session Names in the above repo:
following updates to this issue
I think this is probably more of a bug report, but I cannot find the proper channels. Version Info: EKS: 1.29 Eks Pod ID agent: v1.2.0-eksbuild.1
We are using eks-pod-identity. It's been working great till we started using IAM based kafka.
I directly hit the eks-pod-identity pod inside a container agent via
I took the resulting AccessKey, SecertKey, and Token, and used them to do a
aws sts get-caller-identity
The SessionName is non static, and I can't find any way to force set it. The issue comes from the MSK IAM usage. MSK doesn't allow "reauthentication". We get this error when the session name changes
Most services we've used IAM for to date don't care. The solution normally would be to set
AWS_ROLE_SESSION_NAME
when calling STS:AssumeRole. But we're not calling that, the eks-pod-identity pod is. The ContainerCredentialProvider in the SDK is just calling theAWS_CONTAINER_CREDENTIALS_FULL_URI
URL with the contents ofAWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
as the Authorization header. There is no option as far as I've found to have that include a constant session name to satisfy MSK.Thanks!