I have a deployment with 2 replicas. Time to time one of the pod can't get IAM credentials and it fails when I try to load a secret from secret manager.
Unable to locate credentials. You can configure credentials by running "aws configure".
If I delete that pod, it will work fine. It seems kube2iam can't handle if there is more than one concurrent request.
It is not a heavily loaded EKS cluster. Just a few pods running as a test on c5.xlarge instance. So it is not resource issue.
I set replica 10, half of them failed to get IAM credentials.
I had to add check for return code of aws secretsmanager get-secret-value which is not really perfect solution.
Hi,
I have a deployment with 2 replicas. Time to time one of the pod can't get IAM credentials and it fails when I try to load a secret from secret manager.
If I delete that pod, it will work fine. It seems kube2iam can't handle if there is more than one concurrent request. It is not a heavily loaded EKS cluster. Just a few pods running as a test on c5.xlarge instance. So it is not resource issue. I set replica 10, half of them failed to get IAM credentials. I had to add check for return code of
aws secretsmanager get-secret-valuewhich is not really perfect solution.Is there any solution for this issue?