EKS ALB Ingress Controller with KIAM

[lemonshow]

I am using below reference for EKS, https://github.com/uswitch/kiam/pull/112/commits/3ad8aef0012cb321c46072d24a402f9d3655385a#diff-561cdf8ad04b51d17e7e72996f396697R7

Server/Agent are up.

For configmap below,

apiVersion: v1
kind: ConfigMap
metadata:
name: kiam-config
namespace: kube-system
data:
assume-role-arn: 
role-base-arn:

I used KIAM to allow ingress controller pod to create ALB, worked with kube2iam, but I see below AccessDenied from the server,

{"level":"error","msg":"error requesting credentials: AccessDenied: User: arn:aws:sts::1222222233333:assumed-role/eks-alb-ingress-controller/1534456292466473919 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1222222233333:role/eks-alb-ingress-controller\n\tstatus code: 403, request id: 8a31470d-a19e-11e8-b39a-77382ecc1195","pod.iam.role":"eks-alb-ingress-controller","time":"2018-08-17T02:03:43Z"}
{"generation.metadata":0,"level":"error","msg":"error warming credentials: AccessDenied: User: arn:aws:sts::1222222233333:assumed-role/eks-alb-ingress-controller/1534456292466473919 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1222222233333:role/eks-alb-ingress-controller\n\tstatus code: 403, request id: 8a31470d-a19e-11e8-b39a-77382ecc1195","pod.iam.role":"eks-alb-ingress-controller","pod.name":"alb-ingress-controller-75c99b886d-g2g45","pod.namespace":"kube-system","pod.status.ip":"","pod.status.phase":"Pending","resource.version":"4657153","time":"2018-08-17T02:03:43Z"}

Client log

{"level":"warning","msg":"error finding role for pod: rpc error: code = Unavailable desc = there is no address available","pod.ip":"10.55.5.13","time":"2018-08-17T02:03:48Z"}
{"addr":"10.55.5.39:50834","headers":{},"level":"info","method":"GET","msg":"processed request","path":"/ping","status":200,"time":"2018-08-17T02:03:48Z"}
{"level":"warning","msg":"error finding role for pod: rpc error: code = Unavailable desc = there is no address available","pod.ip":"10.55.5.13","time":"2018-08-17T02:03:49Z"}
{"addr":"10.55.5.13:54314","level":"error","method":"GET","msg":"error processing request: rpc error: code = Unavailable desc = there is no address available","path":"/latest/meta-data/iam/security-credentials/","status":500,"time":"2018-08-17T02:03:50Z"}
{"addr":"10.55.5.13:54314","headers":{"Content-Type":["text/plain; charset=utf-8"],"X-Content-Type-Options":["nosniff"]},"level":"info","method":"GET","msg":"processed request","path":"/latest/meta-data/iam/security-credentials/","status":500,"time":"2018-08-17T02:03:50Z"}
{"level":"warning","msg":"error finding role for pod: rpc error: code = Unavailable desc = there is no address available","pod.ip":"10.55.5.13","time":"2018-08-17T02:03:50Z"}

Here is my setting,

assume-role-arn: arn:aws:iam:: 1222222233333:role/eks-alb-ingress-controller
role-base-arn: arn:aws:iam:: 1222222233333:role/

Ingress controller annotation,

metadata:
annotations:
iam.amazonaws.com/role: eks-alb-ingress-controller

Similar role worked for kube2iam, wonder if I get what KIAM is looking for correctly.

My worker node policy: eks-node-instance,

{
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::1222222233333:role/eks-alb-ingress-controller"
}

My eks-alb-ingress-controller trust,

{
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::1222222233333:role/eks-node-instance"
      },
      "Action": "sts:AssumeRole"
    }

[pingles]

It looks like an issue with the roles- the role your Server is assuming when it attempts to request credentials on behalf of a client isn't able to assume the annotated role.

https://github.com/uswitch/kiam/pull/139 includes an update in the IAM documentation. Could you take a look through and see if anything helps you?

If not, it'd be useful to know:

1) The name of the role your server is assuming via the assume role flag. 2) The name of the role your Pod is attempting to assume 3) That your server is running on nodes that can assume the role in 1) 4) That your role in 2 has sufficient assume/trust policy that the role in 1 can assume it

[stafot]

I am facing exactly the same problem on EKS while trying to get access to AWS Elasticsearch service.