Pods not picking up roles on EKS

[byakku]

Problem: Pods on cluster are not picking up roles. When using aws sts get-caller-identity default node role is returned, when I assume role manually it works.

Environment: EKS 1.16.2 Kiam 3.5

Logs etc.: NS annotation

kubectl describe ns default

Name:         default
Labels:       <none>
Annotations:  iam.amazonaws.com/permitted: .*
Status:       Active

No resource quota.

No LimitRange resource.

PodAnnotation:

[...]
iam.amazonaws.com/role: kiam/<role-name>

Logs from server:

kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:36:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T08:32:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:36:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T08:32:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:36:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T08:32:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:36:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T08:32:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:47:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"requested new credentials","time":"2020-07-27T08:32:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:47:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"requested new credentials","time":"2020-07-27T08:32:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:47:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T08:43:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:47:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T08:43:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:47:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T08:43:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:47:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T08:43:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:58:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"requested new credentials","time":"2020-07-27T08:43:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:58:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"requested new credentials","time":"2020-07-27T08:43:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:58:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T08:54:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:58:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T08:54:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:58:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T08:54:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T08:58:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T08:54:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:09:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"requested new credentials","time":"2020-07-27T08:54:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:09:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"requested new credentials","time":"2020-07-27T08:54:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:09:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T09:05:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:09:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T09:05:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:09:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T09:05:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:09:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T09:05:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:20:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"requested new credentials","time":"2020-07-27T09:05:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:20:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"requested new credentials","time":"2020-07-27T09:05:58Z"}
kiam-server-g75nv kiam-server ERROR: logging before flag.Parse: W0727 09:10:13.106048       1 reflector.go:341] pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/reflector.go:99: watch of *v1.Pod ended with: too old resource version: 13450094 (13459721)
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:20:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T09:16:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:20:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T09:16:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:20:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T09:16:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:20:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T09:16:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:31:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"requested new credentials","time":"2020-07-27T09:16:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:31:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"requested new credentials","time":"2020-07-27T09:16:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:37:38Z","credentials.role":"kiam/pod-name","level":"info","msg":"requested new credentials","time":"2020-07-27T09:22:38Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:37:38Z","credentials.role":"kiam/pod-name","generation.metadata":0,"level":"info","msg":"fetched credentials","pod.iam.role":"kiam/pod-name","pod.name":"pod-name-c4896b4-c8drz","pod.namespace":"default","pod.status.ip":"","pod.status.phase":"Pending","resource.version":"13462277","time":"2020-07-27T09:22:38Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:31:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T09:27:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:31:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T09:27:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:31:58Z","credentials.role":"/kiam/pod-name","level":"info","msg":"role no longer active","time":"2020-07-27T09:27:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:31:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T09:27:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:42:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"requested new credentials","time":"2020-07-27T09:27:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:37:38Z","credentials.role":"kiam/pod-name","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T09:32:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:37:38Z","credentials.role":"kiam/pod-name","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T09:32:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:47:58Z","credentials.role":"kiam/pod-name","level":"info","msg":"requested new credentials","time":"2020-07-27T09:32:58Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:42:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"notified credentials expire soon","time":"2020-07-27T09:38:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:42:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"expiring credentials, fetching updated","time":"2020-07-27T09:38:57Z"}
kiam-server-g75nv kiam-server {"credentials.access.key":"ACCESSKEY","credentials.expiration":"2020-07-27T09:53:58Z","credentials.role":"kiam/cluster_autoscaler","level":"info","msg":"requested new credentials","time":"2020-07-27T09:38:58Z"}

Trust relationship in AWS is set up correctly, I can assume role properly and role works.

What could I miss/how to debug further?

[pingles]

I suspect there's an issue with either 1) the iptables intercept rules not being configured (and so the agent isn't intercepting the call), 2) your role annotation is not on the right object (we used to see this happen some times when people are hand editing yaml rather than using our internal tooling). Your logs show the server is obtaining credentials successfully, so as long as the role you want to see there isn't showing it's likely one of those (and probably 2).

[rehevkor5]

Is the error log message about watch of *v1.Pod ended with: too old resource version a problem, or is that normal?