Running kube2iam in minikube

[jwenz723]

I am having troubles getting kube2iam to authenticate correctly to AWS (not a mock aws metadata provider) while running kube2iam in minikube on my laptop. I am running minikube on a macbook using hyperkit (if that matters).

Since my minikube instance doesn't have an instance profile, I have deployed kube2iam with the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars set with appropriate credentials for my AWS account. Since I am just testing, the user I am currently using has administrator access to AWS, so should be able to do everything needed.

I have created a role that I want my pod to assume using kube2iam, and have granted a trust relationship to trust the root of the account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::1234567890:root",
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

When my pod tries to authenticate via the role specified in the kube2iam annotation on the pod (iam.amazonaws.com/role: Role-testrole) the pod logs the following error:

{
    "errors": [
        "EC2RoleRequestError: failed to get Role-testrole EC2 instance role credentials\ncaused by: EC2MetadataError: failed to make Client request\ncaused by: SignatureDoesNotMatch: Signature expired: 20190911T150310Z is now earlier than 20190911T161213Z (20190911T162713Z - 15 min.)\n\tstatus code: 403, request id: 02f374a0-d4b1-11e9-b51b-e3221c1a4460\n"
    ]
}

And kube2iam logs this error:

time="2019-09-11T15:03:31Z" level=debug msg="Caches have been synced.  Proceeding with server."
time="2019-09-11T15:03:31Z" level=debug msg="Pod OnUpdate" pod.iam.role= pod.name=kube2iam-m7qwv pod.namespace=kube-system pod.status.ip=192.168.64.3 pod.status.phase=Running
time="2019-09-11T15:03:32Z" level=error msg="Error getting instance id Get http://169.254.169.254/latest/meta-data/instance-id: dial tcp 169.254.169.254:80: connect: no route to host"
time="2019-09-11T15:03:32Z" level=info msg="Listening on port 8181"

and this error:

time="2019-09-11T15:08:14Z" level=error msg="Error assuming role SignatureDoesNotMatch: Signature expired: 20190911T150814Z is now earlier than 20190911T161716Z (20190911T163216Z - 15 min.)\n\tstatus code: 403, request id: b7b31ea9-d4b1-11e9-a89f-37725bed58a6" ns.name=kube-system pod.iam.role=Role-testrole req.method=GET req.path=/latest/meta-data/iam/security-credentials/Role-testrole req.remote=172.17.0.14

I am submitting this issue because there doesn't seem to be any documentation on how to run kube2iam in minikube. I think being able to run kube2iam in minikube (or any type of local cluster) is very valuable to testing deployments before deploying into a real cluster, so documentation on this topic I believe could be very beneficial to other people.

I should also make it known that I am able to successfully assume the AWS role Role-testrole using the AWS cli with the same credentials that I am passing into kube2iam env vars using this command:

aws sts assume-role --role-arn arn:aws:iam::197353405550:role/Role-testrole --role-session-name test

[jwenz723]

I should also note that the way I am currently installing kube2iam into my minikube is via helm with the following command:

helm upgrade -i kube2iam --wait \
    --set=extraArgs.base-role-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/ \
    --set extraArgs.default-role=arn:aws:iam::${AWS_ACCOUNT_ID}:role/${DEFAULT_KUBE2IAM_ROLE_FILE} \
    --set verbose=true \
    --set host.iptables=true \
    --set prometheus.service.enabled=true \
    --set prometheus.serviceMonitor.enabled=true \
    --set aws.access_key=${AWS_ACCESS_KEY_ID} \
    --set aws.secret_key=${AWS_SECRET_ACCESS_KEY} \
    --set aws.region=us-west-2 \
    --namespace kube-system \
    stable/kube2iam

[ghostsquad]

I'm also interested in how to do this

[pre]

Signature expired: 20190911T150814Z is now earlier than 20190911T161716Z (20190911T163216Z - 15 min.)

The error message implies that the clock of minikube virtual machine was lagging. Did you get eventually got your setup working eg. after the clock was properly in time?

[morganchristiansson]

Trying to run it in k3s and getting same error

time="2021-05-29T15:11:17Z" level=error msg="Error getting instance id Get \"http://169.254.169.254/latest/meta-data/instance-id\": dial tcp 169.254.169.254:80: i/o timeout"
time="2021-05-29T15:11:17Z" level=info msg="Listening on port 8181"

then pods will exit after a while. (using kube2iam:latest arm64 docker images)

[zyc-sudo]

Same situation and same errors as anyone else in this post... any news?

[ghostsquad]

@zyc-sudo don't think so. Personally, I've moved on to using the OIDC provider instead of kube2iam for AWS access.

[devinnasar]

@ghostsquad, @zyc-sudo, I'm running into the same problem with kube2iam. @ghostsquad My situation is a bit unique - I'm not running on EKS - I'm running on a local kubernetes cluster but I need to back things up to s3. I don't have an OIDC provider through an existing EKS cluster, and I don't have an external IDP. Well, I have AWS SSO, but I don't know if I can use that. Is there any way to set up IRSA given my circumstances, or am I screwed?

[zyc-sudo]

So far I have not yet made it work :(

On Fri, Feb 18, 2022 at 10:39 PM devinnasar @.***> wrote:

@ghostsquad https://github.com/ghostsquad, @zyc-sudo https://github.com/zyc-sudo, I'm running into the same problem with kube2iam. @ghostsquad https://github.com/ghostsquad My situation is a bit unique - I'm not running on EKS - I'm running on a local kubernetes cluster but I need to back things up to s3. I don't have an OIDC provider through an existing EKS cluster, and I don't have an external IDP. Well, I have AWS SSO, but I don't know if I can use that. Is there any way to set up IRSA given my circumstances, or am I screwed?

— Reply to this email directly, view it on GitHub https://github.com/jtblin/kube2iam/issues/232#issuecomment-1045671387, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEOWLFRD2GAFELCFI5QHJWLU34GGXANCNFSM4IVWIECA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[ghostsquad]

@devinnasar https://minikube.sigs.k8s.io/docs/tutorials/openid_connect_auth/

https://aws.amazon.com/blogs/containers/introducing-oidc-identity-provider-authentication-amazon-eks/