consul not using iam role linked to service account on EKS

[Shaiou]

Overview of the Issue

I'm currently running EKS and using the official helm chart to deploy a consul cluster that would join an external cluster using the cloud autodiscover. I followed the AWS doc to modify the serviceaccount in order for it to map to an IAM role. However the consul ( version 1.8.0 ) ignores these credentials and tries to access the node metadata ( which fails because I set up the recommendation in order to block access to node instance profile from pods ) . Can someone help me around that please ?

• the extra config from the pod

  / # cat /consul/userconfig/consul/config.json 
  {"primary_datacenter":"eu-west-3-int-test","retry_join_wan":["provider=aws tag_key=Name tag_value=int-infra-consul-test-server region=eu-west-3"]}
  / # 

• The env vars seem ok from the pod

  / # env |grep AWS
  AWS_ROLE_ARN=arn:aws:iam::XXXXXX:role/YYYYYY
  AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

• The error logs from the pod

  2020-08-18T18:56:45.264Z [INFO]  agent: discover-aws: Address type  is not supported. Valid values are {private_v4,public_v4,public_v6}. Falling back to 'private_v4': cluster=WAN
  2020-08-18T18:56:45.264Z [INFO]  agent: discover-aws: Region is eu-west-3: cluster=WAN
  2020-08-18T18:56:45.318Z [INFO]  agent: discover-aws: Filter instances with Name=int-infra-consul-test-server: cluster=WAN
  2020-08-18T18:57:26.042Z [ERROR] agent: Cannot discover address: cluster=WAN address="provider=aws tag_key=Name tag_value=int-infra-consul-test-server region=eu-west-3" error="discover-aws: DescribeInstancesInput failed: NoCredentialProviders: no valid providers in chain. Deprecated.
  For verbose messaging see aws.Config.CredentialsChainVerboseErrors"
  2020-08-18T18:57:26.042Z [WARN]  agent: Join cluster failed, will retry: cluster=WAN retry_interval=30s error="No servers to join"

• After installing AWS CLI on the pod the vars seem to work ( the assumed role matched the AWS_ROLE_ARN)

  / # aws sts get-caller-identity
  {
  "Account": "XXXXXXXXXXXXXXX", 
  "UserId": "ZZZZZZZZZZZZZZZ:botocore-session-1597777922", 
  "Arn": "arn:aws:sts::XXXXXXXXXX:assumed-role/YYYYYYYYY/botocore-session-1597777922"

Operating system and Environment details

• EKS: v1.17.9-eks-4c6976
• CONSUL: 1.18.0

[nagarwal28]

Are you running this on EKS fargate?

[Shaiou]

Apologies for the delay. I'm not running EKS Fargate. Only the cluster itself is manage. I'm running cusotm nodes/ASG with the standard node ami from AWS.

[nagarwal28]

Even I got this issue, it was on EKS Fargate. You can maybe try with KIAM.

I also found out that consul makes a call to instance metadata, which is why it was failing on fargate. https://www.hashicorp.com/blog/consul-auto-join-with-cloud-metadata

[Shaiou]

Thx, I figured out how to work around it. Just wanted to notify the issue so they make sure they are up to date with all the latest AWS authentication methods in case they are not just using the AWS sdk

[nagarwal28]

Great that you fixed it! Can you share your solution?

[Shaiou]

My bad for the confusion. I did not fix it, I used a dirty workaround to use a DNS entry and an alb to avoid using the aws native discovery. That is until it's fixed in consul itsef

[wildtapir]

Hello, comment for bumping up issue attention.