IRSA with assumeRoleChain problem

[fgiannetti]

Hi All! I neet to manage S3 Buckets in a remote account but I am facing a problem trying to authenticate to the remote AWS account using IRSA and assumeRoleChain.

AWS Config: I configured an IAM Role inside Account1 (1111111111), named local-role, that has this Trust Relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::1111111111:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/SOMETHUMBPRINT"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.us-east-1.amazonaws.com/id/SOMETHUMBPRINT:sub": "system:serviceaccount:crossplane-system:provider-aws-iam-*"
                }
            }
        }
    ]
}

This Role has attached an IAM Policy that allows to assume role in the remote account Acocunt2 (2222222222) named remote-role

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::2222222222:role/remote-role"
    }
}

In Account2, remote-role allows the Account1 local-role to execute AssumeRole. All this configuration was tested manually using the AWS cli and works fine, creating and listing a buckets.

Crossplane Config I was following this guide to configure AWS Provisioner and ProvisionerConfig.

I created an DeploymentResourceConfig annotating the ServiceAccount to use the local-role:

kind: DeploymentRuntimeConfig
metadata:
  name: s3-runtime-config
spec:
  serviceAccountTemplate:
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::1111111111:role/local-role

The Provider is using this DeploymentRuntimeConfig:

kind: Provider
metadata:
  name: provider-aws-s3
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v1.1.1
  runtimeConfigRef:
    name: s3-runtime-config

And the ProvisionerConfig:

kind: ProviderConfig
metadata:
  name: s3-provider-config
spec:
  credentials:
    source: IRSA
  assumeRoleChain:
    - roleARN: "arn:aws:iam::2222222222:role/remote-role"

The generated ServiceAccount is annotated with eks.amazonaws.com/role-arn: arn:aws:iam::1111111111:role/local-role

When I try to create a bucket using this yaml:

kind: Bucket
metadata:
  name: test-bucket-crossplane
spec:
  forProvider:
    region: us-east-1
  providerConfigRef:
    name: s3-provider-config

It is Synced = False and getting this error:

connect failed: cannot initialize the Terraform plugin SDK async external client: cannot get terraform setup: failed to retrieve aws credentials from aws config: failed to refresh cached credentials, operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 30181f33-9cf9-4f62-9cdb-ebf27c657993, api error AccessDenied: User: arn:aws:sts::1111111111:assumed-role/eks-node-nbl-management/i-1f4de328357931e37 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::2222222222:role/remote-role

My questions:

• None of the guides and examples mention this IAM User
• Why it is using an IAM user (arn:aws:sts::1111111111:assumed-role/eks-node-nbl-management/i-1f4de328357931e37) to perform the AssumeRole? The remote account role has not that user configured in its Trusted Relationships.
• If I am using IRSA, I thought that I am assuming de ServiceAccount Role and then reach the remote account assuming remote-role from there.

I am missconfiguring something? Or what I am doing wrong?

Thank you so much in advance!

[erhancagirici]

hi @fgiannetti, assuming that your IRSA configuration is propagating correctly to the provider pod provider-aws-s3-xxx (you can check that AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE env vars are automatically set to the s3 provider pod), it seems that your trust config references a wrong service account name condition.

 "Condition": {
   "StringLike": {
      "oidc.eks.us-east-1.amazonaws.com/id/SOMETHUMBPRINT:sub": "system:serviceaccount:crossplane-system:provider-aws-iam-*"
   }
}

Note the "system:serviceaccount:crossplane-system:provider-aws-iam-*" in the trust relationship condition. The S3 provider pod will have a service account name generated like provider-aws-s3-xxxx, so it won't match your trust policy condition.

Could you try changing the service account referenced in the trust policy to provider-aws-s3-* to match the service account name? Alternatively, you can also configure provider-aws-* to trust all the provider-aws service accounts for local-role.

Why it is using an IAM user (arn:aws:sts::1111111111:assumed-role/eks-node-nbl-management/i-1f4de328357931e37) to perform the AssumeRole? The remote account role has not that user configured in its Trusted Relationships. This is your k8s cluster node's IAM role.

Since the IRSA service account did not match with the trust config, AWS client falls back to the cluster node's IAM role and tries to assume local-role with cluster node's role by default. That is why you see this error.

Hope this helps!

[fgiannetti]

Hi @erhancagirici thank you so much for your answer! Shame on me for that missconfig. Fixed it and the assumeRole was donde just fine! Confirmed with the AWS console :)

But once the role is assumed, I still cannot create the bucket because I get a 403. The assumed role has the S3FullAccess policy attached but, I get this error:

observe failed: failed to observe the resource: [{0 reading Amazon S3 (Simple Storage) Bucket (fer-test): operation error S3: HeadBucket, https response error StatusCode: 403, RequestID: 0K34WCWYG7PGXGYM, HostID: ZrTVen/9xqs6cfPJ7PzsHdipyVllgQT6zE/dPji57KFj7g35NRrMGfNEc1E44UoTxLcBC/NdBbA=, api error Forbidden: Forbidden  []}]

Any idea? If I assume the role manually using the AWS cli, i can execute a create bucket action with no problems

[fgiannetti]

@erhancagirici Discard my last message! It worked like a charm! I get the 403 because I was referencing to a existent bucket (I am the bumbest person ever)

Thank you so so much for your help and your Hawk Eye 🦅 with the wrong service account name reference

Closing this issue!