search

aws-ia / terraform-aws-eks-blueprints

Configure and deploy complete EKS clusters.
https://aws-ia.github.io/terraform-aws-eks-blueprints/
Apache License 2.0
2.67k stars 1.42k forks source link

[Bug]: Crossplane Terrajet/AWS provider IRSA error: Access Denied (403) #741

Closed ka1h closed 2 years ago

ka1h commented 2 years ago

Welcome to Amazon EKS Blueprints!

Amazon EKS Blueprints Release version

4.3.0

What is your environment, configuration and the example used?

I'm using the crossplane example with EKS 1.22, altough any case of enable_crossplane = true with either enabling the AWS or Terrajet provider is relevant for this issue:

  # Crossplane
  enable_crossplane = true

  # Creates ProviderConfig -> jet-aws-provider
  crossplane_jet_aws_provider = {
    enable               = true
    provider_aws_version = "v0.4.1"
    # NOTE: Crossplane requires Admin like permissions to create and update resources similar to Terraform deploy role.
    # This example config uses AmazonS3FullAccess for demo purpose only, but you should select a policy with the minimum permissions required to provision your resources.
    additional_irsa_policies = ["arn:aws:iam::aws:policy/AmazonS3FullAccess"]
  }

What did you do and What did you see instead?

Deploying a S3 Bucket via kubectl apply -f jet-aws-provider-s3.yaml results in the following error (logs from the terrajet-pod):

2022-07-06T20:04:26.684Z    DEBUG   provider-jet-aws    Reconciling {"controller": "managed/s3.aws.jet.crossplane.io/v1alpha2, kind=bucket", "request": "/xplane-jet-aws-s3-bucket"}
2022-07-06T20:04:26.704Z    DEBUG   provider-jet-aws    Cannot connect to provider  {"controller": "managed/s3.aws.jet.crossplane.io/v1alpha2, kind=bucket", "request": "/xplane-jet-aws-s3-bucket", "uid": "a420ca1e-1072-465d-aecf-73418c71377f", "version": "25735", "external-name": "euris-crossplane-test-bucket", "error": "cannot get terraform setup: failed to use pod service account: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: db8fe84c-baf1-46b6-a6a8-48b4ce61e4e9", "errorVerbose": "AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: db8fe84c-baf1-46b6-a6a8-48b4ce61e4e9\nfailed to use pod service account\ngithub.com/crossplane-contrib/provider-jet-aws/internal/clients.TerraformSetupBuilder.func1\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/internal/clients/aws.go:65\ngithub.com/crossplane/terrajet/pkg/controller.(*Connector).Connect\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/vendor/github.com/crossplane/terrajet/pkg/controller/external.go:91\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/vendor/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed/reconciler.go:668\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.13/x64/src/runtime/asm_amd64.s:1371\ncannot get terraform setup\ngithub.com/crossplane/terrajet/pkg/controller.(*Connector).Connect\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/vendor/github.com/crossplane/terrajet/pkg/controller/external.go:93\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/vendor/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed/reconciler.go:668\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/provider-jet-aws/provider-jet-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.13/x64/src/runtime/asm_amd64.s:1371"}
2022-07-06T20:04:26.704Z    DEBUG   controller-runtime.manager.events   Warning {"object": {"kind":"Bucket","name":"xplane-jet-aws-s3-bucket","uid":"a420ca1e-1072-465d-aecf-73418c71377f","apiVersion":"s3.aws.jet.crossplane.io/v1alpha2","resourceVersion":"25735"}, "reason": "CannotConnectToProvider", "message": "cannot get terraform setup: failed to use pod service account: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: db8fe84c-baf1-46b6-a6a8-48b4ce61e4e9"}

So, basically it seems to be an IRSA issue: (...) AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403 (...). The following trust relationship is getting deployed to the IAM role eks-blueprints-dev-jet-aws-provider-irsa:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::XXXXXXXXXX:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXX"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXX:aud": "sts.amazonaws.com",
          "oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXX:sub": "system:serviceaccount:crossplane-system:jet-aws-provider-*"
        }
      }
    }
  ]
}

The Terrajet Provider generates a service account with a random string attached to it, e.g. jet-aws-provider-b42c59584ad8. This obviously makes it necessary to use a wildcard in the trust relationship. StringEquals doesn't work with wildcards though, StringLike would work. I also have to remove the sts.amazonaws.com-line to make it work. (edit: this is wrong and actually works)

The following edit works for me:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::XXXXXXXXXX:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXX"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringLike": {
          "oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXX:sub": "system:serviceaccount:crossplane-system:jet-aws-provider-*"
        }
      }
    }
  ]
}

The same thing applies to the crossplane AWS provider.

Additional Information

Related issue: https://github.com/crossplane-contrib/provider-aws/issues/1172
vara-bonthu commented 2 years ago

Thanks for spotting the issue @ka1h 👍🏼

It does look like its been changed recently from StringLike to StringEquals by one of the commit .

I will raise a PR with hotfix.

Good to see the Crossplane usage here. Did you also checkout this Crossplane Blueprints repo for compositions?

ka1h commented 2 years ago

Hey @vara-bonthu, thank you for handling this issue & linking crossplane-aws-blueprints -- actually didn't know it existed. Great work so far! :)