IAM resources fail in isolated AWS partitions (e.g. AWS GovCloud) unless AWS_DEFAULT_REGION set

[sidpalas]

What happened?

I attempted to create IAM resources in us-gov but the resources were never created and resulted in the following STS error:

managed/iam.aws.upbound.io/v1beta1, kind=role  cannot run refresh: refresh failed: configuring Terraform AWS Provider: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 638c6984-99a6-45d6-b2d3-1edc0ea00017, api error InvalidClientTokenId: The security token included in the request is invalid.:

I believe this has to do with the fact that IAM resources do not have a region field in their forProvider configuration and it ends up falling back to a hardcoded default of us-east-1

I found this issue: https://github.com/crossplane-contrib/provider-aws/issues/596 and attempted to use the custom endpoint configuration for the ProviderConfig to specify a specific endpoint signing region with the following configuration but the STS error remained.

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: creds
  endpoint:
    partitionId: aws-us-gov
    signingRegion: us-gov-west-1
    source: Custom
    url:
      dynamic:
        host: 'iam.us-gov.amazonaws.com'
        protocol: https
      type: Dynamic

Finally, I used a ControllerConfig to set AWS_DEFAULT_REGION in the AWS provider deployment at which point I was able to successfully provision the resources.

I would have expected the custom endpoint to be sufficient, but I think the STS error was being thrown before the custom endpoint could be used.

How can we reproduce it?

1. Install the AWS provider

   apiVersion: pkg.crossplane.io/v1
   kind: Provider
   metadata:
   name: provider-aws-iam
   spec:
   package: xpkg.upbound.io/upbound/provider-aws-iam:v0.36.0

2. Create a ProviderConfig with AWS credentials for a user with permissions in non aws partition (e.g. aws-gov)

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-secret
      key: creds

3. Attempt to create an IAM role

apiVersion: iam.aws.upbound.io/v1beta1
kind: Role
metadata:
  name: crossplane-cluster-role
spec:
  forProvider:
    assumeRolePolicy: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "Service": "eks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
          }
        ]
      }

What environment did it happen in?

• Crossplane Version: v1.12.2
• Provider Version: v0.36.0 (using static credentials in k8s secret)
• Kubernetes Version: v1.27.3
• Kubernetes Distribution: KinD (Kubernetes in Docker)

[jeanduplessis]

Possible area of investigation:

• the region falls back to us-east-1 if not set: https://github.com/upbound/provider-aws/blob/b09b073c081a48c9b4411ebfbd283387f6c722b9/internal/clients/aws.go#L227
• here is where the provider pulls the endpoint config https://github.com/upbound/provider-aws/blob/b09b073c081a48c9b4411ebfbd283387f6c722b9/internal/clients/provider_config.go#L186

[yebolenko]

any update on this issue? the same is also applicable to china (cn-north-1)

[torg28]

Any updates or workaround for this issue? @sidpalas your workaround does not seem to work for me.

[torg28]

I did find a workaround. Essentially I did what was mentioned above, but instead of putting the endpoint config in the default providerconfig I created a new one called "gov" and used that instead of default for any resources that needed it.

[github-actions[bot]]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions[bot]]

This issue is being closed since there has been no activity for 14 days since marking it as stale. If you still need help, feel free to comment or reopen the issue!