LocalStack w/Crossplane Upbound AWS Provider

[blakeromano]

What happened?

I am working on building a testing framework for compositions with Localstack, Kind and Kuttl. I am running into 403 issues and 400 errors when trying to provision resources when pointing to the provider to a localstack instance. This is based off of a sample implementation by @haarchri here https://github.com/haarchri/provider-aws-issue-1017/

The error occuring for S3:

cannot run refresh: refresh failed: reading Amazon S3 (Simple Storage) Bucket (foo-bucket): Forbidden: Forbidden
           status code: 403, request id: Q1ZRGBMEMFW555QZ, host id: FzkA3j5lAbZ+sjGV0mLFv9gTP7cHO+bTX0m0Le0XUn9PCvAaagW4fGONezW9sy317BzfiB5gFJs=:

The error occuring for dynamodb tables:

cannot run refresh: refresh failed: reading Amazon DynamoDB Table (my-table): UnrecognizedClientException: The security token included in the request is invalid.
           status code: 400, request id: HLD8FQ64BIH3B9P8TEMINO094JVV4KQNSO5AEMVJF66Q9ASUAAJG:

How can we reproduce it?

Have a Kuttl test like:

kuttl-test.yaml ```yaml apiVersion: kuttl.dev/v1beta1 kind: TestSuite commands: - script: ./setup.sh startKIND: true kindContext: kind skipClusterDelete: false timeout: 120 testDirs: - e2e/ ``` setup.sh ```bash #!/bin/bash -e helm repo add crossplane-stable https://charts.crossplane.io/stable helm install crossplane crossplane-stable/crossplane --version 1.13.2 --namespace crossplane-system --wait --create-namespace kubectl apply -f ./core-providers.yaml kubectl wait --for condition=healthy --timeout=150s providers.pkg.crossplane.io/upbound-provider-family-aws kubectl wait --for condition=healthy --timeout=150s providers.pkg.crossplane.io/provider-kubernetes kubectl apply -f ./provider-configs.yaml kubectl apply -f ./aws-services.yaml AWS_PROFILE=default && echo -e "[default]\naws_access_key_id = $(aws configure get aws_access_key_id --profile $AWS_PROFILE)\naws_secret_access_key = $(aws configure get aws_secret_access_key --profile $AWS_PROFILE)" > creds.conf kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./creds.conf rm creds.conf localstack start -d kubectl wait --for condition=healthy --timeout=150s providers.pkg.crossplane.io/provider-aws-s3 kubectl create namespace sandbox-dev kubectl create namespace sandbox-qa kubectl create namespace sandbox-prod ``` core-providers.yaml ```yaml --- apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: upbound-provider-family-aws spec: package: xpkg.upbound.io/upbound/provider-family-aws:v0.41.0 controllerConfigRef: name: aws-config --- apiVersion: pkg.crossplane.io/v1alpha1 kind: ControllerConfig metadata: name: aws-config spec: args: - --debug podSecurityContext: fsGroup: 2000 ``` aws-services.yaml ```yaml --- apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: provider-aws-s3 spec: package: xpkg.upbound.io/upbound/provider-aws-s3:v0.41.0 controllerConfigRef: name: aws-config --- apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: provider-aws-dynamodb namespace: crossplane-system spec: package: xpkg.upbound.io/upbound/provider-aws-dynamodb:v0.41.0 controllerConfigRef: name: aws-config ``` provider-configs.yaml ```yaml apiVersion: aws.upbound.io/v1beta1 kind: ProviderConfig metadata: name: aws-provider namespace: crossplane-system spec: credentials: source: Secret secretRef: namespace: crossplane-system name: aws-creds key: creds s3_use_path_style: false skip_credentials_validation: true skip_metadata_api_check: true skip_requesting_account_id: true skip_region_validation: true endpoint: hostnameImmutable: true url: static: http://127.0.0.1:4566 type: Static ``` e2e/s3/00-install.yaml ```yaml apiVersion: s3.aws.upbound.io/v1beta1 kind: Bucket metadata: name: foo-bucket spec: deletionPolicy: Delete providerConfigRef: name: aws-provider ``` e2e/s3/00-assert.yaml ```yaml apiVersion: s3.aws.upbound.io/v1beta1 kind: Bucket metadata: name: foo-bucket spec: deletionPolicy: Delete providerConfigRef: name: aws-provider status: atProvider: id: foo-bucket ``` e2e/dynamodb/00-install.yaml ```yaml apiVersion: dynamodb.aws.upbound.io/v1beta1 kind: Table metadata: name: my-table spec: deletionPolicy: Delete forProvider: attribute: - name: id type: S - name: sk type: S - name: projectType type: S - name: created type: S billingMode: PAY_PER_REQUEST globalSecondaryIndex: - hashKey: projectType name: projectType-index nonKeyAttributes: - id - created projectionType: ALL rangeKey: sk hashKey: id rangeKey: sk region: us-east-2 providerConfigRef: name: aws-provider ``` e2e/dynamodb/00-assert.yaml ```yaml apiVersion: dynamodb.aws.upbound.io/v1beta1 kind: Table metadata: name: my-table spec: deletionPolicy: Delete forProvider: attribute: - name: id type: S - name: sk type: S - name: projectType type: S - name: created type: S billingMode: PAY_PER_REQUEST globalSecondaryIndex: - hashKey: projectType name: projectType-index nonKeyAttributes: - id - created projectionType: ALL rangeKey: sk hashKey: id rangeKey: sk region: us-east-2 providerConfigRef: name: aws-provider status: atProvider: id: my-table ```

What environment did it happen in?

• Crossplane Version: 1.13.2
• Provider Version: 0.41.0
• Kubernetes Version:

  kubectl version --short
  Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
  Client Version: v1.27.4
  Kustomize Version: v5.0.1
  Server Version: v1.27.3

• Kubernetes Distribution: Kind

[whummer]

Hi @blakeromano, looking at the error message, it seems that the AWS provider is reaching out to real AWS, rather than LocalStack. Can you please share some details about your exact setup, ideally a reproducible example that we can look into in more detail..?

Here's a pointer to the docs for the Crossplane<>LocalStack integration - are you using a similar ProviderConfig to point to the local endpoints for testing..? https://docs.localstack.cloud/user-guide/integrations/crossplane

[github-actions[bot]]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.