v0.46.0 broke localstack config

mbbush commented 8 months ago

What happened?

Using provider version v0.45.0 I can successfully test my compositions and managed resources against localstack, which locally simulates the aws api. Starting with provider version v0.46.0, I cannot. The error is also present in v0.47.1, and when I run against the current tip of main. When I revert 7a61c51f89e7a38c640250b68583765f030b6bda it works again.

All resources which use the no-fork reconciler get a Synced: False condition

    - lastTransitionTime: "2024-01-04T07:21:43Z"
      message: "observe failed: failed to observe the resource: [{0 reading IAM Role
        (sample-role): InvalidClientTokenId: The security token included in the request
        is invalid.\n\tstatus code: 403, request id: 201ef32d-6195-48ff-812a-317c6ec29b76
        \ []}]"
      reason: ReconcileError
      status: "False"
      type: Synced

The resources which still use the CLI based reconciler work fine.

Provider logs:

2024-01-04T07:23:43Z    DEBUG   provider-aws    Reconciling {"controller": "managed/iam.aws.upbound.io/v1beta1, kind=role", "request": {"name":"sample-role"}}
2024-01-04T07:23:43Z    DEBUG   provider-aws    Connecting to the service provider  {"uid": "afa9d5be-cd90-4641-8e0e-ef8bfac16423", "name": "sample-role", "gvk": "iam.aws.upbound.io/v1beta1, Kind=Role"}
2024-01-04T07:23:43Z    DEBUG   provider-aws    Reconciling {"controller": "managed/iam.aws.upbound.io/v1beta1, kind=policy", "request": {"name":"sample-user-policy"}}
2024/01/04 07:23:43 [WARN] AWS account ID not found for provider. See https://www.terraform.io/docs/providers/aws/index.html#skip_requesting_account_id for implications.
2024-01-04T07:23:43Z    DEBUG   provider-aws    Observing the external resource {"uid": "afa9d5be-cd90-4641-8e0e-ef8bfac16423", "name": "sample-role", "gvk": "iam.aws.upbound.io/v1beta1, Kind=Role"}
2024/01/04 07:23:43 [DEBUG] Waiting for state to become: [success]
2024-01-04T07:23:43Z    DEBUG   provider-aws    Connecting to the service provider  {"uid": "f330b9b3-09c5-4b5e-afab-a37af7c6fc1f", "name": "sample-user-policy", "gvk": "iam.aws.upbound.io/v1beta1, Kind=Policy"}
2024/01/04 07:23:43 [WARN] AWS account ID not found for provider. See https://www.terraform.io/docs/providers/aws/index.html#skip_requesting_account_id for implications.
2024-01-04T07:23:43Z    DEBUG   provider-aws    Observing the external resource {"uid": "f330b9b3-09c5-4b5e-afab-a37af7c6fc1f", "name": "sample-user-policy", "gvk": "iam.aws.upbound.io/v1beta1, Kind=Policy"}
2024/01/04 07:23:43 [DEBUG] Waiting for state to become: [success]
2024-01-04T07:23:44Z    DEBUG   provider-aws    Cannot observe external resource    {"controller": "managed/iam.aws.upbound.io/v1beta1, kind=role", "request": {"name":"sample-role"}, "uid": "afa9d5be-cd90-4641-8e0e-ef8bfac16423", "version": "1290", "external-name": "sample-role", "error": "failed to observe the resource: [{0 reading IAM Role (sample-role): InvalidClientTokenId: The security token included in the request is invalid.\n\tstatus code: 403, request id: 8bc110b9-c62f-4e25-919e-d99f2d474eec  []}]", "errorVerbose": "failed to observe the resource: [{0 reading IAM Role (sample-role): InvalidClientTokenId: The security token included in the request is invalid.\n\tstatus code: 403, request id: 8bc110b9-c62f-4e25-919e-d99f2d474eec  []}]\ngithub.com/crossplane/upjet/pkg/controller.(*noForkExternal).Observe\n\tgithub.com/crossplane/upjet@v1.1.0-rc.0.0.20231227120826-4cb45f9104ac/pkg/controller/external_nofork.go:475\ngithub.com/crossplane/upjet/pkg/controller.(*noForkAsyncExternal).Observe\n\tgithub.com/crossplane/upjet@v1.1.0-rc.0.0.20231227120826-4cb45f9104ac/pkg/controller/external_async_nofork.go:117\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/reconciler/managed/reconciler.go:903\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1650"}
2024-01-04T07:23:44Z    DEBUG   events  failed to observe the resource: [{0 reading IAM Role (sample-role): InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 8bc110b9-c62f-4e25-919e-d99f2d474eec  []}]    {"type": "Warning", "object": {"kind":"Role","name":"sample-role","uid":"afa9d5be-cd90-4641-8e0e-ef8bfac16423","apiVersion":"iam.aws.upbound.io/v1beta1","resourceVersion":"1290"}, "reason": "CannotObserveExternalResource"}
2024-01-04T07:23:44Z    DEBUG   provider-aws    Cannot observe external resource    {"controller": "managed/iam.aws.upbound.io/v1beta1, kind=policy", "request": {"name":"sample-user-policy"}, "uid": "f330b9b3-09c5-4b5e-afab-a37af7c6fc1f", "version": "1293", "external-name": "sample-user-policy", "error": "failed to observe the resource: [{0 reading IAM Policy (arn:aws:iam::000000000:policy/sample-user-policy): InvalidClientTokenId: The security token included in the request is invalid.\n\tstatus code: 403, request id: 0fe96847-5a23-4bf0-830e-2dee863945a2  []}]", "errorVerbose": "failed to observe the resource: [{0 reading IAM Policy (arn:aws:iam::000000000:policy/sample-user-policy): InvalidClientTokenId: The security token included in the request is invalid.\n\tstatus code: 403, request id: 0fe96847-5a23-4bf0-830e-2dee863945a2  []}]\ngithub.com/crossplane/upjet/pkg/controller.(*noForkExternal).Observe\n\tgithub.com/crossplane/upjet@v1.1.0-rc.0.0.20231227120826-4cb45f9104ac/pkg/controller/external_nofork.go:475\ngithub.com/crossplane/upjet/pkg/controller.(*noForkAsyncExternal).Observe\n\tgithub.com/crossplane/upjet@v1.1.0-rc.0.0.20231227120826-4cb45f9104ac/pkg/controller/external_async_nofork.go:117\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/reconciler/managed/reconciler.go:903\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1650"}
2024-01-04T07:23:44Z    DEBUG   events  failed to observe the resource: [{0 reading IAM Policy (arn:aws:iam::000000000:policy/sample-user-policy): InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 0fe96847-5a23-4bf0-830e-2dee863945a2  []}]    {"type": "Warning", "object": {"kind":"Policy","name":"sample-user-policy","uid":"f330b9b3-09c5-4b5e-afab-a37af7c6fc1f","apiVersion":"iam.aws.upbound.io/v1beta1","resourceVersion":"1293"}, "reason": "CannotObserveExternalResource"}

How can we reproduce it?

Install localstack

helm repo add localstack-repo https://helm.localstack.cloud
helm upgrade --install localstack localstack-repo/localstack --namespace localstack --create-namespace

Set up the following providerconfig:

---
apiVersion: v1
kind: Secret
metadata:
  name: aws-creds
  namespace: upbound-system
type: Opaque
stringData:
  creds: |
    [default]
    aws_access_key_id = 000000000000
    aws_secret_access_key = nope
---
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: upbound-system
      name: aws-creds
      key: creds
  s3_use_path_style: true
  skip_credentials_validation: true
  skip_metadata_api_check: true
  skip_requesting_account_id: true
  endpoint:
    source: Custom
    hostnameImmutable: true
    services:
      - iam
      - sts 
    url:
      type: Static
      static: "http://localstack.localstack.svc.cluster.local:4566"

Run uptest on examples/iam/role.yaml (or anything else)

What environment did it happen in?

Crossplane Version:
Provider Version:
Kubernetes Version:
Kubernetes Distribution: Kind

erhancagirici commented 8 months ago

@mbbush thanks for reporting this, I've sent a PR for this, which I've locally validated the endpoint configuration is properly propagated. If possible, would you be willing to test with https://github.com/upbound/provider-aws/pull/1066 check if the fix works for you too?

mbbush commented 8 months ago

Thanks for the quick fix! Yes, when I build and deploy #1066 it resolves the issue for me.

erhancagirici commented 8 months ago

@mbbush many thanks for testing it, and glad that it is working 🎉 Could you also check that #1066 still works for CLI-based resources too in your localstack setup?

mbbush commented 8 months ago

CLI-based resources work too. One of the resources I'm currently working with is UserPoolClient.cognitoidp, and that works fine with the build from #1066

crossplane-contrib / provider-upjet-aws