[Bug]: Secret never Sync

daniel-maganto commented 7 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Affected Resource(s)

secretsmanager.aws.upbound.io/v1beta1 - Secret

Resource MRs required to reproduce the bug

apiVersion: secretsmanager.aws.upbound.io/v1beta1
kind: Secret
metadata:
  name: test-dmaganto1
spec:
  forProvider:
    name: test-dmaganto1
    region: eu-central-1
    recoveryWindowInDays: 0
  providerConfigRef:
    name: aws-admin-my-account-upbound

Steps to Reproduce

Apply this resource in the cluster

What happened?

It never reach the state Sync: true

Relevant Error Output Snippet

- lastTransitionTime: "2024-02-06T10:12:27Z"
    message: 'observe failed: cannot compute the instance diff: failed to compute
      the customized terraform.InstanceDiff: could not read replica block from config'

Crossplane Version

1.14.5

Provider Version

1.0.0

Kubernetes Version

v1.28.2

Kubernetes Distribution

EKS

Additional Info

No response

turkenf commented 7 months ago

@daniel-maganto, thank you for raising this issue but I could not reproduce the issue with the provided information. Please check your example MR and let us know if there are other fields/parameters on the MR.

apiVersion: secretsmanager.aws.upbound.io/v1beta1
kind: Secret
metadata:
  annotations:
    crossplane.io/external-create-pending: "2024-02-06T11:44:56Z"
    crossplane.io/external-create-succeeded: "2024-02-06T11:44:56Z"
    crossplane.io/external-name: arn:aws:secretsmanager:us-west-1::secret:example-test-112-OnXjOd
    meta.upbound.io/example-id: secretsmanager/v1beta1/secret
    upjet.upbound.io/test: "true"
  creationTimestamp: "2024-02-06T11:44:53Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generation: 2
  labels:
    testing.upbound.io/example-name: secretsmanager
  name: example
  resourceVersion: "1419"
  uid: <...>
spec:
  deletionPolicy: Delete
  forProvider:
    name: example-test-112
    recoveryWindowInDays: 0
    region: us-west-1
    tags:
      crossplane-kind: secret.secretsmanager.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
  initProvider: {}
  managementPolicies:
  - '*'
  providerConfigRef:
    name: default
status:
  atProvider:
    arn: arn:aws:secretsmanager:us-west-1:<..>:secret:example-test-112-OnXjOd
    description: ""
    forceOverwriteReplicaSecret: false
    id: arn:aws:secretsmanager:us-west-1:<..>:secret:example-test-112-OnXjOd
    kmsKeyId: ""
    name: example-test-112
    policy: ""
    recoveryWindowInDays: 0
    tags:
      crossplane-kind: secret.secretsmanager.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
    tagsAll:
      crossplane-kind: secret.secretsmanager.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
  conditions:
  - lastTransitionTime: "2024-02-06T11:45:02Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-02-06T11:44:56Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced
  - lastTransitionTime: "2024-02-06T11:44:57Z"
    reason: Success
    status: "True"
    type: LastAsyncOperation
  - lastTransitionTime: "2024-02-06T11:45:23Z"
    reason: UpToDate
    status: "True"
    type: Test

dmaganto commented 7 months ago

The provider's pods were stuck, after rebooting both, it started working successfully. It is funny because they were installed today so there is no overload at all

daniel-maganto commented 7 months ago

It happens again, here the log traces.

Cannot observe external resource    {"controller": "managed/secretsmanager.aws.upbound.io/v1beta1, kind=secret", "request": {"name":"service-test"}, "uid": "3e8c81da-0c2c-4a4b-b0b3-682ca0df0a4a", "version": "183689711", "external-name": "arn:aws:secretsmanager:eu-central-1:XXXXXXXXX:secret:service-test-dK6yoH", "error": "cannot compute the instance diff: failed to compute the customized terraform.InstanceDiff: could not read replica block from config", "errorVerbose": "could not read replica block from config
github.com/upbound/provider-aws/config/secretsmanager.Configure.func1.2
    github.com/upbound/provider-aws/config/secretsmanager/config.go:44
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).getResourceDataDiff
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_tfpluginsdk.go:427
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_tfpluginsdk.go:498
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_async_tfpluginsdk.go:126
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/reconciler/managed/reconciler.go:903
github.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/ratelimiter/reconciler.go:54
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227
runtime.goexit
    runtime/asm_amd64.s:1650
failed to compute the customized terraform.InstanceDiff
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).getResourceDataDiff
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_tfpluginsdk.go:429
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_tfpluginsdk.go:498
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_async_tfpluginsdk.go:126
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/reconciler/managed/reconciler.go:903
github.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/ratelimiter/reconciler.go:54
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227
runtime.goexit
    runtime/asm_amd64.s:1650
cannot compute the instance diff
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_tfpluginsdk.go:500
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_async_tfpluginsdk.go:126
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/reconciler/managed/reconciler.go:903
github.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/ratelimiter/reconciler.go:54
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227
runtime.goexit
    runtime/asm_amd64.s:1650"}

turkenf commented 7 months ago

I tried a few times and I can successfully create and delete the resource, but I can confirm that diff is detected in the logs:

2024-02-07T18:00:30+03:00   DEBUG   provider-aws    Diff detected   {"uid": "de62d41e-a87f-4373-9c4d-95c11c383a57", "name": "example", "gvk": "secretsmanager.aws.upbound.io/v1beta1, Kind=Secret", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"arn\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"force_overwrite_replica_secret\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"example-test1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"name_prefix\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"recovery_window_in_days\":*terraform.ResourceAttrDiff{Old:\"\", New:\"0\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"replica.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.%\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"3\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.crossplane-kind\":*terraform.ResourceAttrDiff{Old:\"\", New:\"secret.secretsmanager.aws.upbound.io\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.crossplane-name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"example\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.crossplane-providerconfig\":*terraform.ResourceAttrDiff{Old:\"\", New:\"default\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags_all.%\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"3\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags_all.crossplane-kind\":*terraform.ResourceAttrDiff{Old:\"\", New:\"secret.secretsmanager.aws.upbound.io\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags_all.crossplane-name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"example\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags_all.crossplane-providerconfig\":*terraform.ResourceAttrDiff{Old:\"\", New:\"default\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}

@erhancagirici, do you have any idea about this?

erhancagirici commented 7 months ago

Hi, This looks like an issue with the custom diff function for handling replica field changes, affecting the configs that do not have replica set. I will send a fix 👍 Thanks for reporting @daniel-maganto

mbbush commented 7 months ago

Specific impact of this bug, as far as I can tell:

New secrets can be created
Secrets can be deleted.
Secrets which already exist in AWS can be observed/imported, as long as there is no diff whatsoever between the spec and the observed values.
Updating existing secrets trigger the bug, which causes the update to fail. This includes things like setting or changing the crossplane-managed tags

crossplane-contrib / provider-upjet-aws