Status shows no key could decrypt secret for successful created secret

[alita1991]

Which component: sealed-secret-controller: v0.20.2

Describe the bug I reproduced an old issue described in #853, where the status message for a sealed secret was "no key could decrypt secret", but the secret was correctly unsealed, and the logs confirmed this as well.

This issue was discovered via ArgoCD, where the sealed secrets were marked as red. The temporary workaround was to restart the sealed-secret-controller pod, and after this, the status was updated correctly.

Steps to reproduce Could not figure out a way to reproduce the issue (the environment where was discovered is long-lived and we only promote new helm chart versions)

Expected behavior When the creation was successful the status should show SealedSecret unsealed successfully like the logs and the events.

K8s version Server Version: v1.25.16-eks-b9c9ed7

[Gnarfoz]

This appears to also be the same issue as #739 and #1354, #1355 , etc. :-( Seems like the issue is hard to eliminate.

We're facing this even with 0.26.2, and on 0.25.0. We were on 0.18.1 before that, which didn't have this problem.

[WillyRL]

I also found this problem on my sealed-secret installation while using ArgoCD, did you found the solution ? @Gnarfoz

[Gnarfoz]

Hey @WillyRL, there seems to be no permanent solution. We work around it by restarting the Sealed Secrets controller pod from time to time. During startup, it correctly marks all successfully unsealed secrets as healthy, as @alita1991 also wrote.

It seems like a trivial "out of order" thing...

[WillyRL]

Hey @WillyRL, there seems to be no permanent solution. We work around it by restarting the Sealed Secrets controller pod from time to time. During startup, it correctly marks all successfully unsealed secrets as healthy, as @alita1991 also wrote.

It seems like a trivial "out of order" thing...

It still not working on my setup, may I know your syncPolicy ? My configuration:

      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ApplyOutOfSyncOnly=true
          - ServerSideApply=true

[Gnarfoz]

I'm not sure what you're asking. This is not a problem with your ArgoCD configuration, it's a bug in Sealed Secrets.

[alvneiayu]

hi all

As @Gnarfoz , we have included several changes to improve this and solve the problem.

We are using Sealed Secrets in our clusters but we are not reproducing the problem. Could you provide us logs reproducing the issue to continue investigating the reason, please?

Thanks a lot and sorry for the inconveniences.

[Gnarfoz]

I'd be happy to provide logs, @alvneiayu. Before I start: is there some way to increase the log level to "DEBUG" or something like that

[Gnarfoz]

I looked at the relevant code and the only options seem to be INFO and ERROR, so here's some logs:

time=2024-06-07T14:23:25.585Z level=INFO msg=Updating key=the-namespace/the-secret-auth
time=2024-06-07T14:23:25.962Z level=INFO msg="Event(v1.ObjectReference{Kind:\"SealedSecret\", Namespace:\"the-namespace\", Name:\"the-secret-auth\", UID:\"684a93fb-2d5d-4af7-a668-a45716092afb\", APIVersion:\"bitnami.com/v1alpha1\", ResourceVersion:\"1594762022\", FieldPath:\"\"}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully"
time=2024-06-07T14:23:25.971Z level=INFO msg=Updating key=the-namespace/another-secret
time=2024-06-07T14:23:25.973Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=the-namespace/the-secret-auth
time=2024-06-07T14:23:26.331Z level=INFO msg="Event(v1.ObjectReference{Kind:\"SealedSecret\", Namespace:\"the-namespace\", Name:\"another-secret\", UID:\"654c0b6a-74e6-4bb4-b420-2bd9e4ff76c0\", APIVersion:\"bitnami.com/v1alpha1\", ResourceVersion:\"1594762024\", FieldPath:\"\"}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully"
time=2024-06-07T14:23:26.342Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=the-namespace/another-secret
time=2024-06-07T14:23:27.530Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=the-namespace/the-secret-auth
time=2024-06-07T14:23:28.554Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=the-namespace/another-secret
time=2024-06-07T14:26:05.736Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=the-namespace/the-secret-auth
time=2024-06-07T14:26:49.694Z level=INFO msg=Updating key=the-namespace/the-secret-auth
time=2024-06-07T14:26:49.915Z level=INFO msg="Event(v1.ObjectReference{Kind:\"SealedSecret\", Namespace:\"the-namespace\", Name:\"the-secret-auth\", UID:\"1ed36c6b-de14-4540-bb88-b20b4f3dca3c\", APIVersion:\"bitnami.com/v1alpha1\", ResourceVersion:\"1594766393\", FieldPath:\"\"}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully"
time=2024-06-07T14:26:49.921Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=the-namespace/the-secret-auth
time=2024-06-07T14:26:50.642Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=the-namespace/the-secret-auth
time=2024-06-07T14:28:35.724Z level=INFO msg=Updating key=the-namespace/the-secret-auth
time=2024-06-07T14:28:35.989Z level=INFO msg="Event(v1.ObjectReference{Kind:\"SealedSecret\", Namespace:\"the-namespace\", Name:\"the-secret-auth\", UID:\"1ed36c6b-de14-4540-bb88-b20b4f3dca3c\", APIVersion:\"bitnami.com/v1alpha1\", ResourceVersion:\"1594766436\", FieldPath:\"\"}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully"
time=2024-06-07T14:28:35.999Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=the-namespace/the-secret-auth
time=2024-06-07T14:28:36.681Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=the-namespace/the-secret-auth

This is from creating a fresh sealed secret that had never before existed. ArgoCD somehow gets confused by this:

It seems to think that the sealed secret failed to unseal, but it actually unsealed successfully first. Somehow, a unseal failure gets reported after having already unsealed successfully.

[alita1991]

Hi,

I was able to reproduce the issue when I did an upgrade from 2.15.3 to 2.15.4

Controller logs

time=2024-06-10T12:14:50.482Z level=INFO msg="Starting sealed-secrets controller" version=v0.26.3
time=2024-06-10T12:14:50.483Z level=INFO msg="Searching for existing private keys"
time=2024-06-10T12:14:50.588Z level=INFO msg="registered private key" secretname=sealed-secrets-key7f4vz
time=2024-06-10T12:14:50.589Z level=INFO msg="registered private key" secretname=sealed-secrets-keyvpgr2
time=2024-06-10T12:14:50.590Z level=INFO msg="registered private key" secretname=sealed-secrets-keyg7l9x
time=2024-06-10T12:14:50.590Z level=INFO msg="registered private key" secretname=sealed-secrets-key8h5rd
time=2024-06-10T12:14:50.590Z level=INFO msg="Starting informer" namespace=integration
time=2024-06-10T12:14:50.591Z level=INFO msg="HTTP server serving" addr=:8080
time=2024-06-10T12:14:50.591Z level=INFO msg="HTTP metrics server serving" addr=:8081
time=2024-06-10T12:14:56.300Z level=INFO msg=Updating key=integration/minio-secret
time=2024-06-10T12:14:56.892Z level=INFO msg="Event(v1.ObjectReference{Kind:\"SealedSecret\", Namespace:\"integration\", Name:\"minio-secret\", UID:\"93f0990f-d48d-4375-a3c0-35348933c4dc\", APIVersion:\"bitnami.com/v1alpha1\", ResourceVersion:\"81903923\", FieldPath:\"\"}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully"
time=2024-06-10T12:14:56.982Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=integration/minio-secret

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  annotations:
    sealedsecrets.bitnami.com/namespace-wide: "true"
  creationTimestamp: "2024-03-23T16:03:06Z"
  generation: 1
  name: minio-secret
  namespace: integration
  resourceVersion: "81903923"
  uid: 93f0990f-d48d-4375-a3c0-35348933c4dc
spec:
  encryptedData:
    root-password: <PASSWORD>
  template:
    data:
      root-user: <USERNAME>
    metadata:
      name: minio-secret
      type: Opaque
status:
  conditions:
  - lastTransitionTime: "2024-06-10T07:02:40Z"
    lastUpdateTime: "2024-06-10T07:02:40Z"
    message: no key could decrypt secret (root-password)
    status: "False"
    type: Synced
  observedGeneration: 1

Solution A simple pod restart fixed the issue

[Gnarfoz]

Indeed, on restart of the controller pod, all sealed secrets are traversed and their statutes are updated correctly, and ArgoCD is then happy. That's not a long-term solution, though. You'd have to restart the pod every time you roll out a new sealed secret.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[Gnarfoz]

Hey @alvneiayu, are the provided logs useful to you? This issue persists with the latest version (0.27.0).

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[Gnarfoz]

Any chance you can take a look at this, @alvneiayu? The issue persists. I've made a short reproduction:

$ echo -n "test" | oc create secret generic topsecret --dry-run=client --from-file=/dev/stdin -o yaml > topsecret.yaml
$ kubeseal -f topsecret.yaml -w topsecret.sealed.yaml
$ oc create -f topsecret.sealed.yaml
sealedsecret.bitnami.com/topsecret created
$ oc describe sealedsecret topsecret
Name:         topsecret
Namespace:    nptest1
Labels:       <none>
Annotations:  <none>
API Version:  bitnami.com/v1alpha1
Kind:         SealedSecret
Metadata:
  Creation Timestamp:  2024-07-12T10:22:00Z
  Generation:          1
  Managed Fields:
    API Version:  bitnami.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:encryptedData:
          .:
          f:stdin:
        f:template:
          .:
          f:metadata:
            .:
            f:creationTimestamp:
            f:name:
            f:namespace:
    Manager:      kubectl-create
    Operation:    Update
    Time:         2024-07-12T10:22:00Z
    API Version:  bitnami.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:observedGeneration:
    Manager:         controller
    Operation:       Update
    Subresource:     status
    Time:            2024-07-12T10:22:01Z
  Resource Version:  1677057343
  UID:               1fddd4b3-e000-436c-83e0-e9466515b902
Spec:
  Encrypted Data:
    Stdin:  AgBzHRJQSI84X42HJEdHjIphASxTNP3cwVvqFRfDKKjWHKgjnwx7oLqkwaW8zFTIPeDJ6Qo5ZOK6wjjtAn4ZRlztx7W3vjMLbw/z09rs5xQr60HipXNU8tFgRsvB9u1+aLk+nPHwm8ttZ8/QizB71mMzxAqmyY5hHZkQd7Y143pdBDsRfvunPw+qAuxTw2RX6DPa8DNNOIBEYw5IsATdnQcSqj6s9kndNZ964RF8WX04u40+e4LnltytQ3erGzWyJMkk5eHJP00umIKpQz6efHZ1l2K7m+lKEzNo6qJcRNgRR6xRu+rNnEbYoJ8w3Ccc7J1jew7Aw2HIzFHrKS5CizEDzQbDfE1RJJtHf0JrxCybQN2ru7HEtZSwz12qWj+/MJKwKQQfc04hLPaWkWsJ/q2mZS2qnv1wVtxGpRDS6DGPKMZbmLP0S2qRgXm1PwXLCWDiTvYO5mfi6Bc+aNre1WOMSjHVl66uf/VKgwOv5lLCuKpY+np+Qi5NI3bwA1fBCTZtRBKw7vyNzc1RcdTB1Dv9NMaD8aOJGA+6BfAmbATVQrkFMXiNHrvB6RPKp1OKuxMa+o986x74oxZbCKs3LiG1j0HBFVdbi2pyIvKgDY+WTr9brEgnJ3f3Pyd42plXdeuwsqD1Nd+ubhDeoODjEuJxsh2EtL4l7lqO02THeDBFA6KYC2gpDZkTYeMZDSIHY15TuWIt
  Template:
    Metadata:
      Creation Timestamp:  <nil>
      Name:                topsecret
      Namespace:           nptest1
Status:
  Conditions:
    Last Transition Time:  2024-07-12T10:22:01Z
    Last Update Time:      2024-07-12T10:22:01Z
    Message:               no key could decrypt secret (stdin)
    Status:                False
    Type:                  Synced
  Observed Generation:     1
Events:
  Type     Reason           Age                From            Message
  ----     ------           ----               ----            -------
  Normal   Unsealed         90s                sealed-secrets  SealedSecret unsealed successfully
  Warning  ErrUnsealFailed  89s (x6 over 90s)  sealed-secrets  Failed to unseal: no key could decrypt secret (stdin)

$ oc get secret topsecret
NAME        TYPE     DATA   AGE
topsecret   Opaque   1      6m30s

Note how under events, it first unseals correctly and then claims it was unable to unseal it, leaving it in that incorrect state. The "secret" object was indeed created.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[Gnarfoz]

This Issue has been manually marked as "not stale" because it has not been resolved.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[Gnarfoz]

Nothing has changed, the issue persists.

[Gnarfoz]

Whelp, I retract my particular case. It turns out someone installed Sealed Secrets version 0.12.1 in some obscure namespace over two years ago and it has been interfering with our regular, well-known installation ever since. 🤦‍♂️

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[claytonrothschild]

I had this problem as well, using ArgoCD. Keys would unseal and then immediately say cant unseal in the event logs. Additionally, any time I applied new secrets I would have to restart the pod controller.

The problem was indeed self-imposed - environment setup related.

I had duplicate sealed-secrets controllers in other namespaces, interfereing with one another. The reason I did not catch this is a couple reasons, but one reason is that I was aware the sealed-secrets controller should be sealed-secrets-controller as expected by kubeseal, but the helm chart default controller name is sealed-secrets. Thus I wasn't aware that I had duplicate resources (moving too fast).

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 7 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.